THE SUDAN INFRASTRUCTURE CAPACITY BUILDING PROGRAM MID-TERM EVALUATION REPORT 29 OCTOBER 2010 This publication was produced for review by the United States Agency for International Development. It was prepared by Charles Balina and Douglas Shuster, Management Systems International. THE SUDAN INFRASTRUCTURE CAPACITY BUILDING PROGRAM MID-TERM EVALUATION REPORT A subsidiary of Coffey International, Ltd. Management Systems International Corporate Offices 600 Water Street, SW Washington, DC 20024 Contracted under DFD-I-00-05-00251-00, Task Order No. 2 Services Under Program and Project Offices for Results Tracking (SUPPORT) DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. The Sudan Infrastructure Capacity Building Program – Mid-term Evaluation Report i CONTENTS Acknowledgements ........................................................................................................................ii Program Summary .......................................................................................................................iii Acronyms....................................................................................................................................... iv Maps of Project Area.................................................................................................................... vi Executive Summary.....................................................................................................................vii Introduction.................................................................................................................................... 1 Overview......................................................................................................................................... 6 Findings, Conclusions, and Recommendations........................................................................... 8 Establishment of a Southern Sudan Road Authority .......................................................... 8 Establishment of a Roads Fund......................................................................................... 10 Transitioning from SICBP to GOSS/SSRA...................................................................... 11 Road Selection.................................................................................................................. 12 Road Safety....................................................................................................................... 14 IT and VSAT Solution...................................................................................................... 15 Capacity Building and Training........................................................................................ 17 Sustainability of SICBP Efforts........................................................................................ 22 Gender Issues.................................................................................................................... 25 Communication and Coordination.................................................................................... 26 USAID Southern Sudan Objectives.................................................................................. 27 SICBP Efficiency and Effectiveness ................................................................................ 29 Lessons Learned........................................................................................................................... 32 Annex I – Scope of Work............................................................................................................. 33 Annex II – Comments and Responses on Draft Report............................................................ 45 Annex III – Travel Itinerary....................................................................................................... 60 Annex IV – SICBP Organization Chart .................................................................................... 61 Annex V – List of Key Informants Interviewed ........................................................................ 62 Annex VI – List of Documents Reviewed (Secondary Sources)............................................... 64 Annex VII – Performance Indicators......................................................................................... 66 Annex VIII – Getting to Answers (Data Instruments Used).................................................... 67 Annex IX – Data and Tables from Analysis .............................................................................. 96 Annex X – Sudan Tribune Article on SICBP............................................................................ 97 Annex XI – Representative Project Pictures ............................................................................. 98 The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report ii ACKNOWLEDGEMENTS This report could only be produced within the available time frame because of the generous assistance we received from the following individuals: Louis Berger Group: Ed Flint, Richard Nyarsuk, and Richard Ruati USAID/Sudan: David Gosney, Boutros Magaya, and Terry Kramer Government of Southern Sudan, Ministry of Transport and Roads: Under-Secretary Raymond Pitya Morbe, Director-General Jacob Mariel Maker, and other senior officials from the Ministry Officials from Western Equatoria and Lakes states Local contractors and their trainers from Kisii Training Center Uganda National Road Authority: Engineer Ssebbugga-Kimeze, Director of Operations They were generous with their time and patient in responding to our questions and curiosity. The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report iii PROGRAM SUMMARY Program Name Sudan Infrastructure Capacity Building Program (SICBP) Region/Country Africa/Southern Sudan (Geographic Area 650) Program Bi-lateral under Intermediate Result 10.1: “Core Institutional Structures for an Effective, Transparent, and Accountable GOSS Developed” United States Agency for International Development (USAID)/Sudan Sudan 2008 Operational Plan: Program Element 4.4 (Infrastructure) and Program Sub-Element 4.4.3 (Transport Services) Government of Southern Sudan Counterpart Ministry of Roads and Transport (MTR) Funder and Management USAID/Sudan Technical Office: Economic Growth USAID Contracting Officer’s Technical Representative (COTR): Boutros Magaya Managing Contractor Prime: Louis Berger Group, Inc. (LBG) Implementing Partners:  Kenya Ministry of Roads and Public Works  Kisii Training Center  Kenya Institute of Management  Center for Enterprise Development  Africa Expedition (AFEX) Agreement Information Implemented as Task Order #8 under the Sudan Infrastructure Services Project (SISP) – USAID IQC Contract No. 650-1-00-06-00010-00 (also implemented by LBG) Modifications: None SICBP Impact Evaluation Mission TBD SICBP Mid-Term Evaluation Mission May 29 - June 29, 2010 Key Program Dates SICBP Implementation Period November 3, 2008 – November 2, 2011 Total Contract Value US$13,850,000 The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report iv ACRONYMS BOQ Bills of Quantities CAD Computer Aided Design CBC Capacity Building Component COP Chief of Party COTR Contracting Officer's Technical Representative CPA Comprehensive Peace Agreement DG Director General DRC Democratic Republic of Congo FAR Federal Acquisition Regulations FIDIC International Federation of Consulting Engineers (Fédération Internationale des Ingénieurs-Conseils) FSN Foreign Service National FY Fiscal Year GOSS Government of Southern Sudan IMF International Monetary Fund IP Implementing Partner IQC Indefinite Quantities Contract IT Information Technology JICA Japan International Cooperation Agency KTC Kisii Training Center LBG Louis Berger Group LBT Labor-Based Technology LRA The Lord’s Resistance Army MDTF Multi-Donor Trust Fund MOLACD Ministry of Legal Affairs and Constitutional Development MOU Memorandum of Understanding MSI Management Systems International, Inc. MTR Ministry of Transport and Roads RAMS Road Assets Management System RFTOP Request for Task Order Proposals SDG Sudanese Pound (1 USD ≈ 2.5 SDG) SICBP Sudan Infrastructure Capacity Building Program SIS-CBC Sudan Infrastructure Services – Capacity Building Component SISP Sudan Infrastructure Services Project SPLA Sudan People’s Liberation Army SPLM Sudan People’s Liberation Movement SMPI State Ministries of Physical Infrastructure SSRA Southern Sudan Road Authority TCN Third Country National TO8 Task Order 8 UN The United Nations UNOPS United Nations Office for Project Support The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report v UNRA Uganda National Road Authority USAID United States Agency for International Development VSAT Very Small Aperture Terminal WB The World Bank WES Western Equatoria State WFP World Food Program The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report vi MAPS OF PROJECT AREA Fig. 1: Map of the states of Sudan – with Southern Sudan shown in red Fig. 2: Project area - SICBP covers all ten states of Southern Sudan The program works with the Government of Southern Sudan/ Ministry of Transport and Roads (GOSS/MTR) at the national level. It also supports all ten State Ministries of Physical Infrastructure and has trained contractors in all states. The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report vii EXECUTIVE SUMMARY Development Problem and Proposed Solution Since independence in 1956, Sudan has suffered from civil war, with only a decade of troubled peace. Infrastructure throughout Southern Sudan was destroyed during the years of civil conflict. Lack of physical infrastructure and the physical isolation of Southern Sudanese communities constrain economic growth, make it difficult for vulnerable populations to establish sustainable livelihoods, feed ethnic conflict, and constrain provision of critical governance, economic, and social services. Roads are essential to provide security and employment opportunities and to facilitate peaceful reintegration of returnees into society, but a major obstacle is the acute lack of both public and private financial and institutional capacity to provide road maintenance services. The U.S. Agency for International Development’s (USAID’s) solution to the lack of infrastructure is the US$13.85 million Sudan Infrastructure Capacity Building Program (SICBP) carried out by Louis Berger Group (LBG), the prime Implementing Partner (IP). SICBP builds off of a predecessor project, the Sudan Infrastructure Services-Capacity Building Component (SIS-CBC). SICBP runs from November 3, 2008 to November 2, 2011. SICBP is supposed to contribute to the following goals at the highest level:  Create employment and build local capacity of Sudanese contracting firms  Provide peace dividends in the form of infrastructure development  Improve road access to facilitate transportation and marketing of goods in Southern Sudan  Reduce cost of access to food and food production  Stimulate and modernize the transportation sector The objectives of the task order are to build local, state, regional (Southern Sudan), and national institutions necessary for good and effective implementation of transportation infrastructure development activities in Southern Sudan. The project aims to achieve these objectives by (a) establishing a Road Authority, (b) improving planning capacity, (c) improving management and supervision of road construction contracts, and (d) facilitating coordination with donors and collaboration with states to undertake road network improvements. The goals and objectives are to be achieved by carrying out four key activities:  Assistance to the Ministry of Transport and Roads (MTR) and Establishment of the Roads Authority  Capacity building for State Ministries of Physical Infrastructure (SMPIs)  Capacity building for the local contracting industry, including on-the-job training through Training Contracts  Roads Safety Program Mid-Term Evaluation The principle purpose of this mid-term evaluation was to review project design and assumptions to determine if they remain valid and to provide information to help LBG and USAID modify implementation to improve potential impact. Management Systems International (MSI) fielded a Mid￾Term Evaluation Team consisting of two independent external evaluators, Dr. Charles Balina and Mr. Douglas Shuster, under the Services Under Program and Project Offices for Results (SUPPORT) Program from May 29 to June 29, 2010. The evaluators worked collaboratively with Mr. David Gosney, Mr. The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report viii Boutros Magaya, and Mr. Terry Kramer (USAID senior staff); Undersecretary Raymond Pitya Morbe and Director-General Jacob Mariel Maker (Ministry of Transport and Roads); and Mr. Ed Flint, Mr. Richard Nyarsuk, and Mr. Richard Ruati (LBG) to undertake the evaluation. The methodology included an extensive review of project documentation and interviews and surveys with 50 key stakeholders, including officials from USAID, LBG, MTR and other Government of Southern Sudan (GOSS) officials; SMPIs; Training Contractors; the Multi- Donor Trust Fund; and state, county and local government officials in Yambio (Western Equatoria State), Rumbek (Lakes State), and the Yei (Central Equatoria State) Road training site during three field visits. Conclusions and Recommendations The Evaluation Team answered 14 questions from the scope of work. Following are the major conclusions and recommendations based on the findings, as detailed in the full report. Establishment of a Southern Sudan Road Authority (SSRA) Based on the findings, the Evaluation Team concluded that the establishment of the SSRA is one of the highest priorities of the SICBP project. Based upon discussions with stakeholders, the delays in the establishment of an SSRA are most likely indicative of political will. There is only a small window of opportunity for passage before GOSS becomes focused on the Referendum in January 2011 and the Evaluation Team concludes that it is unlikely that the SSRA will be established until after the Referendum. While capacity has been built within MTR and the states, in the absence of an SSRA, SICBP would have minimal long-term impact on the establishment of a functioning roads system. Recommendations  SICBP should utilize some of the resources targeted at the promotion of the establishment of an SSRA to develop an intensive, short term approach to bolster support among key stakeholders for the establishment of an SSRA.  In concert with the SICBP promotion activity, USAID senior management should make a concerted effort over the next few months (July-August 2010) to push for the establishment of the SSRA, perhaps through a coordinated effort with other donors.  SICBP should develop a contingency plan for reallocating resources in case the SSRA is not established in the next three months. Establishment of a Roads Fund Based on the findings, it appears that the Roads Fund is unlikely to move forward in the near future due to the fact that it is less of a priority than the SSRA and can be phased in over time. While a Roads Fund is the optimal long-term financing mechanism, it is not necessary so long as the SSRA is established, since direct transfers are possible and likely to come from GOSS and/or donors. Recommendation  The SICBP project should continue to focus on the establishment of an SSRA but not on a Roads Fund for the near future. Transitioning from SICBP to GOSS/SSRA The general assumption among stakeholders interviewed was that SICBP staff would transition to the SSRA when the new body is created. However, in the absence of regular and continuous supervision from more senior engineers, as well as additional training, capacity building of SICBP staff will not be adequate for their successful transition. There does not appear to be a long-term strategy for training of the SICBP professional staff, and the evaluators found no evidence that the project had undertaken an assessment of key skills needed by engineers. The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report ix Recommendations  SICBP should increase senior engineers’ supervision over junior engineers.  SICBP should develop a long-term strategy for training the SICBP professional staff in skills aligned with technical requirements of SSRA. (This activity may require SICBP working with USAID to find appropriate vehicles for such training). Road Selection The Evaluation Team concluded that the SICBP roads selection criteria are inadequate. States were given the responsibility to select roads and it appears that they generally made good choices based on their own priorities. However, road selection for Training Contracts does not align with the overall SICBP objective to improve maintenance of road infrastructure investments from USAID, GOSS, and other donors. If road sections are selected that have not deteriorated significantly, labor-based contractors should be able to maintain these roads. Other SICBP tasks do not support selection of feeder roads since there are not enough resources to build state capacity and MTR/SSRA does not oversee feeder roads. Recommendations  SICBP should consider selecting more trunk roads for Training Contracts in order to (a) establish local contractor capacity on trunk roads; (b) give MTR and/or SSRA experience maintaining roads within its jurisdiction; and (c) protect USAID, GOSS and other donor investments in road infrastructure.  SICBP should play a more active role in road selection based on the overall project goals and objectives.  If states are left to select the roads for Training Contracts, SICBP should work with them to utilize the tools they have been given (e.g., developing work plans) to select priority roads.  The Road Asset Management system, if/when operational, should also be used to select roads for Training Contracts. Road Safety There does not appear to have been a thorough needs analysis performed prior to developing the scope and budget for the task. Based upon the description of the general needs identified, the scope seems to have grown beyond an appropriate level. However, a formal needs analysis may determine that the scope is too small to address the Road Safety issues for Southern Sudan. Aside from the drafting of the Road Traffic bill, which is completed, this activity does not align well with the other objectives of the program and detracts from the focus on core objectives. While Road Safety is important, many of the issues are not directly relevant to the primary objective of developing institutions for road infrastructure in Southern Sudan. Recommendation  Given the small number of financial and human resources available and the large number of components within the SICBP program, this activity should be reduced or eliminated if an alternative funding source or program vehicle can be identified. Information Technology and VSAT Solution An appropriate Very Small Aperture Terminal (VSAT) system is critical to the functioning of a road infrastructure system. SICBP’s technological approach could be feasible; however, feasibility is difficult to assess in the absence of a feasibility study. The system will not be sustainable unless MTR and the states clearly understand the responsibilities for upkeep and capabilities of the system for road infrastructure development. The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report x Recommendation  SICBP should perform a sustainability analysis for the system, including a cost-benefit analysis, to demonstrate to MTR and the Ministry of Finance the advantages of the system and prospective cost savings for travel and communications with the states. Capacity Building Though the capacity building at MTR has had a measure of success, there continues to be debate about the justification for continued funding of capacity building for a department with no guarantee that the people being trained will be playing the same role in the Road Authority. The program has not been so successful at the state level because the state’s importance in planning, funding and executing feeder roads programs was not adequately addressed in the project’s design. Consequently, the states were excluded from critical steps of the procurement process. The most successful component of SICBP has been building the capacity of local contractors. However, it is still too early for the program graduates to take on large projects (over one million dollars). The use of Federal Acquisition Regulations (FAR) and LBG provisions as special provisions attached to the International Federation of Consulting Engineers (FIDIC) contract form is a problem that needs to be addressed, as some of the special provisions do not apply to a training program. Other key conclusions found were that the start of Training Contracts does not consider weather impacts on construction; follow-up supervision of trained contractors is inadequate; states are not adequately involved in the contract procurement process; and there are unnecessary hurdles in contract costing (e.g., ten percent performance bond, payment process). According to the contract documents, the roads meet standard specifications as used in the East and Central African region. In the absence of continuous daily supervision on site to monitor construction, however, there is no guarantee that specifications were followed through the entire process. Lack of axle load negatively impacts longevity of improved roads. Recommendations  LBG should develop an appropriate contract vehicle that takes into account the fact that this is a training program. Possible options include a purchase order mechanism.  Training contractors should be trained in use and handling of mechanized equipment in order to be able to manage trunk road maintenance and to manage better feeder road improvement contracts.  The curriculum should be expanded to include more planning and financial management and bridge construction and rehabilitation.  LBG should schedule more regular follow-up supervision from site engineers and Kisii instructors.  SICBP should help build the procurement capacity of states through participation in the procurement of Training Contracts. Sustainability of SICBP Efforts There is evidence that the institutions necessary for good and effective implementation of transportation infrastructure development activities have been strengthened, particularly MTR. However, financial sustainability for road infrastructure at the GOSS and state levels in the absence of an SSRA is unlikely. Due to resource constraints, USAID has provided some of the tools for state road infrastructure development but has not built the long-term capacity of the states under this program. They do not have the human or financial resources to plan, budget, and manage road infrastructure projects. The SICBP approach of providing basic tools such as equipment and basic training without adequate capacity building is an unsustainable approach. The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report xi SIS-CBC and SICBP have developed a pool of local and regional contractors that have the capacity to carry out road infrastructure improvement and maintenance but there are limited contract opportunities for small, labor-based contractors. This lack is a major hurdle to the long-term sustainability of this activity. Recommendations  If states remain a focus of the program, efforts should be made to maximize existing resources within SICBP to build their capacity.  If it is decided that the states should not receive additional capacity building, USAID should not channel additional resources (e.g., equipment) into the states.  SICBP and USAID should work closely with GOSS and other donors to develop programs – such as small business or local contractor set-asides, or subcontracts - to promote local contractors and identify and develop opportunities for them. Gender Issues SICBP recognizes the importance of equitable participation of men and women in USAID programs. However, no specific gender strategy was developed for the project. Recommendations  SICBP should develop a pro-active strategy for including women in labor-based contracting. This strategy should be based on research from regional labor-based programs (i.e., Uganda, Ethiopia, and Kenya).  SICBP should identify ways that the program can assist women to take more advantage of the services that the roads facilitate. This work would likely require synergy with other USAID or partner projects in the areas of health, microfinance, and agriculture. Communication and Coordination Better communication is needed between SICBP and its project partners in order to achieve the project’s objectives. Recommendations  The project should work to improve communication with MTR and the states through regular, monthly or quarterly meetings.  The project should maximize any potential advantages of using the VSAT system to communicate with the states in real time (e.g., communications and meetings between MTR/GOSS and the states should be virtual using the new VSAT system, rather than requiring state staff to travel all the way to Juba). SICBP Efficiency and Effectiveness There is a strong possibility that at the end of the project, the states will have a cadre of technical people who can supervise road maintenance and nothing else. More efforts are required to bring the states into claiming ownership of the project at the state and county levels. There is a preponderance of constraints that inhibit the successful execution of the Training Contracts that need to be addressed. Recommendations  SICBP should make a concerted and deliberate effort (within available funding) to identify and address the constraints to the successful operation of the Training Contracts. The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report xii  If funds are available, more resources should be provided to augment the capacity building effort with the states.  Additional efforts are required from LBG in supervising both the contractors and the junior engineers. USAID Southern Sudan Objectives The SICBP program aligns well with USAID’s strategic objectives for Southern Sudan, such as increasing security, good governance, improving access to services, and food security. Roads are generally recognized as an important part of the peace dividend (i.e., through improved security, access to markets and services) and at least some of the roads improved through the SIS-CBC and SICBP are perceived locally as a part of the peace dividend. Roads improved or maintained under the project have provided access to services, increased economic activity, and provided increased security. However, based upon findings under the Sustainability section of this report, there is little evidence to suggest that broader economic and social impacts are being achieved throughout Southern Sudan as a result of the project. This shortcoming is due to the failure of the SSRA to be established and, therefore, the limited impact of the program on road infrastructure improvement and maintenance in the region. SICBP Project Design, Implementation, and Impacts The program overall was well designed to achieve its goals and objectives. However, the objective of developing a road maintenance regime for Southern Sudan capable of protecting the investments of USAID, other donors, and GOSS is not being adequately addressed in the implementation of the project. Some design elements of the original project have since been eliminated (e.g., civil aviation), leading to a project that is more focused on road infrastructure. Southern Sudan is an extremely difficult environment in which to operate due to logistics and security issues and the nascent stage of the government. SICBP is also a complex project with a large and diverse set of stakeholders. LBG is generally performing well considering the challenging environment and project goals and objectives. However, it is behind schedule in terms of a number of key deliverables, including the Information Technology (IT) and VSAT System and the Asset Management System, which are critical not only to the success of the project but also to the implementation of the project’s activities. GOSS’s failure to establish an SSRA jeopardizes the potential impacts of the program. Recommendations  For the current SICBP project, USAID should modify some of the activities and resource allocations, but it should continue the program.  Program extension should be contingent upon establishment of a Road Authority or development of some other institution or vehicle that can ensure the efficient, effective, and transparent management of road infrastructure planning, construction and maintenance. The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 1 INTRODUCTION A. Country Background Sudan is the largest country in Africa, covering an area of 967,499 square miles, with an estimated population of 42 million1 . It has suffered from two conflicts between the North and South since independence in 1956, separated by a period of troubled peace from 1972 to 1983. A separate conflict began in 2003 in the western region of Darfur. In addition to the loss of lives, opportunities and infrastructure, the wars displaced families and divided communities. Consequently, the Sudanese people’s health, education and infrastructure are among the poorest globally. In 2005, peace talks resulted in the signing of a Comprehensive Peace Agreement (CPA), which granted a semi￾autonomous status to the south for six years, after which a referendum is scheduled to be held in January 2011. Southern Sudan consists of ten states that formerly made up the three historic provinces of Bahr-el-Ghazal (Western Bahr el Ghazal, Northern Bahr el Ghazal, Warrap, and Lakes States), Equatoria (Central, Western, and Eastern Equatoria States) and Upper Nile (Upper Nile, Unity, and Jonglei States). The region has an area of 247,105 square miles – about the size of Montana and Wyoming put together – with population estimates ranging from 7.5 million to 10 million2 . Since the signing of the CPA, Sudan has taken steps toward peace, reconciliation and good governance. Despite these efforts, Southern Sudan remains a vulnerable state. The Government of Southern Sudan (GOSS) is working with the international community to ensure that overall economic, political and social development form part of the “peace dividends” from the CPA. USAID has been an important supporter of the GOSS. B. Sector Background The Southern Sudan transport system consists of four major modes of transport – roads, railways, inland water (river) transport and air transport. Of these, roads are the primary mode of travel. The road network has, however, never been classified or inventoried. The region has an estimated network of about 12,642 km (7,900 miles), of which 7,369 km (4,605 miles) are interstate roads, 1,452 km (907 miles) are state primary roads and 3,822 km (2,389 miles) are state secondary roads. Most of these roads were not maintained during the wars and are currently in a state of disrepair. The signing of the CPA brought forth a steady money stream from oil revenue, which the GOSS is targeting for its overall development efforts. Under its transport sector policy, GOSS attention to the road sector has been focused on delivering accessibility, mobility, safety, quality of life, system preservation, management and productivity, and organizational excellence. Work on improving the transport sector has now started. A few of the interstate and primary roads have been improved to gravel standards under the World Food Program (WFP) and the United Nations. GOSS is also undertaking initial work to gravel some roads (the actual quantities are not given in the source documents)3 . The U.S. Agency for International Development’s (USAID’s) involvement in this sector has been necessary because of the need to protect U.S. tax payer investments made under the 3-year Sudan Infrastructure Services – Capacity Building Component (SIS-CBC). 1 CIA – The World fact book: - https://www.cia.gov/library/publications/the-world-factbook/geos/su.html 2 http://sudan.unfpa.org/souther_Sudan/index.htm 3 Ministry of Transport & Roads, GOSS: Transport Sector Policy, October 2007 The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 2 C. Program Description4 The Sudan Infrastructure Capacity Building Project (SICBP), is a new Task Order (Task Order 8) under USAID’s Sudan Infrastructure Services Program (SISP), implemented under contract with the Louis Berger Group (LBG). The project has a budget of $13.85 million and its activities run from November 3, 2008 to November 2, 2011. SICBP is to provide capacity building support for infrastructure development in Southern Sudan. The objectives of the task order are to build local, state, regional (Southern Sudan), and national institutions necessary for good and effective implementation of transportation infrastructure development activities in Southern Sudan. The project aims to achieve these objectives through the following:  Establishing a Road Authority,  Improving planning capacity,  Improving management and supervision of road construction contracts, and  Facilitating coordination with donors and collaboration with states to undertake road network improvements. In addition, SICBP’s activities are supposed to contribute to the following goals at the highest level:  Create employment and build local capacity of Sudanese contracting firms,  Provide peace dividends in the form of infrastructure development,  Improve access by road to facilitate transportation and marketing of goods in Southern Sudan,  Reduce the cost of access to food and food production, and  Stimulate and modernize the transportation sector. SICBP involves four task areas, as follows:  Assistance to GOSS/Ministry of Transport and Roads (MTR) and establishment of the Road Authority,  Capacity building for State Ministries of Physical Infrastructure (SMPIs),  Capacity building for the local contracting industry, and  Road Safety. D. Linkage to USAID/Sudan Strategy and U.S. Government Foreign Assistance Framework SICBP contributes to the USAID/Sudan Strategy by providing tangible peace dividends in the form of feeder road development, thus strengthening support for the CPA. It also directly addresses Intermediate Result 10.1: “Core Institutional Structures for an Effective, Transparent, and Accountable GOSS Developed” by providing equipment, training and technical assistance to the GOSS MTR, as well as to selected state ministries of transportation in Southern Sudan. Under the USAID/Sudan 2008 Operational Plan, this activity addresses Program Element 4.4 (Infrastructure) and specifically, Program Sub-Element 4.4.3 (Transport Services). E. Purpose of the Evaluation This is a formative, mid-term evaluation, slightly past half-way through the 3-year project. Its principle purpose is to review project design and assumptions to determine if they remain valid and to provide information to help the Implementing Partners (IPs) and USAID modify implementation to improve potential impact. The evaluation also assesses impact to date and gathers lessons learned and success stories. 4 See SOW in Annex I. The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 3 This evaluation sought to answer the following, specific questions on the project5 : Project Design  Are there any issues with respect to project design and assumptions (documented or implied) that should be reconsidered based on experience to date? Project Implementation  SICBP is a large, complex project with many activities and moving parts. Have the IPs succeeded in maintaining quality over the various pieces while maintaining production at an appropriate cost?  Are there ways in which the effectiveness and efficiency of SICBP could be improved?  TO8 developed contract management and administrative systems and is engaged in a significant level of procurement and oversight. Are these systems working in a way that is effective, transparent, and designed to minimize corruption?  Are TO8 human resources sufficient to achieve the project’s goals?  The GOSS relies on SICBP for support on a number of levels at the same time that SICBP is encouraging its government partners to become self-sufficient. How well is LBG balancing these competing pressures?  What gender issues have been addressed during the implementation of SICBP?  Please assess TO8 communication and coordination with project partners. Project Impact to Date  Is there evidence that local contractor capacity has been improved?  What evidence is there that state, regional (Southern Sudan), and national institutions necessary for good and effective implementation of transportation infrastructure development activities in Southern Sudan have been strengthened?  What evidence is there to suggest that broader economic and social impacts are being achieved? Lessons Learned/Success Stories  How is SICBP supporting other USAID projects and objectives?  Based on learning from the evaluation, is a shift in resource allocation among project components called for? Special attention should be given to the state level.  What capacity building approaches appear to be particularly productive? The intended audience for this evaluation is USAID and the IPs, who will use the report’s findings, conclusions and recommendations to make any necessary adjustments to the project’s design and implementation in the second half of its cycle. Any lessons learnt will also be used in the design of any follow-on or similar projects. F. Methodology The Evaluation Team This was a five-person, collaborative evaluation team consisting of two external evaluators representing MSI, and three internal evaluators representing USAID, LBG and GOSS/MTR. The team leader and first external evaluator, Dr. Charles Balina, is a practicing civil engineer who also teaches at the University of Maryland University College’s Graduate School of Management and Technology. He has worked internationally on infrastructure projects for over 27 years. He was a Contracting Office’s Technical Representative (COTR) with USAID/Uganda for nearly ten years, during which he managed, among other projects, the Mission’s 5 List shows main questions. See Annex I for complete list of main and sub-questions. The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 4 post-war rural feeder roads program. Dr. Balina is familiar with construction practices in the East and Central African region. The second external evaluator, Mr. Douglas Shuster, is an economist and investment expert who has evaluated over 800 projects in 60 countries for the U.S. Trade and Development Agency. He has completed several assignments in infrastructure planning and investment, working in sectors such as transportation, water and sanitation, and power and energy. USAID was alternately represented on the team by two engineers, Mr. Boutros Magaya, COTR for SICBP, and Mr. Terrence Kramer, senior Roads Engineering Advisor with USAID/Sudan. GOSS was represented by Mr. Jacob Mariel Maker, Director General of Roads and Bridges at the Ministry of Transport and Roads. The IP, LBG, was represented by Mr. Richard Nyarsuk, an engineer and Deputy Manager of TO8. Mr. Nyarsuk and Mr. Richard Ruati, the TO8 Liaison and Public Affairs Officer, accompanied us on all our field trips. They were both fluent in the local languages spoken in the project areas which the team visited. Meetings, Field Activities, and Visits The external evaluators arrived in Juba on June 3, 2010. In the period between then and its departure on June 25, 2010, the team met over 50 people who have closely been involved in or benefitted from the project. The team made three field visits: to Yambio (Western Equatoria State), to the contractor training at the Yei (Central Equatoria State) Road facility and to Rumbek (Lakes State). The evaluation team’s complete travel itinerary and list of people met are included in the Annex. Data Collection Procedures and Instruments Used The team completed its “Getting to Answers” worksheet (see Annex VIII) in Juba. Evaluators used a variety of quantitative and qualitative data collection methods, including document review, observation, group interviews, key respondent interviews, phone interviews, site visits and surveys. Together with the project documents, these data provided the variety of data sources necessary to triangulating findings. The lists of key people interviewed, documents reviewed, and the data instruments used are all included in the Annexes. Data Analysis The Evaluation Team analyzed the data obtained in several ways: by comparing and contrasting, content pattern analysis, trend analysis, cross-tabulations, unit cost calculation and frequency distributions. Data Limitations Constraints that included scheduling conflicts, availability, and timing rendered it impossible to meet all relevant individuals and collect all relevant data. The team was not able to meet and benefit from the input of the following people:  MTR Chief Engineer James Alam, who has deep knowledge of MTR’s operations.  GOSS Legal Affairs Advisor Bill Kosar, who has been a key player in reviewing the SSRA bill.  The local representatives of the WFP and the local representative of the United Nations Office of Project Services (UNOPS), the two bodies that are repairing many GOSS roads.  USAID/Sudan General Development Officer Carmelita Maness, who has been on the program for over a year, working with the Office of Economic Growth.  LBG Chief of Party David Little and the LBG Contracting Officer, who were the most knowledgeable people on SICBP’s contracting procedures and documentation. The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 5 Another major limitation was that the project data (for example, all training done through the program) were not always available in the desired format, or were dispersed between so many source documents that there was not enough time to reformat it in a way that enabled complete analysis. Therefore, even when individually available, some of the data were not used in the compilation of this report. Some data, such as a consolidated report of the planned and current status of all the project performance indicators; a complete list of all people trained, the training they received and which state (or body) they came from; and the complete list of performance indicator data were either not received or were not received in a timely manner from the IPs. Therefore, a complete analysis of these data could not be incorporated into the report. Further, only a limited number of sites were visited (three states of Central Equatoria, Western Equatoria, and Lakes) out of the ten states where the project is active. The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 6 OVERVIEW Based on the extensive findings detailed in the report, the Evaluation Team has made the following conclusions and recommendations regarding the overall design, branding, indicators, implementation, and impacts of the SICBP project. Design Conclusion The program overall was well designed to achieve its goals and objectives. However, the objective of developing a road maintenance regime for Southern Sudan capable of protecting the investments of USAID, other donors, and GOSS is not being addressed in the implementation of the project. Some design elements of the original project have since been eliminated (e.g., civil aviation), leading to a project that is more focused on road infrastructure. Recommendation Other design elements should be reassessed, including: whether the Road Safety activity detracts from the project focus and what should be done in the absence of an SSRA. Branding Conclusion Branding of the project is not effective. SICBP does not capture the terms “Road Authority” or “Road Maintenance.” The project is often referred to as Task Order 8 or TO8, which is even less informative. While the SIS-CBC project started in the absence of an MTR, and was therefore the lead for all road infrastructure capacity building in Southern Sudan, the existence of other capacity building projects funded through other donors such as, the Multi-Donor Trust Fund (MDTF) and the Japan International Cooperation Agency (JICA), now obscure the focus of SICBP. Recommendation USAID should consider changing the branding in order to increase stakeholders’ focus on the program’s key objectives, thereby improving the effectiveness of the program. Indicators Conclusion Performance indicators appear to be geared toward reporting to the U.S. Department of State’s Foreign Assistance Bureau, such as number of people trained. Indicators such as the number of contractors securing non-SICBP funded contracts would be more revealing of the project’s success. Recommendation A few highly relevant performance indicators that help maintain project focus and measure project success should be developed and tracked. The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 7 Implementation Conclusion Southern Sudan is an extremely difficult environment in which to operate, due to logistics and security issues and the nascent stage of the government. SICBP is also a complex project with a large and diverse set of stakeholders. According to our observations and interviews with senior staff from USAID, LBG is generally performing well considering the challenging environment and the project’s ambitious goals and objectives. However, it is behind schedule in terms of a number of key deliverables, including the Information Technology (IT) and Very Small Aperture Terminal (VSAT) System and the Asset Management System, which are critical not only to the success of the project, but also to the successful implementation of project activities. LBG’s SICBP project has a highly professional staff, including a very strong group of local Sudanese, particularly considering the challenging recruitment environment. However, local staff morale is low. Recommendation The low morale should be addressed through open communications and other interventions (e.g., incentives, additional training) to ensure that it does not impact project implementation. Impact Conclusion GOSS’s failure to establish a Southern Sudan Road Authority (SSRA) jeopardizes the potential impacts of the program. Recommendation For the current SICBP project, USAID should modify some of the activities and resource allocations, but should continue the program. Extension of the program should be contingent upon establishment of a Road Authority or development of some other institution or vehicle that can ensure the efficient, effective, and transparent management of road infrastructure planning, construction and maintenance. In order to avoid a potential gap in implementation, USAID should develop a plan for additional support in the event that such an institution is established within a few months of the end of the current SICBP project. The evaluation team’s specific findings, conclusions, and recommendations follow. The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 8 THIS SECTION OF THE REPORT ADDRESSES THE FOLLOWING EVALUATION QUESTIONS: 1.a An operating Southern Sudan Road Authority (SSRA) has been considered a critical institutional piece for a functioning roads system. Can the project have long-term impact in the absence of an SSRA? 1.a.i What is the likelihood of the SSRA becoming a reality? FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS Establishment of a Southern Sudan Road Authority Findings SSRA as a Critical Institutional Piece According to project documents and interviews with senior staff from USAID, LBG, and MTR, establishment of an SSRA is one of the highest priorities of the SICBP project. The fact that GOSS is building its institutions from scratch offers a unique opportunity to implement international best practices such as, a roads authority without the burden of dismantling a large, overstaffed, bureaucratic force account system, as have other countries in the region, such as Ethiopia and Uganda. It is one of the key deliverables under Activity 1. Assistance to MTR and Establishment of Roads Authority in the approved 2009 and 2010 work plans is “Road Agency established and operational.” According to all senior staff interviewed at USAID, LBG, MTR, and the World Bank/MDTF, as well as the Southern Sudan Roads Authority bill, 2010 Cabinet Memo, an SSRA is important for a functioning roads system for the following reasons:  increased technical capacity in an SSRA – due to implementation of an alternative to GOSS civil service regulations regarding hiring/firing – can promote a performance-based model for staffing;  independence from political influence; and  transparency through established procurement and financial processes (auditable). MTR’s Director General (DG) and Undersecretary spoke of the SSRA as a priority for the development of the roads sector and demonstrated a clear understanding of the benefits of the establishment of the SSRA. All of the 14 State representatives surveyed knew about the SSRA and had a general, though more limited, understanding of the benefits. All of the state officials surveyed ranked the SSRA as “important” or “very important.” Timing and Process for Establishment of an SSRA According to MTR’s DG for Roads and Bridges, the SSRA bill has passed through the Ministry of Legal Affairs and is currently with the Minister for Transport and Roads, who has indicated that it will be presented to the Cabinet of Ministers shortly after the Cabinet is appointed. Upon approval by the Cabinet, the bill would be sent forward for Parliamentary approval. However, it was also noted that the Cabinet of Ministers will most likely want changes to the bill before it goes forward to Parliament and that the speed with which Parliament reviews and potentially passes the bill depends upon the number and prioritization of bills it receives for consideration. According to interviews with senior staff at USAID, LBG, and MTR, there was an expectation that there would be an SSRA established within the first 6 months of the project, which began in November 2008, since the groundwork had been laid during the SIS-CBC program. USAID and LBG senior staff agreed that, while there have been legitimate delays due to the process of passing legislation under a fledgling government, there are likely people within GOSS that do not want the SSRA to be established due to the potential loss of control of funding and authority. However, senior staff at USAID, LBG and MTR all described the current Minister as more proactive than the previous two Ministers in pushing for an SSRA. The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 9 Regarding the timing for the establishment of an SSRA:  USAID senior staff thought it was unlikely that an SSRA would happen before the Referendum.  A senior MTR official thought the SSRA would be established in the next 3 months.  LBG senior staff thought the SSRA would be established before the Referendum. USAID is providing close to $240 million for roads, and senior staff members consider that it should have some weight to push for a viable road maintenance agency. Senior World Bank staff also indicated strong support and said that when the Bank does have loan funding for road projects in Southern Sudan (it currently only manages MDTF funds), it would likely make establishment of an SSRA “a condition for negotiation.” According to interviews with senior USAID and LBG staff, neither SICBP nor USAID has systematically promoted establishment of an SSRA. LBG is planning a policy workshop at the end of July 2010 for stakeholders and the establishment of an SSRA is a focus of that event. Long-Term Impact in the Absence of an SSRA Senior staff within USAID, the World Bank/MDTF, and LBG agreed that there can be some long-term impact of SICBP in the absence of an SSRA, but not as much as expected. (Note: see Sustainability of SICBP Efforts section of this report for findings and conclusions regarding long-term impacts). A senior MTR official believes there can be long-term impact without an SSRA, through the development of existing project units within MTR. However, senior staff at USAID indicated that the Agency had spent close to $20 million on developing an SSRA through the SIS-CBC and SICBP projects, is not interested in setting up units within MTR, and purposely did not set up units in MTR because doing so would potentially build an entrenched civil service without the capacity of an SSRA. USAID originally expressed concern about MDTF setting up such units at MTR for the same reason. Moreover, a senior USAID official indicated that the Agency was not likely to spend any additional funds on strengthening MTR in the absence of an SSRA. Options in the Absence of an SSRA According to interviews with senior staff from USAID, LBG, and MTR, there was no contingency plan during the design of SICBP in case the SSRA was not established and there is currently no plan in place. Senior USAID and LBG staff thought SSRA funding should be reallocated to other areas within the project, while MTR thought that SSRA funding should continue to go to capacity building for MTR staff. Conclusions Based upon discussions with different stakeholders, the Evaluation Team determined that delays in the establishment of an SSRA are most likely indicative of political will for the organization. Based upon the fact that the SSRA has not been established 18 months into the program and there is only a small window of opportunity for passage before GOSS becomes focused on the Referendum in January 2011, the Evaluation Team concludes that it is unlikely that the SSRA will be established until after the Referendum. While capacity has been built within MTR and the states (see Sustainability of SICBP Efforts and Capacity Building sections of this report), the Evaluation Team concludes that without an SSRA, SICBP would have minimal long-term impact on the establishment of a functioning roads system due to the critical nature of the SSRA for road infrastructure development. The only significant, long-term impact of the project in the absence of an SSRA would be in the area of developing the capacity of local Southern Sudanese and regional contractors in road construction contracting. This subject is discussed in the Sustainability of SICBP Efforts section of this report. The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 10 THIS SECTION OF THE REPORT ADDRESSES THE FOLLOWING EVALUATION QUESTIONS: 1.b Medium and long-term financial sustainability would seem to be dependent on an operational financing mechanism, such as a Roads Fund. What is the likelihood of this becoming a reality? 1.b.i What are USAID and GOSS options in the absence of a functioning financing mechanism or to develop one ? Recommendations  SICBP should utilize some of the resources targeted at the promotion of the establishment of an SSRA to develop an intensive, short-term approach to bolster support for the establishment of an SSRA among key stakeholders, with documentation providing a justification for the SSRA based upon regional experience.  Such an approach could include coordination with other donors (e.g., the MDTF), the states, other ministries and other stakeholders. The proposed July 2010 policy workshop could be a good venue for intense promotion of the SSRA.  In concert with the SICBP promotion activity, USAID senior management should make a concerted effort over the next few months (July-August 2010) to push for the establishment of the SSRA, perhaps through a coordinated effort with other donors.  SICBP should develop a contingency plan in case the SSRA is not established in the next three months, with funding being reallocated from Task 1. Assistance to MTR and Establishment of Roads Authority to other areas.  In line with a suggestion by USAID, such a plan could include putting more funding into Training Contracts, particularly for the maintenance of USAID-funded roads under SISP. Establishment of a Roads Fund Findings Likelihood of Roads Fund Becoming a Reality According to interviews with senior staff at LBG, USAID, and MTR, the establishment of the Roads Fund is unlikely in the near future. They told the Evaluation Team that the establishment of a Roads Fund, whereby the SSRA collects and spends revenues collected independent of the Ministry of Finance, is unconstitutional, according to the Ministry of Legal Affairs. LBG senior staff stated that they have stopped working on the Roads Fund under SICBP due to the issues which the Ministry of Legal Affairs raised. They said it was “off the table” in terms of their work under SICBP. According to senior staff at the World Bank, USAID and LBG, the International Monetary Fund (IMF) is not supportive of a Roads Fund, as they prefer a single account mechanism for funding flowing into the country to maintain accountability of GOSS. The World Bank indicated that it has become more supportive of the Roads Fund model for financing roads infrastructure due to the positive results in other African countries, such as Ethiopia, Ghana and Uganda. Options in the Absence of a Roads Fund A senior MTR official told the Evaluation Team that MTR does not see the Roads Fund as critical to the establishment of a well-functioning road system. Senior staff members at USAID and the World Bank see a Roads Fund as the optimal long-term financing mechanism, but something that can be phased in over time. The example was given of Uganda’s road authority, which was established years ahead of an operational roads fund. Senior staff at USAID, MTR, LBG and the World Bank agreed that it was not necessary to have a Roads Fund since funding could flow directly from GOSS and/or donors to the SSRA, when established. As an auditable agency, the SSRA would be likely to attract donor financing. A senior USAID official also noted that it is possible that a special unit within the Ministry of Finance could be established to serve as a virtual Roads Fund in order to get around the constitutionality issues. The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 11 THIS SECTION OF THE REPORT ADDRESSES THE FOLLOWING EVALUATION QUESTIONS: 1.d Many individuals are working for LBG under SICBP with the understanding that they will transition to government after the SICBP. How will those individuals be retained and operate after this project? 1.d.i Is there potential for this to be a successful approach? Conclusions Based upon the findings, it appears that the Roads Fund is unlikely to move forward in the near future, primarily due to the fact that it is less of a priority than the SSRA and can be phased in over time. The Evaluation Team concluded that while a Roads Fund is the optimal long-term financing mechanism, it is not necessary so long as the SSRA is established, since direct transfers are possible and likely to come from GOSS and/or donors. Recommendations  The SICBP project should continue to focus on the establishment of an SSRA, but not on a Roads Fund, for the near future. Transitioning from SICBP to GOSS/SSRA Findings According to the Road Authority bill, the road construction and maintenance functions of MTR will be transferred to the Road Authority. A senior MTR official stated that the government expectation is that SICBP Sudanese staff will be transitioned to SSRA. All of the senior USAID and LBG staff agreed that this was the expectation. SICBP engineers were told in pre-interview briefings that this was a major potential benefit to taking the position. As a way of getting ready for this transition, all of the seven engineers feel they need training in road maintenance management, road design, environmental management and road asset management. In the absence of a senior engineer to mentor these staff, this role was being undertaken by the training engineer. However, a senior LBG official admitted that the training engineer lacked some skills necessary to take on this role. Unlike local contractors and state supervisors, the local engineers do not get the benefit of a full road maintenance course similar to the one in Kisii, and are only able to sit in when the instructors from Kisii come to deliver their on-the-job training in Juba. Two of the seven local engineers have not received this training. Two engineers, however, recently received USAID environmental engineering training. Morale is low amongst the local SICBP staff members, who feel they have had little opportunity to build capacity and get ready to compete for jobs in the SSRA. They feel the overall culture does not favor training and they do not get the benefit of learning from more experienced engineers on the job. The SICBP manager and his deputy are busy, and the senior engineer from Kenya has only come once in the last 6 months, for a 2-week workshop. They have written to management to bring up the issue of training, but never received any formal response. When the deputy SICBP manager was invited to present a paper in the U.S. at a meeting of the International Road Federation, he had to charge some of his time off to vacation, even though the project leadership had endorsed his trip. Currently, there appears to be no specific plan for training. Senior SICBP and USAID staff agreed that if the SSRA was formed, then training would be better planned, more specific and appropriate. Senior LBG staff members acknowledge that the junior engineers had little gravel road experience when recruited by LBG. Though it was difficult getting experienced Sudanese engineers, LBG made exceptional efforts with USAID encouragement to do so. LBG, they thought, was successful in forming an all-Sudanese engineering team. The team was clearly building some capacity and confidence through on-the-job learning. However, observations of site engineer oversight in Rumbek as well as reviews of field reports and interviews regarding the decision￾making processes in payment certification demonstrated that there was not adequate oversight of SICBP engineers in some cases. The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 12 THIS SECTION OF THE REPORT ADDRESSES THE FOLLOWING EVALUATION QUESTIONS: 1.e How were the roads under SICBP selected? What criteria (if any) were used? Conclusions Based upon the hiring interviews, the general assumption among the SICBP engineers was that SICBP staff would transition to the SSRA when the new body is created. If the new body is not created by the time SICBP comes to an end, it is unlikely the local engineers will be willing to join the MTR under the current civil service compensation plans. It is, therefore, unlikely that GOSS/MTR will benefit from these engineers. In the absence of regular and continued supervision from more senior engineers, capacity building of SICBP staff will not be adequate for successful transition, which is partly demonstrated in the quality of their field reports and decision-making processes in payment certification. When the SSRA is created, without adequate capacity building of the local professional staff, the current approach may not be successful in ensuring an available initial pool of capable engineers to run the roads maintenance regime in the SSRA. There does not appear to be a long-term strategy for SICBP professional staff training and the evaluators received no evidence that an assessment of key skills the engineers needed had been undertaken. Recommendations  SICBP should increase senior engineers’ supervision of the junior engineers.  SICBP should develop a long-term strategy for training the SICBP professional staff, which may require working with the USAID COTR to find appropriate vehicles for such training under the contract.  SICBP should provide more training for local staff in skills aligned with SSRA technical requirements. Road Selection Findings According to a senior USAID official, one of the original SICBP objectives was to develop institutional, technical, and financial capacity to maintain roads rehabilitated or improved through USAID and other donor investment, which have primarily been trunk roads. However, this objective was not used as a criterion for selecting roads for Training Contracts. According to an interview with an MTR senior official, MTR was not consulted during the road selection process. According to all of the senior staff interviewed at LBG and USAID, the selection process was to ask the states to identify a number of kilometers of roads which they wanted to be improved or maintained. According to the group interview with 14 SMPI representatives and the Evaluation Team’s observations during site visits to Lakes and Western Equatoria States, states selected roads based upon their own priorities, which varied. Criteria mentioned included security, access to services (i.e., hospitals, schools), access to agricultural production areas and connecting population centers. Senior state and county officials in Western Equatoria said they largely viewed SICBP as a road improvement program, most likely because road improvement was the greatest benefit they saw from the project. A senior USAID official reminded state officials that SICBP was primarily a capacity building program and the road improvements were only a side benefit. According to a senior LBG staff member, some of the states had development plans in place that had identified priority road projects. Others selected roads based upon discussions between state and county officials. The Road Asset Management System, which was expected to be operational by October 2009, is still not operational and was not used for road selection. The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 13 Despite requests to LBG, detailed data on the types of roads used for Training Contracts were not provided in a timely manner to the Evaluation Team. Senior LBG staff said that 5 of the 13 Training Contracts were for trunk roads and 4 were for town roads. Therefore, it can be deduced that four were for feeder roads. A senior LBG staff member stated that feeder roads were better for Training Contracts than trunk roads because feeder roads provided a good venue for learning and practicing technical skills in road construction. It was stated that training on recently improved trunk roads would not address technical skills which contractors needed. The Evaluation Team found different opinions regarding whether labor-based contractors could manage contracts to maintain trunk roads.  The senior World Bank officials interviewed estimated that 30 to 50 percent of Southern Sudan’s roads could be maintained through labor-based contracts. His experience in Ethiopia demonstrated that this is a very successful approach.  MTR senior officials stated that labor-based contracts are not appropriate for trunk roads.  A senior LBG staff member stated that maintenance of trunk roads was too easy for Training Contracts. However, he also said that maintenance of trunk roads would require more equipment than the labor-based contractors have access to or are trained to manage.  A senior USAID official stated that maintenance of trunk roads should be selected for Training Contracts and that labor-based contractors could manage these projects as long as they were maintained in a timely manner before major degradation occurred. Conclusions The Evaluation Team concluded that SICBP does not have any selection criteria for the roads. States were given the responsibility to select roads and it appears that they generally made good choices based upon their own priorities. Road selection for Training Contracts does not align with the overall SICBP objective to improve maintenance of road infrastructure investments from USAID, GOSS, and other donors. If road sections that have not deteriorated significantly are selected, labor-based contractors should be able to maintain these roads. Selection of feeder roads is not supported by other SICBP tasks since there are not enough resources to build state capacity to oversee feeder road maintenance and MTR/SSRA does not oversee feeder roads. Southern Sudan has vastly different terrain and availability of labor and other materials. One kilometer of road in one location could require ten times the amount of work as one kilometer of road in another location. There is also a large difference between improvement and maintenance of roads. Recommendations  SICBP should consider selecting more trunk roads for Training Contracts in order to  Establish local contractor capacity on trunk roads,  Give MTR and/or SSRA experience maintaining roads within its jurisdiction, and  Protect USAID, GOSS and other donor investments in road infrastructure.  SICBP should play a more active role in road selection based upon the overall project goals and objectives.  If states are left to select the roads for Training Contracts, SICBP should work with them to utilize the tools they have been given (e.g., developing work plans) to select priority roads.  The Road Asset Management system, if/when operational, should also be used to select roads for Training Contracts. The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 14 THIS SECTION OF THE REPORT ADDRESSES THE FOLLOWING EVALUATION QUESTIONS: 1.g Is the scope of the road safety program appropriate based on the needs identified? 1.g.i Should it be expanded or reduced ? Road Safety Findings Needs Assessed Based upon the Evaluation Team’s review of the Work Plans and interviews with LBG staff, there was no needs analysis for the Road Safety component of the SICBP project. A senior USAID official told the Team that the concept for a Road Safety component came from the general knowledge that better roads lead to more accidents in developing countries due to the increased number and speed of vehicles on the roads. The SICBP program is currently defining the needs for a Road Safety program through analysis of traffic accident data, including the translation of Arabic language data, and development of a database. A former MTR senior official informed the team that the number of vehicles has grown from tens to thousands in just a few years. Therefore, MTR and USAID determined that it is vital to address road safety under SICBP. A senior MTR official informed the Evaluation Team that the Road Safety Directorate receives the lowest budget and has the least number of staff of all the directorates at the MTR. Focus of Road Safety Programs A senior MTR official told the team he expects the program to focus on all of Southern Sudan. However, all of the senior staff members interviewed at USAID and LBG think that the task focuses primarily on Juba Town. A senior USAID official also believes the program should focus at least in part on USAID road project investments. Relevance to Overall Project Goals and Objectives A senior USAID official stressed that the Road Traffic Act, which was developed under this task, addresses a lot more than just road safety. It is fundamental to the development of the sector. He also stated that the Road Safety component does not really provide capacity building, aside from the Road Traffic Act. The work on the Road Traffic Act has been completed. There is a similar and competing bill, which the Ministry of Internal Affairs presented, but at this point it has been left to MTR to decide how and when to present the bill. MTR officials agreed that the Ministry of Internal Affairs would definitely play a significant role in road safety. The SSRA will be involved in some areas of road safety when the program is established. One area would be incorporating road safety into road design. A senior MTR official told the Evaluation Team that safety design concerns are not currently coordinated well with the Directorate of Roads and Bridges. When interviewed, three mid-level staff at LBG ranked Road Safety as the least important component of SICBP and agreed that that this component did not fit well with the rest of the SICBP program. A senior LBG staff member agreed that capacity building in this area could be performed under another component of the SICBP project, without a separate Road Safety component. Value for Money A senior LBG staff member said that the Road Safety component provided the best value for money of the entire SICBP program. The budget for 2010 for this component is about $250,000. A senior USAID staff member stated that road safety is important and valid. It is also a high profile activity that involves the greater road user community. This individual would like to keep the Road Safety program within the SICBP program. However, he was also concerned that the scope of the activity may be growing beyond the original plans, pointing to the work on developing databases as an example. The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 15 THIS SECTION OF THE REPORT ADDRESSES THE FOLLOWING EVALUATION QUESTIONS: 1.h Is the IT solution for communication among states and Juba scaled appropriately? 1.h.i Is it sustainable and/or cost-effective? The Evaluation Team met with three individuals at LBG who are currently working on the Road Safety component. Two individuals were working on collecting, translating and analyzing traffic data and the Policy Advisor was working on the Road Traffic Act. The Public Affairs Officer was also working on developing a road safety awareness campaign. The entire safety office at MTR, including non-technical staff (e.g., drivers) is less than 30 people, according to the organization chart provided in the April 2010 Monthly Report. Despite the number of staff, MTR’s Directorate of Road Safety thinks there is no one experienced in Road Safety at LBG. Conclusions No needs analysis appears to have been performed prior to developing the scope and budget for the Road Safety task; rather, the needs were defined based upon anecdotal evidence. These needs are being redefined through a research effort to gather statistics, develop a database, etc. Based upon the description of the general needs identified, the scope of the activity seems to have grown beyond an appropriate level. However, a formal needs analysis may determine that the scope is too small to address the road safety issues for Southern Sudan. There are also other stakeholders, such as the Ministry of Internal Affairs, with whom a larger program should work. There is no clearly defined scope for the Road Safety component, either geographically (Juba versus country-wide) or by road type (trunk, town, feeder). While road safety is important, many of the issues are not directly relevant to the primary objective of developing institutions for road infrastructure in Southern Sudan. The task does not align well with the other objectives of SICBP, aside from the drafting of the Road Traffic Act, which is completed, and therefore this component detracts from the focus on core objectives. Recommendations  SICBP and USAID should clearly define the goals and objectives of the Road Safety component of the project (i.e., geographic focus, road type).  USAID should perform a formal needs analysis for Road Safety, as it is possible that the scope of the program should be much larger in order to address the road safety issues in Southern Sudan.  USAID should identify prospective donors or other USAID contract vehicles that could take over the Road Safety component. IT and VSAT Solution Findings Requirement for an IT and VSAT Solution A VSAT system for all ten states with accompanying IT equipment such as computers will be installed in the next month, according to a senior LBG staff member. Senior staff at USAID and LBG agreed that this system is very important to SICBP implementation and project goals and objectives, as it will significantly improve communications and coordination with all of the ten states, as well as governance through increased communications and coordination between GOSS and all of the states. The VSAT system was supposed to be operational by October 2009 according to the 2009 Work Plan, but is still not operational. All of the LBG senior staff interviewed agreed the delay is hampering the efficiency of the SICBP program. The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 16 Several state officials stated that there is currently very little communication and coordination between GOSS and the states. The states see this IT and VSAT system as critical to maintaining and improving roads, as seen in the results of a survey of 14 SMPI officials presented in Annex 10. However, during discussions with state officials, they demonstrated only a limited understanding of the capabilities and utility of the system. According to LBG senior staff, the existing VSAT system, which was supposed to be rehabilitated under SICBP, was not technically functional, particularly during bad weather. The recommended system is technically superior, and significantly more expensive. The original budget for rehabilitating the existing VSAT system was $120,000. The installation for the new system is now budgeted at $592,000. The new system is expected to have lower maintenance and broadband costs than the old system and the costs would mostly fall on MTR, which LBG considers to be more sustainable than having the states bear the costs as with the old system. A senior USAID official informed the Evaluation Team that, due to concerns about financial sustainability of the system, he had requested a sustainability plan from LBG around December 2009. USAID had not received the report to date. LBG confirmed receiving a verbal request, but not a written request. According to senior staff at LBG, the estimated cost for broadband is $3,000-4,000 per month and maintenance will be the responsibility of MTR after the first year of operation. According to a senior LBG staff member, the Ministry of Finance is supportive of the system and willing to pay for operations. However, an MTR senior official did not know what the annual costs would be and stated that there is no provision for the system in the 2011 budget and it would be difficult to get a budget for this activity in the future. One MTR technical staff member is being trained under the project to manage the system. However, a senior USAID official stated that outsourcing of maintenance should be considered in the sustainability analysis. A senior LBG manager also noted that other facilities, such as the SSRA facility generator, have not been maintained or financed, despite a memorandum of understanding (MOU) assigning responsibility to MTR. LBG plans to sign an MOU with MTR for operation of the VSAT system when SICBP hands it over. LBG senior staff agreed that there is a need to increase awareness of the capabilities and costs of this system. However, there are no specific plans for a forum to discuss the benefits or long-term financing of the system. Conclusions An appropriate VSAT system is critical to the functioning of a road infrastructure regime and the scale of the system to cover all ten states is appropriate to facilitate communications between SICBP and the states, as well as between GOSS and the states. The technological approach seems to be feasible, particularly taking the recurring costs away from the states, thus avoiding the situation of the failed VSAT system. However, with no feasibility analysis, this element is difficult to assess. It is also difficult to assess the cost effectiveness without a feasibility and/or sustainability analysis on the new system. The increased cost of the new system (over $450,000) could have been used to rehabilitate the old system and pay for broadband costs for many years. It seems unlikely that GOSS/MTR will have funds to maintain the VSAT system after the first year subscription expires, given the fact that other facilities such as the SSRA facility generator have not been maintained or financed despite an MOU assigning responsibility to MTR. The system will not be sustainable unless MTR and the states clearly understand the responsibilities for upkeep and capabilities of the system for road infrastructure development. The presence of a failed VSAT system financed by USAID, as observed by the Evaluation Team at SMPIs in two states, should raise red flags regarding sustainability of the new system. Recommendations  SICBP should perform a sustainability and cost-benefit analysis for the VSAT system to demonstrate the advantages and prospective cost savings for communications with the states. The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 17 THIS SECTION OF THE REPORT ADDRESSES THE FOLLOWING EVALUATION QUESTIONS: 1.f The project has attempted to address capacity constraints for a diverse set of stakeholders, including local contractors, GOSS staff, and state officials. How well has this been done? 1.f.i Has this strategy been effective in terms of capacity built? 2. SICBP is a large, complex project with many activities and moving parts. Have the IPs succeeded in maintaining quality over the various pieces while maintaining production at an appropriate cost? 2.a How effective is the training? 2.a.i How highly do trainees rate the quality of the assistance they have had in terms of appropriateness, timeliness, and quality? 2.a.ii Are the right people being trained 2.a.iii Are they applying the training to their work?  SICBP should present the sustainability/cost-benefit analysis report to MTR and the Ministry of Finance in order to establish a basis for financing future costs to be borne by MTR.  USAID should try to ensure that MTR understands and is prepared to fulfill its responsibilities under any MOU.  SICBP should organize a meeting of the states and MTR to discuss the responsibilities for upkeep and to demonstrate and train the states in the use and capabilities of the system.  In line with recommendations in the Sustainability section of this report, more capacity building for the states is critical if USAID wants this investment to be sustainable. Capacity Building and Training Findings As part of its capacity building effort, SICBP has offered various types of training in the form of road Training Contracts, seminars and workshops, on-the-job training, training in road maintenance technology, and formal college undergraduate and graduate level classes. The training offered has been unique to each organization’s requirements, i.e., MTR, states, local contractors. The complete list of training courses offered and number of participants are provided in the annexes. Building the Capacity of GOSS/MTR and SMPIs There has been a lot of focus on training of MTR personnel, mainly because of the shortage of qualified people in the ministry and in part due to the failure to establish an SSRA. The training covers policy workshops, seminars, and management issues and, based upon interviews with MTR, USAID and LBG, is in line with the ministry’s requirements. The shortage of staff has, however, meant that many of the trainees have ended up taking multiple classes. SMPI officials consistently stated that they require more capacity building. There has only been limited training for SMPIs and very little hands-on training. For example, Kisii field supervision training did not happen last year. Overall, the Evaluation Team’s observations showed that LBG’s program management is very good in terms of administratively managing a diverse set of activities in an extremely challenging environment. But the breadth of the program comes at the expense of a focused program on the core mission of developing a road maintenance regime for Southern Sudan. There exists the possibility of too much capacity built within MTR in areas that should ultimately be the responsibility of the SSRA with no clear indication that the SSRA will be established. The Training Contracts could serve as an important vehicle for overall training; however, the contract vehicle and program decisions do not allow for SICBP to serve as a training tool for the states, MTR or the contractor. The documents do not create or imply any contractual relationships between MTR/SMPIs and the local contractors. The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 18 As such, the ability to involve MTR/SMPIs in the execution of the Training Contracts was hampered by the fact that they were not parties to the contract. LBG told the evaluators on several occasions that they could not share the contract documentation with MTR/SMPIs due to confidentiality issues. Certain sections of the contract documents are not synchronized and a few are either inapplicable or unenforceable under the circumstances. Examples include the following Part IIB Conditions of particular application clauses that are seemingly inappropriate or irrelevant:  # 8 (CPA to prepare all local contractor accounting books),  # 10 (local contractors have no right to sublet),  # 14 (Contractor – i.e., LBG – to indemnify sub-contractor),  # 15 (insurance limits and amounts),  # 20 (Sub-contractor to be an Internal Revenue Service entity),  # 30 (arbitrations under rules of American Arbitration Association), and  # 31 (laws of New Jersey to govern, with sub-contractor waiving rights to seek changes in venue). The USAID SICBP COTR told the team that he thought that the capacity and oversight for contracts given to the States under SICBP was not enough. Due to the policy of decentralization and state jurisdiction over roads not categorized as interstate or international, it is necessary to build that capacity in order to have a sustainable roads infrastructure regime. Training offered under SICBP to officials from GOSS/MTR has consisted of policy workshops, management skills, college training, and on-the-job training. Training for the states has primarily targeted supervisors to facilitate their oversight of the states’ road improvement contracts. The Evaluation Team found the following regarding the training:  MTR is satisfied with the training given to policy makers and lab technicians.  The training is offered in different locations inside and outside Sudan.  After completion of the training, state supervisors in the project areas the Evaluation Team visited were either not supervising road maintenance under SICBP or were unclear about their role in the project.  Logistics is a huge constraint.  MDTF officials feel more staff education is necessary to promote team building at MTR.  The Asset Management System requires data collection, but at a huge cost (one estimate was $3 million). The system is more than 6 months behind schedule and still requires data collection on Central Equatoria for Beta Testing.  In Western Equatoria State, it was said that the state supervisors are on site only about ten times in a 6- month period.  LBG reported that there are still several major constraints to capacity building at the state level, including factors such as, the inability of the states to hire and retain competent staff, flaws in states’ staffing procedures, and lack of support for the SMPI at the highest level, all of which weaken the training efforts.  With a few exceptions, MTR, the states and contractors are sending people in positions that will benefit from the training. Building the Capacity of Local Contractors A total of 30 local contractors in two groups have undertaken a seven-week course in labor-based technology at the Kisii Training Center. The seven weeks consisted of four weeks of theory (road technology, contract management and business management) and three of field work. The first group, which consisted of 13 contractors, followed this training with three weeks of on-the-job training at the Yei Training Facility in Juba, after which they were given a one-ton pedestrian roller, steel culvert molds and signed an initial road improvement or maintenance contract of up to $90,000 each. Although these are called contracts, the entire process is still an extension of the training; hence, being called Training Contracts. The second group, which consisted of 17 contractors, graduated from the three￾week on-the-job training on June 19, 2010 and has not yet been awarded Training Contracts. The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 19 A total of 24 local contractors completed the Evaluation Team’s survey questionnaires. Here are the findings based upon this survey  Over 75 percent of local contractors surveyed agree that the training they received was relevant and meets a need that their company was lacking.  Over 75 percent of trainees surveyed see the lack of training in mechanized road construction as the major shortfall in the training curriculum.  The satisfaction level is high with this aspect of the program and there appeared to be consensus among USAID, LBG, MTR, and SMPI that it is the most successful task under SICBP.  There was a divided opinion among SMPI officials regarding sustainability of equipment-based technology, as seen in the chart in the annex.  Contractors would like basic bridge construction and rehabilitation training. Additional key findings:  In 2009, a total of 92 kilometers of roads were awarded at a cost of $679,287, which works out at $7,384 per kilometer.  The first Training Contract was negotiated with each contractor (sole source). However, subsequent contracts are bid competitively.  The contracts signed are a combination of the International Federation of Consulting Engineers (FIDIC) short form, the Federal Acquisition Regulations (FAR), and LBG contracting provisions.  One contractor in Rumbek, which the Evaluation Team observed during a site visit, was not following the contract, yet expected payment for work done outside the contract activities.  It costs $50,000 for the two trainers from Kisii to travel for 5 weeks to the states for follow up with the trained contractors.  One of the contractors visited complained about productivity using labor-based technology. He observed that gravel excavated manually over one month can only fill one pothole.  Heavy trucks use the repaired roads. The WES commissioner estimated that 20 to 30 trucks use the Gangura –Nabiapai road per day.  Because of logistical problems, most of the contractors participating in the program are from East, Central and Western Equatoria States. Many of the other states are under-represented, and it is difficult to get contractors from there. However, LBG has gone to great lengths to encourage state participation, contractor training and Training Contracts in all ten states.  Newspaper advertising was the primary means of identifying prospective contractor trainees, yet the national newspapers do not reach some of the states.  A large number of contracts began during the rainy season, making the work more difficult and increasing problems with the road during improvement. Conclusions Because the design assumption was that the SSRA would be operational within the first year of this project, its delay has had effects on various components of the design. The project now has to reconsider how it approaches capacity building in GOSS/MTR, since this component is expected to transition to the Road Authority. Though the capacity building has had a measure of success, there continues to be debate about the justification for continued funding of building the capacity of a department with no guarantee that the people being trained will be playing the same role in the Road Authority, or if they will even be there at all. The program has not been so successful at the state level because the states’ importance in planning, funding and executing feeder roads programs was not adequately addressed in the project’s design. Consequently, less capacity building was envisioned for the states and the implementing partners have excluded them from critical steps in the procurement process. The most successful component of SICBP has been building the capacity of local contractors, although a senior LBG staff member believes that it is still too early for the program graduates to take on large projects (more than one million dollars). The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 20 IN ADDITION TO THE ABOVE QUESTIONS, THIS SECTION OF THE REPORT ADDRESSES THE FOLLOWING EVALUATION QUESTIONS: 2.b Do the roads being built meet standard specifications? 2.c Do they meet appropriate environmental standards? 4. SICBP developed contract management and administrative systems and is engaged in a significant level of procurement and oversight. Are these systems working in a way that is effective, transparent, and designed to minimize corruption? The use of FAR and LBG provisions, as special provisions attached to the FIDIC contract form is a problem that needs to be addressed, as some of their special provisions are not applicable to SICBP. The start of Training Contracts does not consider weather impacts on construction. Follow-up supervision of trained contractors is inadequate. States are not adequately involved in the contract procurement process. There are unnecessary hurdles in contract costing (e.g., ten percent performance bond, payment process). Recommendations  A simpler and more appropriate contract vehicle that takes into account the legal requirements of each party to the contract needs to be identified and used instead of the current forms. (Possible options include a purchase order mechanism).  The training curriculum should be broadened in order to train the local contractors with a demonstrated financial and technical capacity in trunk road maintenance, which should include training in use and handling of mechanized equipment.  The curriculum should also be expanded to include planning and financial management to help stem cash flow concerns. Bridge construction and rehabilitation should also be added.  SICBP should schedule more regular follow-up supervision by site engineers and Kisii instructors.  SICBP should make every effort to schedule Training Contracts to coincide with favorable weather.  The project should build the states’ capacity in the procurement process through participation in contract oversight.  The project should continue its efforts in lowering bond requirements.  SICBP should streamline the contractor payment process. Findings The road specifications are part of the contract given to all of the Training Contractors and are based upon similar specifications in Uganda and Kenya. There is a danger that the repaired roads could still deteriorate due to the following:  Contractors use a one-metric-ton roller to compact the road base and wearing course.  Many commercial vehicles use the improved roads, which surpasses the design axle load limits of the roads, with some exceeding ten metric tons.  Contractors are not regularly supervised during implementation of their contracts.  State officials think roads are too narrow and should be wider.  Tree stumps were not cleared from road shoulders in old contracts, which are now a problem. Environmental Standards It was not clear what environmental standards were applied during road designs. GOSS/MTR is said to have its own standards, and believes that the project is in compliance. Based upon visual observation, the project took into The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 21 consideration issues like adequate and proper disposal of storm-water runoff and made efforts to prevent erosion. Borrow pits have, however, not been covered. The contractor in Yambio stated this was because they were expecting to use the borrow pits when they come back for the maintenance part of the contract. LBG reported that environmental mitigation was not a line item in the current Bills of Quantities (BOQs), though it will be in the budget for the next tranche of contracts. The project’s site engineers have received environmental training for road construction, sponsored by USAID. LBG officials believe the environmental component is becoming more formalized as the contracts go forward. Procurement LBG does not involve the states in the procurement process and, therefore, there has been little opportunity for corruption on the part of the state officials. MTR, however, participates in all aspects of tendering. Currently, the contractors have to obtain the forms from MTR, which is a cumbersome and inconvenient process. Twenty two contracts have been let so far. Of these, ten have been negotiated and 12 competitively bid between trained contractors. The bid contracts were for the newly graduated contractors. Though the system in MTR is somewhat different (the officials acknowledge that at times, “political consideration” goes into determining how the contract is awarded), these GOSS procedures cannot be used to reverse the bid process under SICBP. Conclusions According to the contract documents, the roads meet standard specifications as used in the East and Central African region. In the absence of continuous daily supervision on the site to monitor construction, however, there is no guarantee that specifications were followed through the entire process. Based upon the Evaluation Team’s observations, the roads do generally meet the specifications, although the erosion control mechanisms are inadequate at some culvert head-walls and wing-walls. Elements such as the contractor’s compaction of the gravel wearing courses can only be inferred. Lack of axle load enforcement negatively impacts longevity of improved roads. The building of this capacity of local contractors to build roads that meet specifications, and the success of the LBG engineers in overseeing the Training Contracts, are significant accomplishments of the project. Recommendations  Either now or sometime in the future, USAID or other donors should consider building GOSS capacity to establish and maintain an axle load enforcement mechanism to minimize the deterioration of repaired roads.  The project should consider going through a formal environmental analysis, especially for the trunk roads.  SICBP should help build the capacity of states in the procurement process through their participation.  SICBP should continue working on lowering bond requirements and streamlining the payment process as a way of helping the new contractors. The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 22 THIS SECTION OF THE REPORT ADDRESSES THE FOLLOWING EVALUATION QUESTIONS: 1.c How sustainable are the efforts under SICBP (and its predecessor project)? 6. The GOSS relies on SICBP for support on a number of levels at the same time that SICBP is trying to encourage its government partners to become self-sufficient. How well is LBG balancing these competing pressures? 10. What evidence is there that state, regional (Southern Sudan), and national institutions necessary for good and effective implementation of transportation infrastructure development activities in Southern Sudan have been strengthened? 10.a How sustainable are any notable improvements? Sustainability of SICBP Efforts Findings MTR Sustainability According to senior staff at LBG, USAID and the World Bank, as well as several technical staff at MTR, the efforts to date to establish a functioning roads system in Southern Sudan are not sustainable without the establishment of the SSRA. There was agreement among senior staff at USAID, LBG and MTR that through SICBP and SIS-CBC training, certain units within MTR have definitely built the technical capacity to continue functioning in the absence of SICBP or a similar program. One example was laboratory testing for roads construction, in which MTR staff received hands-on training and have been applying the training in laboratories established through the project. MTR officials have also worked closely with SIS-CBC and SICBP program staff and expert consultants to draft the National Transport Policy and the Roads Authority bill, which demonstrates capacity in policymaking, a key function of the MTR after the formation of the SSRA. During a group interview with a wide range of MTR officials, several MTR staff gave examples of “on the job” training during SIS-CBC and SICBP for project units that oversee MDTF-funded contracts through WFP. They noted that the contracts were let using standards and specifications developed and taught through SICBP training. Inspectors trained at the Uganda National Road Authority (UNRA) are now supervising those WFP projects. However, the WFP, not MTR, manages those projects and there were no examples of road improvement or maintenance contracts that were let directly by MTR in a manner consistent with the procurement processes in which MTR was trained through the SICBP project. According to a senior USAID official, there have been hundreds of millions of dollars of contracts let by MTR on a sole source, negotiated basis with no transparency. Three junior MTR officials confirmed this behavior. There is currently a nonoperational road asset management database that is still being developed and the beta test should be launched in the next few months. Senior staff at USAID, LBG, and MTR agree that this database is critical to sustainable road infrastructure planning. Regarding financial sustainability of the sector, an MTR senior official confirmed that the 2011 GOSS budget being developed does not have a road maintenance line item. Senior officials at USAID and MTR said that USAID would like to see $20 million budgeted for SSRA in the GOSS 2011 budget, or at least some financial commitment to Road Maintenance. A senior MTR official responded that while GOSS can earmark funding for a line item such as road maintenance, it will not happen. Under the current budget process, MTR receives a ceiling amount of funding from the Ministry of Finance and all of the money is likely to go to rehabilitation and construction of roads. Regarding the transition of MTR to self sufficiency, a senior LBG staff member stated that there is no plan for transitioning responsibility to MTR so that LBG can take more of a facilitator/advisor role versus the leading role in SICBP activities. LBG does most of the implementation of SICBP programming. A practical example is the operation of the SSRA facility, which has been handed over to MTR and yet there was no maintenance done on the generator, which LBG eventually repaired and maintained using SICBP funds. All of the LBG senior staff The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 23 interviewed agreed that there does not appear to be a sense of ownership of the project within MTR. A senior LBG manager said that it would be helpful if USAID could raise the issue of responsibilities for program oversight as well as facilities with senior MTR officials. He stated that the most important impact of MTR failure to take on facilities management and oversight is that SICBP funding needs to be diverted from other parts of the program to cover these responsibilities. Sustainability at the State Level According to all of the senior staff interviewed at LBG and USAID and a senior MTR official (and confirmed by the 2009 and 2010 SICBP Work Plans), the states, by design, were only given limited training and capacity building under SICBP. The reason is that the states’ capacity to plan and implement road infrastructure projects is extremely limited and the limited resources available under the program are not sufficient to build the capabilities of the states to the same level as MTR or the SSRA. The observations and interviews with state DGs and Directors of Roads and Bridges officials during field visits to Yambio and Rumbek confirmed that there is limited capacity within the states. In Rumbek, the supervisors were only recently trained at Kisii and had the same amount of experience as the contractor. The DG in Rumbek told the Evaluation Team that other supervisors were receiving on-the-job training with large international contractors, but also had limited capabilities in site supervision. In Yambio, the contractors did not feel that the supervisors had the capacity to supervise their work adequately. USAID and LBG stated that by design, the states are not given the capacity to oversee the contracts, primarily because of confidentiality issues between LBG and its subcontractors and the concern that state officials will demand payments from the contractors. A senior SICBP manager stated that, “State involvement in Training Contracts is almost incidental.” State capacity building has been mostly limited to supervisor training, workshops and short-term training in Juba. Kisii trainers are traveling to five states to provide on-the-job training in June/July 2010, which will be the first time they will travel outside of Juba to facilitate Training Contract site supervision. According to a senior USAID official, the states’ training primarily came about due to the fact that a vacuum existed from the absence of an established SSRA, which was supposed to be receiving extensive training under SICBP. The states are receiving a number of tools under the SICBP and SIS-CBC programs, including trucks, computer equipment, digital cameras, and VSAT systems. According to an MOU signed by the SMPIs, SMPIs are supposed to use the trucks for road planning, construction and maintenance. All of the LBG senior staff interviewed said that the states were using the tools given them, including the trucks and the standard documents. The DG of Roads and Bridges for Lakes State confirmed this information. However, there was at least one incidence of misuse of the vehicle in Western Equatoria State, where, according to a senior SICBP manager, the Minister of Physical Infrastructure was using the truck and not allowing the Directorate for Roads and Bridges to use it. The Minister handed back the vehicle upon hearing of an upcoming site visit from SICBP. Regarding financial sustainability of the states to plan, build, and maintain road infrastructure, all of the 14 state officials interviewed stated that they had not received any of the 1.5 million Sudanese pounds that were supposed to have been transferred from MTR to the states for road projects. Several of them had not heard of the proposed funds transfer. The DG for Roads and Bridges in Lakes State told the Evaluation Team that his Directorate had not prepared a budget or work plan since receiving training from SICBP in Juba. In Western Equatoria, when asked whether there was any plan to maintain the roads built through SICBP, the SMPI DG for Roads and Bridge’s answered no. The SICBP deliverables under this task did not include development of these work plans and budgets; only the training. Local Contractors’ Sustainability According to a senior USAID official, there was a pool of regional contractors that developed their capacity during SIS-CBC, including Civicon, a Kenyan company that had no road construction experience prior to its work in Southern Sudan under USAID and MDTF contracts. The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 24 According to a senior SICBP manager, 6 of the 13 labor-based contractors trained under SIS-CBC and SICBP are capable of independently bidding on and managing road construction projects and 4 of the labor-based contractors have received additional road contracts outside of the SICBP or SIS-CBC projects. However, 4 of the 13 contractors interviewed noted the lack of available contracts as the biggest constraint on contractors gaining more experience. All of the LBG senior staff interviewed confirmed this scarcity as a major constraint and noted that SICBP has supported contractors to secure additional work on an ad hoc basis, such as working with MDTF to set aside contracts for labor-based contractors. According to the World Bank/MDTF representative interviewed, the MDTF has allocated $2.2 million for 222 kilometers of roads (9 contracts) for labor-based contractors to be let in July-August 2010. Conclusions There is evidence that Southern Sudan institutions, necessary for good and effective implementation of transportation infrastructure development activities in Southern Sudan, have been strengthened, particularly MTR. In its current state, the efforts associated with Task 1 (Assistance to MTR and Establishment of Roads Authority) and Task 2 (Capacity Building for State Ministries of Physical Infrastructure) are not sustainable. The SSRA has not been established and is not operational and therefore the MTR is responsible for planning, budgeting, and managing road infrastructure. While several units have been established to perform this function, the work is largely performed by WFP, not the Ministry. MTR has received some training and capacity building in procurement, contracting and management of road infrastructure, but has not exercised that training. Therefore, it lacks the capacity to manage road infrastructure at this time. LBG is generally doing a good job of balancing the competing pressures of supporting GOSS on a number of levels and trying to encourage its government partners to become self-sufficient. Due to resource constraints, USAID has provided some of the tools for state road infrastructure development, but has not built the long-term capacity of the states under this program. The states do not have the human or financial resources to plan, budget and manage road infrastructure projects. The supervisors have limited capacity to supervise road construction and the Directorates of Roads do not have prepared work plans or available funding for road infrastructure. Based upon our observations in Lakes and Western Equatoria States and interviews with LBG, USAID and the states, providing basic tools such as, equipment and basic training without adequate capacity building is an unsustainable approach. It is highly unlikely that the tools given to the states will be utilized efficiently or effectively for the management of road infrastructure without significant additional resources for capacity building. SIS-CBC and SICBP have developed a pool of local and regional contractors that have the capacity to carry out road infrastructure improvement and maintenance, but there are limited contract opportunities for small, labor￾based contractors, which is the main hurdle to long-term sustainability of this part of the project. Financial sustainability at the GOSS level in the absence of an SSRA is unlikely. Financial sustainability at the state level for road infrastructure is also unlikely. Recommendations  If states remain a focus of the program, efforts should be made to maximize existing resources within SICBP (i.e., utilization of VSAT network, site engineer visits, Kisii Trainers, greater participation in Training Contracts) to build the capacity of states.  For USAID investments in the states to be sustainable, the SICBP project should direct more resources toward developing the states’ capacity to manage road infrastructure projects.  If it is decided that the states should not receive additional capacity building, USAID should not channel more resources into the states, as those investments will be unsustainable. The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 25 THIS SECTION OF THE REPORT ADDRESSES THE FOLLOWING EVALUATION QUESTIONS: 1.i Are there any gender issues that have been or should be addressed? 7. What gender issues have been addressed during the implementation of SICBP?  LBG and MTR should collaboratively develop a clearly defined transition plan for greater MTR responsibility for SICBP activities and facilities.  USAID should work with MTR to ensure that MTR follows through with all its previously agreed upon responsibilities.  SICBP and USAID should work closely with GOSS and other donors to develop innovative programs (e.g., local contractor set asides, subcontract requirements for larger road contracts, etc.) to promote and identify opportunities for local contractors. Gender Issues Findings According to the USAID/Sudan strategy statement, all projects need to ensure that women participate equitably in all USAID-funded activities. Gender is a theme that cuts across all strategic objectives. SICBP maintains records on the number of women trained or employed. The project documents have targets of number of women to be trained or employed, and there is some ad hoc pushing for more women participating in the program through hiring. There is, however, no available strategy for increasing women’s participation in the program. In interviews, the evaluators specifically asked the female members of the groups to identify any actual or perceived issues that may have negatively affected their participation in the program and the consensus was that any under￾representation of women in the pool of employees or trainees is part of the universal shortage of women in construction. In the road Training Contracts, up to 25 percent of the workers were women in some states, such as Western Equatoria, according to the contractors. Conclusions SICBP recognizes the importance of equitable participation of men and women in USAID programs. Women’s participation in the Training Contracts indirectly supports other USAID/Sudan objectives, like support for education and economic growth through increases in household income and single mothers having the chance to support their families. No specific gender issues were identified in the project. Recommendations  SICBP should continue using the USAID gender strategy6 to further encourage more participation of women in its programs. The project should utilize lessons learnt from regional, labor-based contractor programs that have been running much longer (i.e., in Uganda, Ethiopia and Kenya).  SICBP should identify ways that the program can assist women to take more advantage of the services that the roads facilitate, which would likely require synergy with other USAID or partner projects in the areas of health, microfinance and agriculture. 6 USAID/Sudan Strategy Statement 2006 – 2008, December 2005 (or as amended) Andrea Ariketa (pictured here with the Commissioner (left) and SMPI Director￾General (right) of Western Equatoria) is one of two women workers the evaluators spoke to. Both said they were the heads of their households and between them supported seven children, including paying for their education. The road contracts were their only source of income. The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 26 THIS SECTION OF THE REPORT ADDRESSES THE FOLLOWING EVALUATION QUESTION: 8. Assess SICBP communication and coordination with project partners. Communication and Coordination Findings There was general discontent from the MTR and states regarding communication with the project. While SICBP holds two, semi-annual work-planning sessions, there are no other scheduled meetings between the project and these project partners. One MTR senior official stated that he was at times bypassed in SICBP’s communication with the Ministry. The state officials interviewed reported that they neither received copies of the contractors’ Training Contracts from SICBP officials, nor were they adequately involved in the contract procurement process. However, they are supposed to supervise the contractors. The state officials generally feel they are out of the loop on contractor training. SICBP cited the need for maintaining confidentiality of financial information as their reason for keeping the states out of some stages of the procurement process. In addition to the above human factors, communication between SICBP and the states is further complicated by the logistics involved (roads, air-travel, etc.). The hope among all the stakeholders is that a functioning VSAT system will greatly improve communication with the states. Conclusions There is a need for better communication between SICBP and its project partners in order to achieve the project’s objectives. By keeping the states out of the procurement process, SICBP is working counter to strengthening their capacity to negotiate and manage contracts. More formal systems are required to build the communication channels between all parties involved. Recommendations  The project should work to improve communication with MTR and the states through regular monthly or quarterly meetings.  The project should maximize any potential advantages of using the VSAT system being installed to communicate with the states in real time, e.g., communications and meetings between MTR/GOSS and the states should be virtual using the new VSAT system rather than having state staff travel all the way to Juba.  If funds allow, MTR and state officials involved in procurement should receive training in procurement procedures as part of their capacity building.  Since the States will be responsible for all aspects of the management of feeder roads contracts after SICBP ends, concerns over maintenance of document confidentiality must be addressed through ethics training for SMPI staff. The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 27 THIS SECTION OF THE REPORT ADDRESSES THE FOLLOWING EVALUATION QUESTION: 11. What evidence is there to suggest that broader economic and social impacts are being achieved? 12. How is SICBP supporting other USAID projects and objectives? 12.a What has been the overall impact on governance and CPA support? USAID Southern Sudan Objectives Findings USAID Strategic Objectives The SICBP project was designed to address a number of USAID goals and objectives, including:  Intermediate Result 10.1: “Core Institutional Structures for an Effective, Transparent, and Accountable GOSS Developed” by providing equipment, training and technical assistance to the GOSS Ministry of Transportation and Roads, as well as to selected state ministries of transportation in Southern Sudan; and  Under the USAID/Sudan 2008 Operational Plan, Program Element 4.4 (Infrastructure) and specifically Program Sub-Element 4.4.3 (Transport Services). According to the Concept Note for the project, SICBP contributes to the USAID/Sudan Strategy by providing tangible peace dividends in the form of feeder road development, thus strengthening support for the CPA. According to interviews with state and county officials in Western Equatoria and Lakes States, roads are an important part of the peace dividend. According to the Commissioner for Yambio, roads are the number one priority for his constituency. He stated that nothing is possible without roads, particularly security, economic development and access to services. According to the USAID Southern Sudan Strategy Statement for 2005-2008 (the Evaluation Team was not provided a more recent strategy statement), USAID will support the reintegration of ex-combatants and those associated with fighting forces through programs focused on labor-based infrastructure. The labor-based contracting approach aligns with this strategic goal. In Lakes State, the DG for Roads and Bridges stated that young men are the biggest source of instability and that labor-based contracting helps diffuse potential security issues by providing work for this group. The Strategy also states that overall lack of security and rule of law are threats to implementation of the CPA, and USAID will work closely with other U.S. Government agencies that are addressing these issues. It also states that increased attacks by the Lord’s Resistance Army (LRA) in Southern Sudan are undermining previously stabile regions of Equatoria. During the site visit to Yambio, the Evaluation Team visited a road improved under the SICBP project that is the major route to the Democratic Republic of Congo and into territory where the LRA is active. The completed sections of road provide access to two military camps – Sudan People’s Liberation Army (SPLA) and Ugandan military – that are fighting against LRA in the area. In Rumbek, the SMPI DG for Roads and Bridges told the Evaluation Team that several of the road sections selected for a Training Contract were prioritized in order to facilitate security patrols of the area. Regarding the goal of facilitating unity within Southern Sudan, MTR and state officials noted that road infrastructure is critical to connecting the states, which helps bring greater economic and political interaction and leads to improved stability of Southern Sudan. In terms of governance, according to a senior MTR official, increased coordination between MTR and the states has led to better planning for road infrastructure, thus improving governance within Southern Sudan. The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 28 Other USAID Road Infrastructure Programs A senior USAID official stated that one of the primary objectives of the project is to ensure that there is a viable maintenance schedule in place to protect USAID, other donor and GOSS investments in road infrastructure. The project plays an important role in building the institutions and contractor capacity necessary for that maintenance. However, according to all the senior staff interviewed at USAID and LBG, there has been no effort to target USAID-funded roads for maintenance under the SICBP Training Contracts. USAID senior staff informed the Evaluation Team that a recently-funded USAID agriculture program for Southern Sudan focusing on the three Equatoria States will have a feeder road component. USAID senior staff told the Evaluation Team that there has already been some coordination between that program and SICBP regarding the feeder road component. There is a potential for funding to flow into SICBP, sharing of lessons learned, and use of labor-based contractors trained under SICBP. Economic and Social Impacts A majority of the 14 state officials interviewed identified improvements to security, getting produce to market (increasing food security), and access to services were as important results of improved roads. Several examples of economic and social impacts were identified during site visits. Interviews with laborers showed that a common use of wages was to pay for school fees for their children. One of the roads to be rehabilitated in the next 6 months will provide year-round passage to a recently renovated county hospital. According to the hospital administrator, the road is currently impassable several months of the year and it is difficult to get doctors to visit the hospital due to the travel time required. Several of the 14 state and county officials interviewed noted that labor-based contracting was important because it brought money directly to the populations around the roads. Two labor-based contractors in Yambio told the Evaluation Team that many of the laborers bought bicycles, the primary mode of transportation in the area, after being employed. All of the LBG SICBP senior staff interviewed agreed that the project’s performance indicators are not adequately capturing these economic and social benefits. The Public Relations Officer was only recently hired and has begun writing success stories to capture some of these benefits. Conclusions The SICBP program aligns well with USAID’s strategic objectives for Southern Sudan such as, increasing security, good governance, improving access to services, and food security. Roads are generally recognized as an important part of the peace dividend (i.e., improved security, access to markets and services) and at least some of the roads improved through the SIS-CBC and SICBP are perceived locally as part of the peace dividend. Roads improved or maintained under the project have provided access to services, increased economic activity, and provided increased security. However, based upon findings under the Sustainability section of this report, there is little evidence to suggest that broader economic and social impacts are being achieved throughout Southern Sudan. This shortfall is a result of the failure of the SSRA to be established and therefore the limited impact of the program on road infrastructure improvement and maintenance in the region. The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 29 THIS SECTION OF THE REPORT ADDRESSES THE FOLLOWING EVALUATION QUESTION: 3. Are there ways in which the effectiveness and efficiency of SICBP could be improved? 5. Are TO8 human resources sufficient to achieve the project’s goals? 13. Based on learning from the evaluation, is a shift in resource allocation among project components called for? Special attention should be given to the state level. SICBP Efficiency and Effectiveness Findings Based on interviews, the evaluation team found that the channels of communication between SICBP and GOSS/MTR lack clarity and there are feelings that there is not adequate sharing of information. The project has a set of indicators shown with the work plan. These indicators have been attached in the annexes. Responsibility for road construction and maintenance is shared between MTR and the states7 . MTR is responsible for trunk roads (international and interstate), while the states are responsible for the county, urban and feeder roads. The roads selected under SICBP fall into all these road classifications. The program’s capacity building approach, however, focuses more on working with MTR, and inclusion of the states was not one of the initial objectives. When it comes to the budget and funding, the states set their own priorities and MTR cannot determine how much they will allocate to feeder road maintenance. When it comes to road selection, there is a lack of information on the current condition of the road network. SICBP is working on establishing a Road Asset Management System (RAMS), which will play a pivotal role in future road maintenance planning8 . Save for support in training road supervisors at Kisii and at the Yei Road facility in Juba, the states receive much less capacity building than MTR. The two states visited feel left out in the decision making process and consequently do not fully consider SICBP road maintenance as their project. One consequence is that all SICBP Training Contractors find it difficult to rent equipment from the states if they want to perform any day-work items from the contract. There are also some problems in the road Training Contracts. Some of the problems encountered include the following:  While Training Contracts are mostly focused on labor-based technology, there are tasks for which mechanized equipment would be more efficient and could result in cost savings for the contractor. Examples include tree removal and borrow pit excavation.  Citing security concerns, the Western Equatoria SMPI was reluctant to release (for rent) equipment to work on a section of the Saura – Lirangu road that can only be fixed using mechanized equipment. On the other hand, the contractor in Lakes State informed the Evaluation Team that he has an arrangement with the SMPI, through which he can rent equipment for work on state roads at below-market rates.  In order to receive a $23,000 advance, one of the contractors said that he had to pay $18,000 in bonds and insurance, leaving him a net balance of $5,000 for mobilization. He lost more money when the bank that acted as its surety went bankrupt.  Logistical challenges, as mentioned in the section on capacity building, complicate processes as simple as negotiating a contract or exchanging documents. 7 Ministry of Transport & Roads, Government of Southern Sudan: Strategic Plan for Road Sector, July 2006 8 The Louis Berger Group: Sudan Infrastructure Services Project. Task Order 8 Capacity Building Program: 2010 Work Plan Narrative and Program Costs, November 2010 The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 30 Problems also exist within SICBP human resources, including the following:  Partial records which LBG provided (see Annex for schedule of site visits, August 2009 – May) indicate infrequent visits to the states, as shown in the table below. State Maximum Possible Visits Number of Actual Visits Recorded Central Equatoria 12 1 Western Equatoria 12 4 Jonglei 12 4 Eastern Equatoria 12 3 Upper Nile 12 4 Bahr el Ghazal 12 2 Lakes 12 1 *Based on two visits per month. Assumes continuous operations (design and maintenance) in state throughout period.  LBG attributes the infrequent visits to location of some projects in insecure areas, slow progress or lack of actual progress on the ground, the election process and travel or logistics restrictions.  Both LBG senior and technical staff stated that lack of formal channels of communication between management and professional staff has resulted in loss of staff morale. Loss of morale has resulted in discontent over training, professional development and security for local staff on field site visits outside Juba.  A senior USAID official was concerned that the USAID foreign service national (FSN) and third country national (TCN) engineers are stretched too thin in their work, given that the infrastructure budget is the largest in the Mission.  Due to changes in the LBG security policy which management announced in the last year, LBG site engineers are being asked more and more to travel with no security contingent. This change has been a challenge for LBG senior officials, as site engineers are very concerned about their personal safety. Resource Allocation Three of the four SICBP project tasks are closely connected: (1) creation of the SSRA and building capacity of MTR, (d) building capacity of the states and (3) building capacity of local contractors. There was debate between USAID and the implementing partners as to whether road safety fits this mould, much as there is overall agreement that road safety is a critical component of road use. A senior USAID official was more concerned about scope creep than resources spent on safety. Senior GOSS officials confessed that the road safety department has the lowest priority in the ministry. SICBP has worked with MTR to draft a traffic bill, though there have since been delays because of the need to reconcile the MTR bill with that prepared by the Police Traffic department, according to GOSS/MTR. Conclusions Based upon the findings, the Evaluation Team drew the following conclusions:  The primary objective of SICBP is capacity building – of MTR, the states and local contractors – and roads are improved or maintained simply as a by-product of this capacity building effort. So while keeping count of, for example, total length of roads improved or repaired is an easily measurable indicator, shifting the emphasis to an indicator that more directly measures contractor capacity built (e.g., number of local contractors trained, quality of local contractors trained, success rate of local contractors, etc.) would more appropriately match the project goals. The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 31  There is a strong possibility that at the end of the project, the states will have a cadre of technical people who can supervise road maintenance, and nothing else. More efforts are required to bring the states into claiming ownership of the project at the state and county levels.  Funds currently allocated to road safety would have a bigger impact if channeled to such capacity building efforts as additional training from Kisii, capacity building in the states, or additional site supervision.  There is not adequate support from the states to the local contractors.  SICBP has fostered a working relationship and built the capacity of MTR, making it a viable partner.  There is a preponderance of constraints that inhibit the successful execution of the Training Contracts that need to be addressed.  LBG needs to provide more supervision and mentoring for junior engineers.  As already discussed under capacity building and training, the SICBP project staff members are stretched thin and are not able to make biweekly visits to the projects. Senior supervision of the engineers is also inadequate.  The USAID staff supervising the project may also be too extended and require additional staff. Recommendations  In order to properly assess the success of Task 3 of this project for local contractor capacity building, SICBP needs to identify and use indicators that correctly measure capacity building of local contractors. Total length of road renovated or maintained may still be used, but it should not be the principal indicator.  SICBP should make a concerted and deliberate effort (within available funding) to identify and address the constraints to the successful operation of the Training Contracts. (Note: Since these are Training Contracts, some other recommendations appear under the Capacity Building section of this report.)  If funds are available, more resources should be provided to augment the states’ capacity building.  LBG should increase supervision of both the contractors and the junior engineers.  Much as there was no overall agreement on how road safety fits into the overall project objectives, the external evaluators’ view is that it detracts from the focus on the more critical aspects of this project. This recommendation is based upon the premise that another partner could be obtained to take on the road safety component. The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 32 LESSONS LEARNED The following are the main lessons learned from the design and implementation of SICBP: Problems and delays with formation of the SSRA highlight the need for projects centered on major reforms in legislative policy or procedures to consider beforehand alternative scenarios if the underlying assumptions are delayed or are not enacted during project execution. This planning should be in the form of an appropriate risk analysis with proposals for mitigation plans. Project design should assess and adjust the contribution of each task to a project’s overall objectives, and allocate the resources accordingly. The need for road safety in the absence of any road safety regime and growth in road traffic gave way to a disproportionate amount of resources being focused on this area of the program. Even in exceptionally challenging areas such as Southern Sudan, programs should strive to maintain rigorous standards for project implementation. In the case of this project, there were several cases where those standards (e.g., road selection criteria, quality of finished roads) were relaxed, resulting in a less￾focused program that was less likely to achieve its key goals and objectives. A project’s performance and teamwork are enhanced when all stakeholders clearly understand and buy into the overall project objectives. While this project’s primary objective is capacity building, there was frequent dissatisfaction among stakeholders at the GOSS/MTR and state levels who mistakenly believed “road rehabilitation” was the major objective. It is important to use all avenues to maintain the focus of all stakeholders on the key objectives of the project. In the SICBP case,  The more frequent use of the term “Task Order 8” resulted in loss of focus on key project components such as, creation of the Road Authority, training, capacity building, road maintenance, and local contractor training, and their importance to the project’s success.  Preference should be for performance indicators that are directly correlated with project objectives, e.g., “number of contractors trained” provides a metric that is more in line with the project objectives (i.e., how much contractor capacity has been built) than “total kilometers repaired.” Project design should review the roles and responsibilities of all key players and streamline their contribution to the overall goals. While the main SICBP implementing partner, GOSS/MTR, is responsible for all trunk roads, feeder road selection and maintenance is the purview of the SMPI. This mismatch between the Ministry and states often becomes a source of confusion and misunderstanding during implementation of the project. New projects should build on previous projects as much as possible. The desperate need for all types of road development in the states and the desire to build political relationships between GOSS and the states gave way to putting road selection in the hands of the states versus managing that process to focus more on maintaining the trunk roads already built under previous projects. One effect of this loss of focus has been to increase the pool of roads that will require maintenance. Expectations and terms of service of local staff of implementing partners need to be clear to all parties. Many local project staff felt that not being able to access the training available to GOSS project staff places them at a competitive disadvantage in a future SSRA. The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 33 ANNEX I – SCOPE OF WORK 1. Program to be Evaluated Program Identification: “Sudan Infrastructure Capacity Building Program” (SICBP) Implemented as Task Order #8 under the Sudan Infrastructure Services Project: USAID IQC Contract No. 650-1- 00-06-00010-00, also implemented by Louis Berger Group, Inc. Program Funding: $13,850,000 Program Beginning/End dates: November 3, 2008 – November 2, 2011 Key Agreement/Contract Modifications: Modifications: None Implementing Partners(s): Prime: Louis Berger Group, Inc. (LBG) Kenya Ministry of Roads and Public Works, Kisii Training Center Kenya Institute of Management Center for Enterprise Development Africa Expedition (AFEX) USAID/Sudan Technical Office: Economic Growth COTR: Boutros Magaya 2. Evaluation Purpose This will be a formative mid-term evaluation, slightly past one-half the way through the second three-year phase of the project. Its principle purpose will be to review project design and assumptions to determine if they remain valid and to provide information to help the Implementing Partners and USAID modify implementation to improve potential impact. The evaluation will also assess impact to date and gather lessons learned/success stories. The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 34 3. Background A. Country Context Sudan is the largest country in Africa, borders 9 countries, and has a population estimated at 40 million. Since independence in 1956, Sudan has suffered from civil war, with only a decade of troubled peace from 1972 to 1983. Southern Sudan and the critical border areas (consisting of the northern states of Southern Kordofan and Blue Nile, plus Abyei – a commonly referred to as the Three Areas) are characterized by years of underdevelopment, war, famine, drought and flood, producing a crisis of enormous proportions across the region and resulting in the devastation of economic, political and social structures. In addition to the loss of lives, opportunities and infrastructure, the war displaced families and divided communities. In consequence, the health, education and infrastructure status of the Sudanese people are among the poorest globally. After decades of civil war, Sudan's warring parties signed a Comprehensive Peace Agreement (CPA) in January of 2005. Since that time the country has taken steps toward peace, reconciliation and good governance, although the pace has been slower than expected or desired. Despite the signing of the CPA, Sudan remains a vulnerable state. Its children, many of whom are orphans, returning refugees and ex-combatants, are particularly at risk - especially in the "hot spots" of the Three Areas. It is essential that displaced and other affected people, particularly orphans and ex-combatant youth, be safely reintegrated into their communities. In the case of the youth, affected by the many conflicts and tensions during the past 21 years, the provision of basic education is critical to providing a solid foundation upon which their future success and contribution to society can be based. The provision of education can also be seen as a tangible result of the "peace dividends" expected by Sudanese citizens and, in turn, will contribute to stabilization in the region. Durable stability is contingent upon demonstrative and observable change "on the ground" and education, highly valued by the Sudanese, is both a necessary and visible symbol of that change. In many areas, education services are largely provided by GOSS and state Ministries of Education, while health services are primarily provided by non-governmental organizations (NGOs). In both sectors, GOSS has received considerable support from development partners offering humanitarian relief including, NGOs, faith-based organizations (FBOs), and multilateral and bilateral aid agencies. As peace is consolidated, USAID will continue to support a responsible transition from emergency to development assistance that seeks to improve access to and quality of basic education. Education and health activities are reinforced by investment in other essential services, such as WASH (water, sanitation and hygiene) and livelihoods, in an effort to rebuild local communities, reduce tensions, and provide the much sought-after peace dividends. B. Sector Context Infrastructure throughout Southern Sudan was destroyed during years of civil conflict. There is an acute lack of financial and institutional capacity to provide road maintenance services on the part of both the public and private sectors. Lack of physical infrastructure and the physical isolation of southern Sudanese communities constrain economic growth, make it difficult for vulnerable populations to establish sustainable livelihoods, feed ethnic conflict, and constrain provision of critical governance, economic, and social services. Roads and transport infrastructure in particular are essential to provide security and employment opportunities and to facilitate peaceful reintegration of returnees into society. It is crucial to upgrade the existing roads to all-weather maintainable gravel surface conditions that GOSS and local communities could maintain in order to provide safe and affordable services. At a very basic level, it is very hard to imagine vibrant economic activity throughout Southern Sudan expanding over time without a strong all-weather road networks. Thus, improved transport infrastructure is critical to rebuilding areas of Sudan affected by civil war. This is acute in both Southern Sudan and the Three Areas, where infrastructure was destroyed or deteriorated during the war, institutions disappeared, and human resources not renewed. The state of road infrastructure is deplorable and in dire need of urgent repair or rehabilitation. Roads and transport infrastructure in particular is essential to provide security and to facilitate peaceful reintegration of returnees into society The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 35 A major obstacle for the development of infrastructure in Southern Sudan is the lack of technical skills, lack of institutional capacity to develop and manage infrastructure services, fragile legal framework, and inexperienced public and private sectors. The Ministry of Transport and Roads (MTR) lacks critical technical and managerial capacities. Institution building is needed across the board. Considerable assistance is required to establish policy making capabilities, conducting the necessary research and analysis, producing policy guidelines in the infrastructure sector. In 2005, USAID Sudan contracted with the Louis Berger Group (LBG) to implement a three-year Sudan Infrastructure Program (SIP), Capacity Building Component (CBC). CBC provided initial capacity building support to the GOSS Ministry of Transport and Roads (MTR). Under the SIP/CBC, LBG provided technical assistance and training services to develop the GOSS Transport Sector Policy, refurbish offices, procure supplies and equipment, and provide technical assistance and training to GOSS/MTR staff. Two very significant policy/institutional constraints that remain are the lack of a Southern Sudan Roads Authority to effectively oversee roads operations up to contemporary standards and a financing mechanism, such as a Roads Fund, to increase the financial sustainability of investments in roads infrastructure. These have been under discussion for years, but have yet to be realized. C. Program Description SICBP is designed based on the SIP/CBC experience. It is targeted to improve approximately 1,000 km of roadway in nine States of Southern Sudan while improving the capacity of local contractors. Skills training of local construction contractors and state officials combined with ‘learn-by-doing’ Training Contracts are employed to build capacity. Training contracts are only executed with local contractors after they successfully complete an intensive skills training program. To jump-start efforts during the first year, local contractors which previously participated in the CBC training program and which performed successfully were offered another ‘Training Contract’ for a longer and more challenging section of roadway. While the first batch of contractors were completing their Training Contracts, SICBP identified new local contractors to participate in the program, identified new road sections for rehabilitation in consultation with state officials, and conducted surveys and prepared BOQs for new road sections. SICBP involves four task areas:  Training of private Sudanese construction contractors to undertake approximately 1000 km of labor-based road improvement and maintenance works using ‘Training Contracts’ on selected road sections in nine States of Southern Sudan;  Training, capacity building, and support of the Ministry of Transport and Roads, to include institution building, transport policy, and strengthening of planning, dissemination and implementation capabilities with secondary emphasis on the Directorate of Air Transport9 ;  Training, capacity building, and technical support of the Southern Sudan Roads Agency (SSRA); and  Training, capacity building and support of nine state Ministries of Physical Infrastructure. D. Linkage to USAID/Sudan Strategy and USG Foreign Assistance Framework SICBC contributes to the USAID/Sudan Strategy by providing tangible peace dividends in the form of feeder road development, thus strengthening support for the Comprehensive Peace Agreement. It also directly addresses IR 10.1: “Core Institutional Structures for an Effective, Transparent, and Accountable GOSS Developed” by providing equipment, training and technical assistance to the GOSS Ministry of Transportation and Roads, as well as to selected state ministries of transportation in Southern Sudan. Under the USAID/Sudan 2008 Operational Plan, this activity addresses Program Element 4.4 (Infrastructure) and specifically Program Sub-Element 4.4.3 (Transport Services). 9 Work with the Directorate of Air Transport has been removed from the scope of SICBP. The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 36 E. Project Strategic Summary SICBP is expected to result in the following:  Creating employment;  Building capacity of Sudanese contracting firms;  Providing peace dividends in the form of infrastructure development10;  Improving access by road, linking Kenya and Uganda to Sudan, Southern Sudan to Northern Sudan and the River Nile to the road network11;  Reducing cost of access to food and food production; and  Stimulating and modernizing the transportation sector. SICBP will achieve these results through the following principle objective:  Build local, state, regional (Southern Sudan), and national institutions necessary for good and effective implementation of transportation infrastructure development activities in southern Sudan. This principle objective will be accomplished by working in the following task areas:  Training private Sudanese construction contractors to undertake approximately 1000 km of labor-based road improvement and maintenance works using ‘Training Contracts’ on selected road sections in nine states of the ten southern Sudan;  Training, capacity building, and support of the Ministry of Transport and Roads, to include institution building, transport policy development, and strengthening planning, dissemination and implementation capabilities, with a secondary emphasis on the Directorate of Air Transport12;  Training, capacity building, and technical support of the Southern Sudan Roads Agency (SSRA); and  Training, capacity building, and support of nine of the ten southern state Ministries of Physical Infrastructure. F. Geographic Orientation The project plans to work with the following companies in the following states on the following roads: State Company Road Eastern Equatoria Eco-Builders, Ltd. Torit – Katire Eastern Equatoria Botimore & Co., Ltd. Torit – Katire Western Equatoria Card, Ltd. Yambio – Nabiapai Western Equatoria Gbudue Construction Association Yambio – Nabiapai Jonglei Rhino Star Construction Jallei – Maar 10 This statement is included in the Mission's 2008 Operational Program. The new strategy is more focused on "development and economic growth" and less on the Fragile States strategy. 11 Early on, it was thought that the Nile River transport would be more important than it turned out to be. Therefore, it was mentioned in the Goals Statement, but SICBC never developed any activities in this area. 12 There have been no activities related to Directorate of Air Transport. It was decided that this would dilute the SISCB focus on roads as there weren't enough resources to do both roads and air. The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 37 State Company Road Upper Nile Okoj Engineering Malakal – Khorfulus Eastern Equatoria TBD Torit – Magwi Torit – Lafoni Western Equatoria TBD Maridi – Tareka Yambio – Sakure Jonglei TBD Bor – Anyidi Lakes TBD Rumbek – Pacoug Western Bahr el Ghazal TBD Wau Northern Bahr el Ghazal TBD Aweil Warrab TBD Kwajok Lakes TBD Rumbek 4. Available Information to Support the Evaluation The following information will be provided to the evaluation team in advance of its arrival in Juba. 1. Task Order proposal [Need a formal approval document for Task Order/work ] including aggregated budget; 2. All available performance monitoring data as of the most recent available date (FY09 PPR) 3. Monthly Progress reports [Need Apr-09; Oct-09; Nov-09; Dec-09; Jan-10] 4. Quarterly and Annual reports for SICBP and predecessor project SIP/CBC 5. 2009 and 2010 Work Plan 6. Final report from the predecessor project, SIP/CBC 7. Annual reports from predecessor project, SIP/CBC 8. Various analyses by the project 9. Activity Approval Document for SICBP 10. Environmental compliance documentation (IEE for entire project and any ERFs for specific roads) 11. EG PMP including the USAID Strategic Framework for Sector (note the latter is currently being updated) 12. USAID/Sudan Strategy 2006 strategy which has been extended as modified by annual operational plans. 13. MSI Evaluation and Special Study Guide 14. USAID Evaluation Standards and Guidelines for Foreign Assistance 15. USAID HICD policy and toolkit. The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 38 5. Evaluation Focus and Questions The evaluation will focus mainly on design and implementation issues, with secondary emphasis on impact, while also attempting to extract targeted lessons learned to be shared. Impact issues can be considered in light of both the relatively short period of the SICBP, as well as the prior three years of its predecessor project, SIP/CBC. The following questions must be addressed by the evaluation team, in light of the purpose described above: Project Design 1. Are there any issues with respect to project design and assumptions (documented or implied) that should be reconsidered based on experience to date? a. An operating SSRA (Southern Sudan Roads Authority) has been considered a critical institutional piece for a functioning roads system. Can the project have long-term impact in the absence of a SSRA? i. What is the likelihood of the SSRA becoming a reality? b. Medium- and long-term financial sustainability would seem to be dependent on an operational financing mechanism, such as a Roads Fund. What is the likelihood of this becoming a reality? i. What are USAID and GOSS options in the absence of a functioning financing mechanism or to develop one? c. How sustainable are the efforts under SICBP (and its predecessor project)? d. Many individuals are working for LBG under SICBP with the understanding that they will transition to government after the SICBP. How will those individuals be retained and operate after this project? i. Is there potential for this to be a successful approach? e. How were the roads under TO 8 selected? What criteria (if any) were used? f. The project has attempted to address capacity constraints for a diverse set of stakeholders, including local contractors, GOSS staff, and state officials. How has this been done? i. Has this strategy been effective in terms of capacity built? g. Is the scope of the road safety program appropriate based on the needs identified? i. Should it be expanded or reduced? h. Is the IT solution for communication among states and Juba scaled appropriately? i. Is it sustainable and/or cost-effective? i. Are there any gender issues that have been or should be addressed? Project Implementation 2. SICBP is a large, complex project with many activities and moving parts. Have the IPs succeeded in maintaining quality over the various pieces while maintaining production at an appropriate cost? The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 39 a. How effective is the training? i. How highly do trainees rate the quality of the assistance they have had in terms of appropriateness, timeliness and quality? ii. Are the right people being trained? iii. Are they applying the training to their work? iv. Does training offered meet needs identified in the Training Needs Assessment and by the Ministry? b. Do the roads meet standard specifications? c. Do they meet appropriate environmental standards? 3. Are there ways in which the effectiveness and efficiency of SICBP could be improved? 4. TO 8 developed contract management and administrative systems and is engaged in a significant level of procurement and oversight. Are these systems working in a way that is effective, transparent, and designed to minimize corruption? 5. Are TO8 human resources sufficient to achieve the project’s goals? 6. The GOSS relies on SICBP for support on a number of levels at the same time that SICBP is encouraging its government partners to become self-sufficient. How well is LBG balancing these competing pressures? 7. What gender issues have been addressed during the implementation of SICBP? 8. Please assess TO8 communication and coordination with project partners. Project Impact to Date 9. Is there evidence that local contractor capacity has been improved? a. How broadly and how equitably among states? 10. What evidence is there that state, regional (Southern Sudan), and national institutions necessary for good and effective implementation of transportation infrastructure development activities in Southern Sudan have been strengthened? b. How sustainable are any notable improvements? 11. What evidence is there to suggest that broader economic and social impacts are being achieved? Lessons Learned/Success Stories 12. How is SICBP supporting other USAID projects and objectives? a. What has been the overall impact on governance and CPA support? 13. Based on learning from the evaluation, is a shift in resource allocation among project components called for? Special attention should be given to the state level. The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 40 14. What capacity building approaches appear to be particularly productive? 6. Evaluation Methods and Procedures The External Evaluators will be provided the information provided in Section 3, above, before arriving in Sudan. They will be expected to be familiar with this information prior to arriving in Juba. A Team Planning Meeting (TPM) will be held upon arrival in Juba to agree on how team members will work together, how they will interact with the client and other stakeholders, and to develop a work plan and finalize a Travel Schedule. The team will need to visit project site(s) so some transportation will be arranged prior to the team’s arrival. The team will most likely travel to sites in Central Equatoria, Western Equatoria, and Western Bahr el Ghazal (i.e. Wau). During the TPM the team will finalize the methodology to be used and produce the evaluative instruments to be employed. The team will use the “Getting to Answers” approach detailed in Annex II of the MSI Evaluation and Special Study Guide to develop detailed methodological approaches to meeting the terms of this Scope of Work. USAID Evaluation Guidelines and Standards for Foreign Assistance will also be utilized in developing methodology, reviewing data collection procedures, and reporting formats for the results. The team will finalize its methodology during the TPM. But we expect that in addition to basing the evaluation’s findings on interviews and review of project documents, the team will also consider using the following simple approaches:  Development of an interview guide to ensure that the correct evaluation questions are being addressed the appropriate individuals and that they are being posed and recorded consistently.  Key informant interviews with individuals who participated in the training.  A customer satisfaction survey of contractors assisted by SICBP.  Consistent and systematic approach to assessing road quality and environmental compliance;  Framework for detailing capacity strengthening needs and the degree to which those needs are met by the project.  Framework for detailing training provided and the degree to which it is addressing capacity gaps.  A group meeting or focus group of local contractors assisted by the project to be held in Juba.  Indicate other basic methodological approaches appropriate to the task. Once the methodology has been finalized at the TPM it will shared with USAID as part of the work plan approval process. 7. Team Composition and Participation Team Composition USAID/Sudan is conducting the Mid-Term Review in a collaborative manner to maximize USAID, GOSS and Implementing Partners learning opportunities. Accordingly, the team will be comprised as follows:  Two External Evaluators (skill sets detailed below), provided by MSI  One representative of USAID  One representative of GOSS, most likely from Ministry of Transport and Roads The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 41  One representative of Implementing Partners Additional inputs may come from other staff from these agencies, as needed, and as coordinated by the respective team member. USAID’s representative may be a person from the Economic Growth Team. GOSS and the Implementing Partners (IPs) may choose their representatives as they see fit, but persons selected should have experience with similar programs in Sudan. Given the significant contributions to the team expected from each team member, all are expected to be available to participate throughout the evaluation period. Team Member Roles and Responsibilities USAID, GOSS, and IP team members will provide historical, contextual and programmatic background information that will inform the assessment. They will be expected to participate in the Team Planning Meeting (TPM), field visits, interviews, brainstorming on Findings, Conclusions, and Recommendations, and in the frequent reflections on evaluation learning, often occurring after a long day of interviews and traveling. These individuals participate as representatives of their respective organizations and are expected to share their learning with their home organizations so that all three key organizations are kept abreast of progress. It may well happen that the External Evaluators will ask USAID, GOSS, or IP representatives to be excluded from certain portions of interviews in order to ensure candid responses. The External Evaluators will take the lead in conducting the evaluation, leading interviews, framing the analysis, facilitating group discussion and consensus, preparing for the debriefing, and drafting the evaluation report. One of the External Evaluators will serve as the overall Evaluation Team Leader. The Evaluation Team Leader will take full responsibility for managing the team, organizing its work, and ensuring quality control and delivery of a final report acceptable to USAID. Precise division of labor among the two External Evaluators will be determined at the TPM. Between the two External Evaluators, the following capacities must be brought to the team: 1. Strong skills in assessment and analysis of USAID projects, especially with economic growth and infrastructure programs; 2. Extensive experience working in East Africa, Sudan, and/or similar post conflict environments; 3. Facilitation experience, experience leading participatory evaluations, or at least evaluations where evaluation teams include critical stakeholders as active participants; and 4. Experience arranging meetings, setting up travel schedules for field visits, reporting on meeting outcomes, and generally managing the logistics of the evaluation (although significant logistical assistance will be provided by the SUPPORT team in Juba). 5. Experience in implementing or evaluating the following: a. Roads construction b. Institutional capacity building c. Training d. Managing large project in remote areas e. Anti-corruption f. Policy reform g. Information technology The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 42 The Team Leader will be the formal representative of the team and will arrange for updates regarding progress against the evaluation work plan to the COTR (or his/her delegate) and MSI’s Chief of Party (COP) or Assessment, Monitoring and Evaluation Specialist (AME), as determined at the TPM. 8. Activities, Logistics, and Timing Prior to arriving in Juba, the External Evaluators will have familiarized themselves with the background material provided to them, as referenced in Section 3, above. All team members should be present for the TPM and for initial briefings and discussions with USAID’s Economic Growth Office and other Mission officers, as well as IP and GOSS officials. A Work Plan and travel program for the in-country visit as well as the subsequent report writing period will be submitted to USAID for approval during the first few days of work in Juba. The Work Plan will also include a schedule for periodic MSI and USAID progress reports and possible submissions of specific work products, as determined by the parties. Approximately four days prior to departure the Evaluation Team will present to USAID, Implementing Partners, and the GOSS an out-briefing, with succinct supporting documents. The Draft Evaluation Report will be submitted prior to the External Evaluators’ departure from Juba. The Mission and the IP will each submit its comments on the draft report within ten work days of receipt the draft report. The Draft Final Report will be submitted to USAID ten work days after the Team Leader’s receipt of USAID’s and the IP’s final written comments on the draft. It is envisioned that all External Evaluators will be in Sudan the entire duration of the evaluation’s in-country component (six-day work weeks are authorized), including the TPM, a debriefing, and submission of a draft report to MSI’s COP or AME prior to departure from Sudan. In addition to travel days, additional days are provided for the External Evaluators to complete reading and processing all background information prior to departure for Sudan. Additional days are provided to finalize the report. (See table in Section 9, below.) MSI’s field office in Juba will be responsible for travel arrangements (travel, housing in the field, etc.) for the USAID and GOSS team members. MSI will fund travel-related costs for GOSS team member(s), but not for IP or USAID team member(s).13 MSI and the Implementing Partners will jointly arrange all meetings for the team, in coordination with GOSS. The team will be provided office and meeting space, as needed, at SUPPORT’s Juba Office Compound. 9. Projected Level of Effort (LOE) and Timeline Tasks (both external evaluators, unless otherwise noted) Work Days (six-day weeks in Sudan, five￾day weeks outside Sudan) Initial Preparation Review advance background documents and SUPPORT Project’s Evaluation and Special Study Guide, make travel preparations and travel days to Juba. 5 In-Country Evaluation Initial briefings, meetings, field visits, draft report preparation, and debriefings. 20 13 If the USAID representative is an Institutionally-Contracted Staff member provided by MSI, his/her travel costs will be provided by MSI separately. The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 43 Tasks (both external evaluators, unless otherwise noted) Work Days (six-day weeks in Sudan, five￾day weeks outside Sudan) Return Travel 2 Final Report Preparation in Home Country Incorporate collective Sudan feedback, complete final report, and submit to MSI. 1 each, plus 3 additional for team leader Total for each Evaluation Team Member 28 Total for Evaluation Team Leader (3 additional days) 31 10. Report Production and Format The team will present for approval by USAID a draft outline of the report during its first week in country. The evaluation report must:  Distinguish clearly between findings, conclusions (based strictly on findings) and recommendations (based clearly on the report’s findings and conclusions);  Comply with all instructions of the SUPPORT Project’s “Evaluation/Special Study Quality Management Guide” and meet the specific requirements of the “Evaluation Report Review – Score Sheet”, contained therein;  Include a Table of Contents; a list or acronyms, an Executive Summary of no more than three pages; a section describing the project to be evaluated and purpose of the evaluation; a section on the methodology employed, including relevant skill sets of the evaluators;  Include any annexes the team considers useful to the reader; and  A copy of this SOW as an Annex. A formal debriefing will be provided to USAID, the IP and the GOSS, as scheduled during the TPM and recorded in the evaluation work plan. The team will present key Findings, Conclusions and Recommendations for comment from the stakeholders. The team will record all relevant feedback from the meeting and will respond to all comments in completing its draft reports. The External Evaluators need not include all suggestions in the report, but must consider such suggestions in finalizing the Draft Report. An electronic (in MS Word) version of the Draft Report will be presented to the IP and USAID in Juba with four hard copies being provided to the USAID/Sudan Mission and one hard copy to the IP prior to the departure of the Team Leader. The document will not exceed 30 pages, excluding annexes and Executive Summary. The Mission and the IP will each submit its respective comments on the draft report electronically to MSI’s COP – using the “track changes” and “comments” functions in MS WORD as much as possible. Each organization will combine internal comments, resulting in a unified set of comments from USAID and a unified set of comments from the IP. The Mission will receive ten paper copies of the final report as well as an electronic version, once the Mission has accepted the product. 11. Deliverables  A draft work plan, ensuring that all aspects of Getting to Answers (from the TPM) are addressed  A schedule of travel and key activities The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 44  Interim progress briefings to MSI and the Mission, as determined during the TPM  Preliminary report outline  Draft Findings, Conclusions and Recommendations to MSI prior to completion of the first Draft Report  Out-briefing, with supporting documents  Draft report  Final report 12. Compliance to USAID Regulations The Evaluation Team will ensure that the evaluation is fully compliant with the terms for Project Evaluations contained in the USAID Automated Directives System (ADS) Series 203 and other relevant regulatory requirements, as may be determined by USAID. Additionally, the Team will utilize MSI’s SUPPORT Project’s “Evaluation/Special Study Quality Management Guide” as well as the USAID Evaluation Guidelines and Standards for Foreign Assistance. These guides will be presented to the Team members prior to their initial TPM. The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 45 ANNEX II – COMMENTS AND RESPONSES ON DRAFT REPORT Comments & Responses on Draft Report Mid-term Evaluation of TO8 - Sudan Infrastructure Capacity Building Project Rec No. TO8 Mid-term Evaluation Conclusions TO8 Mid-term Evaluation Recommendations USAID Comments LBG Comments Evaluation Team Response 1 Establishment of a Southern Sudan Road Authority (SSRA) Based on the findings, the Evaluation Team concluded that the establishment of the SSRA is one of the highest priorities of the SICBP project. Based on discussions with stakeholders, the Evaluation Team determined that it is most likely that delays in the establishment of an SSRA are indicative of political will. There is only a small window of opportunity for passage before GOSS becomes focused on the Referendum in January 2011 and the Evaluation Team concludes that it is unlikely that the SSRA will be established until after the Referendum. While capacity has been built within MTR and the States, in the absence of an SSRA, SICBP would have minimal long-term impact on the establishment of a functioning roads system. SICBP should utilize some of the resources targeted at the establishment of an SSRA to develop an approach for actively promoting the establishment of an SSRA. In concert with the SICBP promotion activity, USAID senior management should make a concerted effort over the next few months (July-August 2010) to push for the establishment of the SSRA, perhaps through a coordinated effort with other donors. SICBP should develop a contingency plan for reallocating resources in case the SSRA is not established in the next three months. 1) We strongly agree that SICBP should develop a contingency plan for reallocating resources if SSRA is not established before 30 Sep 2010. 2) However, it is not clear what is meant by "utilize some of the resources … for actively promoting establishment of SSRA". TO8 has already spent over $300,000 to promote the establishment of SSRA, including regional study tours for senior GOSS officials and 3 inclusive workshops to sensitize GOSS officials and stakeholders. The Transport Policy was presented to the past parliament and approved. 1) TO8 is actively engaged in facilitating movement of the Bill through the MoLACD. The MTR legal advisor is working closely with the Director of Legislation at MoLACD in finalizing Bill for presentation to the Council of Ministries. 2) USAID is meeting with the Minister of Transport and Roads currently scheduled for the 4th of August. 3) TO8 is providing an optional activity plan for the MTR Directorate of Roads and Bridges as a way forward to improve efficiency in the Roads and Bridges Directorate to improve efficiency and sustainability of managing GoSS road networks after our program ends. USAID comments noted and recommendation in report adjusted for clarification. LBG comments noted. The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 46 2 Establishment of a Roads Fund Based on the findings, it appears that the Roads Fund is unlikely to move forward in the near future due to the fact that it is less of a priority than the SSRA and can be phased in over time. While a Roads Fund is the optimal long￾term financing mechanism, it is not necessary so long as the SSRA is established, since direct transfers are possible and likely to come from GOSS and/or donors. Recommendation: The SICBP project should continue to focus on the establishment of an SSRA but not on a Roads Fund for the near future. We agree that establishment of SSRA is the priority. Noted and agreed Noted. No action necessary. 3 Sustainability of SICBP Efforts There is evidence that institutions necessary for good and effective implementation of transportation infrastructure development activities have been strengthened, particularly MTR. However, financial sustainability for road infrastructure at the GOSS and State levels in the absence of an SSRA is unlikely. Due to resource constraints, USAID has provided some of the tools for state road infrastructure development but has not built the long-term capacity of the states under this program. They do not have the human or financial resources to plan, budget, and manage road infrastructure projects. The SICBP approach of providing basic tools such as equipment and basic training without adequate capacity building is an unsustainable approach. CBC and SICBP have developed a pool of local and regional contractors that have the capacity to carry out road If states remain a focus of the program, efforts should be made to maximize existing resources within SICBP to build their capacity. If it is decided that the states should not receive additional capacity building, USAID should not channel additional resources (e.g., equipment) into the states. SICBP and USAID should work closely with GOSS and other donors to develop programs to promote local contractors and identify and develop opportunities for them. This could include small business or local contractor set-asides, or subcontract requirements for larger road contracts. 1) Establishing a capacity for State Ministries of Physical Infrastructure (SMPIs) to independently manage road maintenance programs is not an objective of TO8. But since TO8 is providing valuable and needed support to the SMPIs, it should be continued, and therefore the objectives and activities for strengthening the SMPI should be better articulated and focused. 2) We agree with the second recommendation that GOSS and other donors seek ways to identfy and develop more opportunities for the local contractors trained under TO8. This is already happening to a small degree, but TO8 can do more to promote the local contractors. Noted and agreed Comments in body of report: Response: TO8 agrees with this recommendation and agrees the scope of the contracts. TO8 must provide this scope in better terms of understanding by LBG Contracts Managers. The essence of the training is to instill methodologies including implementation and management of the projects as a learning guide. Under CBC the roads were implemented using routine progress payments which did not allow for developing roads using standard construction criteria. Many of the roads completed were not successful in delivering a satisfactory deliverable. Under TO8 all roads are measure and pay. Nonconformance letters are issued for defective products and the contractors have to USAID comments: TO8 has provided training in tendering, procurement and project management to SMPI staff. Agree that the objectives and activities for strengthening the SMPI should be better articulated and focused. The states play a critical role in routine feeder roads maintenance, yet their needs are of such magnitude that the current TO8 approach is unlikely to maximize impact on the program's sustainability. Agree with comment #2. LBG comments noted. Report not revised. The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 47 infrastructure improvement and maintenance but there are limited contract opportunities for small labor based contractors. This is a major hurdle to the long-term sustainability of this activity. correct these items prior to pay. This necessitates additional oversight of the projects. Efforts to bring this constraint forward are challenging. Senior LBG staff are learning the scope of the contract training however and TO8 will continue to push the value of the training over simplistic issues, but straightforward, of implementing subcontracts. 4 Transitioning from SICBP to GOSS/SSRA The general assumption among stakeholders was that SICBP staff would transition to the SSRA when the new body is created. However, in the absence of regular and continued supervision from more senior engineers, as well as additional training, capacity building of SICBP staff will not be adequate for their successful transition. There does not appear to be a long-term strategy for training of the SICBP professional staff, and the evaluators found no evidence that an assessment of key skills needed by engineers had been undertaken. SICBP should increase supervision by senior engineers over junior engineers. SICBP should develop a long-term strategy for training of the SICBP professional staff in skills aligned with technical requirements of SSRA. This may require SICBP working with USAID to find appropriate vehicles for such training. The transitioning of TO8 staff to SSRA should begin as soon as GOSS decides to establish SSRA. We think that if GOSS fails to make or delays that decision, then TO8 resources should not be used to continue training LBG TO 8 engineers in the hope that there will be a future institution and positions for them. 1) TO8 currently has a Senior Engineer providing oversight of Junior Engineers. In addition, the Deputy Task Order 8 Manager schedules a significant part of his time in developing computer skills such as MS Excel for the engineers. 2) the work that the engineers are conducting is capacity building in a the area of supervisory oversight of road projects. Engineer’s exposure to practical on￾going work develops their ability to understand all the intricacies of implementing road projects. Computer skills in a formal setting are being planned in CAD development. That plan has not been approved by USAID. Agree with USAID comments. However, if TO8 continues under the premise that the SSRA will be established, as it has since its inception, we recommend that training of TO8 engineers should proceed per our recommendation. No action necessary. LBG comments are noted for the record, as they were taken into consideration in drawing up report conclusions and recommendations. The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 48 5 Road Selection The Evaluation Team concluded that the SICBP roads selection criteria are inadequate. States were given the responsibility to select roads and it appears that they generally made good choices based on their own priorities. However, road selection for training contracts does not align with the overall SICBP objective to improve road maintenance of road infrastructure investments by USAID, GOSS, and other donors. If the proper road sections are selected which have not deteriorated significantly, labor based contractors should be able to maintain these roads. Selection of feeder roads is not supported by other SICBP tasks since there are not enough resources to build state capacity and MTR/SSRA does not oversee feeder roads. Recommendations: SICBP should consider selecting more trunk roads for training contracts in order to: establish local contractor capacity on trunk roads; give MTR and/or SSRA experience maintaining roads within its jurisdiction; and protect USAID, GOSS and other donor investments in road infrastructure. SICBP should play a more active role in road selection based on the overall project goals and objectives. If states are left to select the roads for Training Contracts, SICBP should work with them to utilize the tools they have been given (e.g. developing work plans) to select priority roads. The Road Asset Management system, if/when operational, should also be used to select roads for Training Contracts. 1) We agree with the recommendation that the local contractors be trained to carryout road maintenance on primary roads that have recently been upgraded and rehabilitated through investments by donors and GOSS. This may require additional training especially in the use of graders and rollers to maintain the road surfaces. 2) We feel that TO8 contractor training funds should not be used by SMPIs to construct/rehabiliate feeder and urban roads that are outside the responsibility of MTR/SRRA except in exceptional circumstances. However, SMPIs are encouraged to use their own funds and/or other donor funds to hire the local contractors trained under TO8. 1) Noted. Some contracts are developed for work on trunk roads. Forty kilometers of road improvements were completed last year and maintenance contracts are being let for this year. 2) road selection is a multiple step process. Some roads are political in nature and some are economically driven. Roads of all types are good for developing the skills of the contractors in improving and maintaining roads. Rejection of road candidates is based on difficulty of the road candidates depending on complexity of the proposed project and or use of mechanized equipment is needed to implement the project. Comments in body of report - see below Agree with USAID comment #1. Regarding USAID comment #2, our overall recommendation is that SMPIs should be given a greater role in the SICBP implementation, including road selection. However, if that recommendation is not implemented we agree with USAID comment #2. LBG comments support our observations that the individual parties generally made good choices based on their own priorities, and that road selection for training contracts does not align with the overall SICBP objective to improve road maintenance of road infrastructure investments by USAID, GOSS, and other donors. The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 49 Comments in body of report 3. Response: most of the roads were selected in coordination with the State’s MPI’s as noted in follow on statements included below. On trunk roads, consultation and approval was granted by MTR for work on the Juba￾Yei road km 10-50. Coordination of efforts also included meetings conducted with MTR and WFP in determination of planned projects. The types of roads varied and include trunk, feeder and urban road. Each utilizes basic skills of the contractors in starting their work with the basics of L￾based road implementation. Advanced development of contractors comes with experience gained through road projects. Some of the contractors are more aggressive and have moved forward in tendering additional work proposals and are arguably satisfactory road constructors in a National sense. Others will require more experience working on smaller contracts. Not all will succeed in maturing to the level of National or International contractor. Same response as above. The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 50 6 Road Safety There does not appear to have been a thorough needs analysis performed prior to developing the scope and budget for the task. Based on the description of the general needs identified, the scope seems to have grown beyond an appropriate level. However, a formal needs analysis may determine that the scope is too small to address the Road Safety issues for Southern Sudan. Aside from the drafting of the Road Traffic bill, which is completed, this activity does not align well with the other objectives of the program and detracts from the focus on core objectives. While Road Safety is important, many of the issues are not directly relevant to the primary objective of developing institutions for road infrastructure in Southern Sudan. Given the small number of financial and human resources available and the large number of components within the SICBP program, this activity should be reduced or eliminated if an alternative funding or program vehicle can be identified. We agree that the Road Safety activity should be reduced and re-scoped. The original intent was to provide public awareness to communities and road users along the Juba-Nimule Road. The drafting and enactment of the Road Traffic Act should continue to be supported; but this activity is much broader than just road safety. Noted but disagree. Funding minimal and supports activities that make a positive impact in road usage and concern for pedestrians alike. Road safety is a basic component of good highway design. Road safety is essential in developing the regulatory aspects of road design and construction. The efforts of the TO8 team in recognizing interventions to develop road safety awareness is a small part of the overall budget and has been successful in bringing forward challenges within MTR in understanding the consequences that poor planning has on road usage. Road safety is a major design aspect of road projects for development of passive and non-passive highway furniture. Other ministries are involved in road safety from the enforcement side but not the regulatory side owned by the MTR. Currently TO8 has two short term staff working on this aspect; both have advanced degrees in engineering and teaching, respectively. An impact on the overall budget developed for the roads tasks is also minimal. Agree with USAID comments. We agree with LBG comment that road safety is a basic component of good highway design. Our conclusion however is that, just like drainage, alignment, or any other item, safety during development of highway furniture should be an integral component of road design and not a stand-alone feature. Support to the Traffic Act covers the regulatory component. TO8 was to then have some targeted focus on public awareness. Broadening into the larger aspects of traffic, while necessary, may not be the most effective use of TO8's limited resources. No action necessary. The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 51 7 Information Technology and VSAT Solution An appropriate VSAT system is critical to the functioning of a road infrastructure regime. The technological approach by SICBP could be feasible, however, with no feasibility analysis this is difficult to assess. The system will not be sustainable unless there is a clear understanding by MTR and the states of the responsibilities for upkeep and capabilities of the system for road infrastructure development. SICBP should perform a sustainability analysis for the system, to be presented to MTR and the Ministry of Finance, including a cost-benefit analysis to demonstrate advantages of the system and prospective cost savings for travel and communications with the States. I agree that LBG should prepare the sustainability analysis that was requested by USAID in Dec 2009. Both financial and technical sustainability should be adressed in the analysis. Noted and agreed Agree with all comments. No action necessary. 8 Capacity Building Though the capacity building at MTR has had a measure of success, there continues to be debate about the justification for continued funding of capacity-building for a department with no guarantee that the people being trained will be playing the same role in the Road Authority. The program has not been so successful at the State level, and this is because the state’s importance in planning, funding and executing feeder roads programs was not adequately addressed in the project’s design. Consequently, the States were excluded from critical steps of the procurement process. The most successful component of TO8 has been building the capacity of local contractors. However, it is still too early for the program graduates to take on large projects (over one million dollars). ..................According to the LBG should develop an appropriate contract vehicle which takes into account the fact that this is a training program. Possible options include a purchase order mechanism. Training Contractors should be given training in use and handling of mechanized equipment in order to be able to manage trunk road maintenance and to better manage feeder road improvement contracts. The curriculum should be expanded to include more planning and financial management and bridge construction and rehabilitation. LBG should schedule more regular follow￾up supervision by site engineers and Kisii instructors. SICBP should help build the capacity of states in procurement through participation in the procurement of Training 1) We agree that LBG should look into simplifying the training contracts; however, it isn't clear to me how the complexity of the existing contracts have hindered the local contractors other than on the issue of performance guarantees. 2) While we agree that local contractors should receive additional training in road maintenance and management of mechanized equipment, the training should only be provided to local contractors with a demonstrated financial and technical capacity. 3) We also agree with the conclusions on bridge maintenance a well as the need for TO8 to do more on-the￾job follow-up and training for the local contractors. 4) We do not agree that SMPIs should participate in the procurement of TO8 Training Contracts because the logistics of their participation is prohibitive. Response - Executive summary: Noted and agreed. In the interim, TO8 will use the LBG contracting platform for training and advanced contracts. USAID comment 1: In addition to the financial hindrances, the contracts are not providing the desired training in contract interpretation and compliance that is imperative for the success of any contractor. Sections of the contract documents are either inappropriate or un￾enforceable. LBG also finds it difficult sharing the contract as it is now with the SMPI due to concerns about confidentiality. USAID comment 4: TO8, MTR and SMPI will have to directly address and reach mutual agreement on the role of states in procurement as this was a point of displeasure that created some degree of ambivalence towards the program in the states we visited. Unless they The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 52 contract documents, the roads meet standard specifications as used in the East and Central African region. In the absence of continuous daily supervision on the site to monitor construction, however, there is no guarantee that specifications were followed through the entire process. Lack of axle load controls negatively impacts longevity of improved roads. Contracts. GOSS and States haven't yet given the State Road Departments the authority, guidance and documents they require in order to do their own procurements. buy into the program, the states will continue to think this is not fully their project and its final sucess at state level will remain questionable. Agree with comments 2 & 3. The report has been revised to incorporate USAID comments 1 & 4. Cont'd… Cont'd… - Response - Main report: Response 1: TO8 agrees on measuring the quantitative success of improving and maintaining roads. During initial evaluation of the indicators it was noted that the budgeted costs for this task amounted to a little over $3000/km. In order to focus on this work plan task, TO8 has provided initiative in reducing scopes to continue the goal of training contractors in the implementation of road maintenance and improvement projects within the scope and budget of the contract. Currently, TO8 has reduced the cost of road improvement contracts to less than $8000/km and road maintenance contracts to a cost of less than $5000/km. Though this constrains the efforts to improve and maintain roads the lessons learned and results indicate Noted. No action necessary. The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 53 that the contractors are learning the basic guiding principals in implementing road infrastructure projects on various types of roads including trunk and feeder roads. Cont'd… Cont'd… - Response 2: Additional effort in state’s capacity building is fundamentally sound but problematic. Efforts to conduct technical training for states' roads supervisors and directors of roads and bridges should continue. A major constraint in this effort is still the ability of the states to hire and retain competent staff. Most of the State’s staff are appointed based on who they know, family connections, etc. Inadequate support at the highest level of the State’s MPI weakens the efforts for training. For TO8 the recent elections created instability of the States to continue work programs resulting in delays and changes in project design at the SMPI level. Noted - report observations revised accordingly. The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 54 Cont'd… Cont'd… Response 3: Part 1. Supervision of contractors is provided by TO8. TO8 recognizes that the contractors require more oversight for some of them and less for others. In 2009, TO8 was directly involved in the management of the two contractors we had in Torit as they were unsatisfactory performance implementers. With the assistance from TO8 the contractors successfully completed their respective 10 km and currently (July 28th) the roads are very serviceable. TO8 also provides adequate coverage of road projects. The major constraint for any road supervision is linked to travel and lodging costs and security. TO8 realigned the program budget to increase funding to improve the field costs and TO8 is looking for other methods of embedding engineers regionally to reduce lodging and per diem costs. These LBG comments are noted for the record, as they were taken into consideration in drawing up draft report conclusions and recommendations. The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 55 Cont'd… Cont'd… - Response 3: Part 2. There is agreement that formal training of junior engineers is different than training conducted at the MTR Director and above level. Training conducted by TO8 has been focused on supervisory and managerial training versus technical skills learning for engineers. TO8 has developed a plan to provide CAD training for select engineers (Dep. Dir., etc.) and submitted to USAID for approval. TO8 engineers are developing field supervisory skills but would like more formal training and essentially preparing them with better managerial skill sets than we can offer MTR. Carryover into the planned RA will not only look at the education background but also the field skills learned. They should be more confident and competent in their ability to supervise road projects. These LBG comments are noted for the record, as they were taken into consideration in drawing up draft report conclusions and recommendations. Also see USAID comments on training of TO8 staff. The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 56 9 Gender Issues The SICBP recognizes the importance of equitable participation of men and women in USAID programs. However, no specific gender strategy was developed for the project. SICBP should develop a pro￾active strategy for including women in labor-based contracting. This should be based on research from regional labor-based programs (i.e., Uganda, Ethiopia, and Kenya). This requires expansion of monitoring of gender issues beyond simply employing women or training women contractors. The question therefore becomes how TO8 can do more to assist women to take more advantage of the services that the roads facilitate. This would likely require synergy with other USAID or partner projects in the areas of health, micro-finance and agriculture. Noted. Noted. Report revised to incorporate USAID comments. 10 Communication and Coordination There is a need for better communication between SICBP and its project partners in order to achieve the project’s objectives. The project should work to improve communication with MTR and the states through regular monthly or quarterly meetings. The project should maximize any potential advantages of using the VSAT system to communicate with the states in real time. We agree, but think that communications and meetings between MTR/GOSS and the States should be virtual using the new VSAT system rather than States staff traveling all the eway to Juba Response: noted. Currently the only scheduled meetings are with the UN and the MTR at the currenly monthly T&D meetings. These meetings address the work ongoing with donor and implementing partners. States representation is not included but can be reviewed for inclusion. Noted and agree. Report revised to more specifically clarify recommendation. The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 57 11 USAID Southern Sudan Objectives The SICBP program aligns well with USAID’s strategic objectives for Southern Sudan, such as increasing security, good governance, improving access to services, and food security. Roads are generally recognized as an important part of the peace dividend (i.e., improved security, access to markets and services) and at least some of the roads improved through the CBC and SICBP are perceived locally as a part of the peace dividend. Roads improved or maintained under the project have provided access to services, increased economic activity, and provided increased security. However, based on findings under the Sustainability Section of this report, there is little evidence to suggest that broader economic and social impacts are being achieved throughout Southern Sudan as a result of the project. This is due to the failure of the SSRA to be established and therefore the limited impact of the program on road infrastructure improvement and maintenance in the region. We agree; no further comments Noted. No action necessary. 12 SICBP Efficiency and Effectiveness There is a strong possibility that at the end of the project the states will have a cadre of technical people who can supervise road maintenance and nothing else. More efforts are required to bring the states into claiming ownership of the project SICBP should make a concerted and deliberate effort (within available funding) to identify and address the constraints to the successful operation of the Training Contracts. If funds are available, more resources should be provided to augment the capacity building effort We agree; no further comments Response: TO8 utilized not only our site engineers to assist the contractors in implementing projects but also has used trainers from KTC who scheduled and traveled between all states except Unity working with the new L-based contractors and These LBG comments are noted for the record, as they were taken into consideration in drawing up draft report conclusions and recommendations. The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 58 at the state and county levels. There is a preponderance of constraints which inhibit the successful execution of the Training Contracts that need to be addressed. going to the states. Additional efforts are required from LBG in both supervising the contractors and supervising the junior engineers. the on-going contracts being implemented by the “older” contractors. Cont'd… Cont'd Comments in body of report, pg 26: Efficiency and Effectiveness – pg 26: This is technically incorrect. Contracts are for 6 months maxing at 12 possible site visits. Central Equatoria site visits through previous contract implementation was conducted weekly since it is closest to the TO8 offices. The Jonglei contract oversight was constrained due to security issues in the region and location of the road (74 km north of Bor). The contracts in the Bahr eGhazal region and Lakes states were mobilized late December and very little effort was made over the holidays in physical progress. Travel restrictions including little to no air travel in April and May due to the pre- and post-election process. All contractors in May were slow in restarting their works. Noted. Draft report observations were based on documents availed by LBG. Report has been revised to incorporate the observations identified here. Please not that even with these revisions, the conclusions and recommendations still stand. The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 59 13 SICBP Project Design, Implementation and Impacts The overall design of the program was well constructed to achieve the goals and objectives. However, the objective of developing a road maintenance regime for Southern Sudan capable of protecting the investments of USAID, other donors, and GOSS does not come through clearly in the implementation of the project. Some design elements of the original project have since been eliminated (e.g., civil aviation), leading to a project that is more focused on road infrastructure. Southern Sudan is an extremely difficult environment in which to operate due to logistics and security issues and the nascent stage of the government. SICBP is also a complex project with a large and diverse set of stakeholders... ...Given the failure of GOSS to establish an SSRA, the potential impacts of the program are severely jeopardized. For the current SICBP project, USAID should make some modifications to the activities and resource reallocations, but should continue the program. Regarding possible extension of the program, this should be contingent upon establishment of a Road Authority or development of some other institution or vehicle that can ensure the efficient, effective, and transparent management of road infrastructure planning, construction and maintenance. We agree; no further comments Noted. No action necessary. The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 60 ANNEX III – TRAVEL ITINERARY Date Location Meeting Visit Thursday, June 3, 2010 Juba Team Planning Meeting Friday, June 4, 2010 Juba Team Planning Meeting Saturday, June 5, 2010 Juba Sunday, June 6, 2010 Juba to Yambio Travel Day Monday, June 7, 2010 Yambio DG – SMPI and Commissioner WES; Gbudue Construction Gangura – Nabiapai Road Tuesday, June 8, 2010 Yambio DG – SMPI and Commissioner WES; Honorable James Bage – MP; Gbudue Consturction; CARD Construction Saura – Lirangu Road, Yambio Chamber of Commerce Wednesday, June 9, 2010 Yambio to Juba Travel Day Thursday, June 10, 2010 Juba GOSS/MTR – Under-Secretary and staff, IT Dept. Friday, June 11, 2010 Juba LBG Saturday, June 12, 2010 Juba Contractor training – Yei Road Sunday, June 13, 2010 Juba Monday, June 14, 2010 Juba Graduation ceremony; DG – SMPI; local contractors; LBG – Ed Flint; WB/MDTF Tuesday, June 15, 2010 Juba to Rumbek Travel Day Wednesday, June 16, 2010 Rumbek DG – SMPI Lakes and staff; Contractors Rumbek Urban Road Thursday, June 17, 2010 Rumbek to Juba Travel Day Friday, June 18, 2010 Juba GOSS – Dr. Wani Saturday, June 19, 2010 Juba LBG; USAID – David Gosney Sunday, June 20, 2010 Juba Monday, June 21, 2010 Juba Evaluation Team Meeting, Draft Report Tuesday, June 22, 2010 Juba Evaluation Team Meeting, Draft Report Wednesday, June 23, 2010 Juba Presentation Thursday, June 24, 2010 Juba Friday, June 25, 2010 Travel Travel Day Saturday, June 26, 2010 Travel Travel Day The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 61 ANNEX IV – SICBP ORGANIZATION CHART The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 62 ANNEX V – LIST OF KEY INFORMANTS INTERVIEWED USAID/SUDAN  David Gosney – Team leader, Office of Economic Growth  Boutros Magaya – COTR/Engineer  Terrence Kramer – Engineering Advisor GOSS/Ministry of Transport & Roads  Raymond Pitya Morbe – Under-Secretary  Jacob Mariel Maker – Director General, Roads & Bridges  Maurice Reham - Director General, Road Transport & Safety  Gabriel Maker Amuor - Director, Roads & Bridges  James Khor Aguto – Inspector of IT  Morris – Road Safety State Ministries of Physical Infrastructure  Director-Generals of 6 states & their key staff (Directors & Dep. Directors) The Louis Berger Group  Ed Flint – Manager, Task Order 8  Richard Nyarsuk – Dep. Task Manager, TO8  Richard Ruati – Public Affairs Officer  Anthony Murithi – TO8 Policy Advisor  Immaculate Otieno – Admin. Engineer (RAMS)  Steve Atura – Site Engineer TO8  Phillip Thon - Site Engineer TO8  Professional staff (Engineers)  Local staff GOSS/Ministry of Housing  Dr Daniel Wani – Under-Secretary The World Bank  Tesfamichael Tsusenya - Representative/MDTF Coordinator The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 63 Western Equatoria State (Yambio)  Hon. James Bage – Member of Parliament  Nelson Abbas – Gbudue Construction  David Billy Epanetus – Commissioner (Care-Taker)  Richard Issi Mizan – Director-General, SMPI & Senior staff (Directors)  James Okuara – CARD Construction  Bianca Paulina Mbusa – Clinical Officer, Lirangu Hospital  Clement Pasio – Dep. Administrator, Lirangu Payam  David Nimeri – State Town Planning  Wilson Abdal – Director of Transport  Adari Martin Fali – Director of Surveys  Patrick Peter – Yambio Chamber of Commerce  Alison William– Yambio Chamber of Commerce  Mbiko Barakati– Yambio Chamber of Commerce Lakes State (Rumbek)  Joseph Maker - Director-General, SMPI & Senior staff (Directors)  Peter Bull Perweng - Contractor Uganda National Road Authority  Ssebbugga-Kimeze – Director of Operations (by phone) Kisii Training Center  Pius Nyamila – Chief Instructor  Henry Matoke – Instructor The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 64 ANNEX VI – LIST OF DOCUMENTS REVIEWED (SECONDARY SOURCES) 1. Ministry of Transport & Roads, Government of Southern Sudan: Institutional Framework for Road Funding and Management, July 2006 2. Ministry of Transport & Roads, Government of Southern Sudan: Strategic Plan for Road Sector, July 2006 3. Ministry of Transport & Roads, Government of Southern Sudan: Transport Sector Policy for the Ministry of Transport & Roads, Government of Southern Sudan, September 2007 4. The Louis Berger Group: Sudan Infrastructure Services Project. Task Order 8 Capacity Building Program: 2010 Work Plan Narrative and Program Costs, November 2009 5. The Louis Berger Group: Sudan Infrastructure Services Project. Task Order 8 Capacity Building Program: Program Review & 2009 Work Plan Realignment Narrative & Budget, 29 August 2009 6. The Louis Berger Group: Sudan Infrastructure Services Project. Task Order 8 Capacity Building Program: 2010 Work Plan Narrative and Program Costs, November 2010 7. The Louis Berger Group: Sudan Infrastructure Services Project. Task Order 8 Capacity Building Program: Monthly Progress Report – April 2010 8. The Louis Berger Group: Sudan Infrastructure Services Project. Task Order 8 Capacity Building Program: Monthly Progress Report – March 2010 9. The Louis Berger Group: Sudan Infrastructure Services Project. Task Order 8 Capacity Building Program: Monthly Progress Report – February 2010 10. The Louis Berger Group: Sudan Infrastructure Services Project. Task Order 8 Capacity Building Program: Monthly Progress Report – January 2010 11. The Louis Berger Group: Sudan Infrastructure Services Project. Task Order 8 Capacity Building Program: Monthly Progress Report – December 2009 12. The Louis Berger Group: Sudan Infrastructure Services Project. Task Order 8 Capacity Building Program: Monthly Progress Report – November 2009 13. The Louis Berger Group: Sudan Infrastructure Services Project. Task Order 8 Capacity Building Program: Monthly Progress Report – October 2009 14. The Louis Berger Group: Sudan Infrastructure Services Project. Task Order 8 Capacity Building Program: Monthly Progress Report – September 2009 15. The Louis Berger Group: Sudan Infrastructure Services Project. Task Order 8 Capacity Building Program: Monthly Progress Report – August 2009 16. The Louis Berger Group: Sudan Infrastructure Services Project. Task Order 8 Capacity Building Program: Monthly Progress Report – July 2009 17. The Louis Berger Group: Sudan Infrastructure Services Project. Task Order 8 Capacity Building Program: Monthly Progress Report – June 2009 18. The Louis Berger Group: Sudan Infrastructure Services Project. Task Order 8 Capacity Building Program: Monthly Progress Report – May 2009 The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 65 19. The Louis Berger Group: Sudan Infrastructure Services Project. Task Order 8 Capacity Building Program: Monthly Progress Report – April 2009 20. The Louis Berger Group: Sudan Infrastructure Services Project. Task Order 8 Capacity Building Program: Monthly Progress Report – March 2009 21. The Louis Berger Group: Sudan Infrastructure Services Project. Task Order 8 Capacity Building Program: Monthly Progress Report – February 2009 22. The Louis Berger Group: Sudan Infrastructure Services Project. Task Order 8 Capacity Building Program: Monthly Progress Report – January 2009 23. The Louis Berger Group: Sudan Infrastructure Services Project. Task Order 8 Capacity Building Program: Monthly Progress Report – December 2008 24. The Louis Berger Group: Sudan Infrastructure Services Project. Task Order 8 Capacity Building Program: Monthly Progress Report – November 2008 25. The Louis Berger Group: Report on Kisii Training Center Participation by Task Order 8 Training Engineer, 23 February – 9 March 2009, April 21, 2010 26. The Louis Berger Group: Sudan Infrastructure Services Project. Revised Performance Monitoring , Transport Water & Energy Sector programs , June 2009 27. The Louis Berger Group: Environmental Guidelines, Directorate of Roads & Bridges November 2007 28. Southern Sudan Laws: The Southern Sudan Roads Authority Bill 2010 – Cabinet Memo 29. Schedule of LBG Engineers Site visits 8/8/2009 to 10/5/2010 30. The Louis Berger Group: FY 2008 Statutory Checklists 31. USAID/Sudan: performance Management Plan – Economic Growth, July 2009 32. Concept Note: New Task Order (TO) Capacity Building Component (CBC) Sudan Infrastructure Service Project RFTOP to Louis Berger Group (LBG) 33. The Louis Berger Group: Final Report & Quarterly Report No. 13 - Sudan Infrastructure Program. Capacity Building Component, July – September 2008 34. The Louis Berger Group: USAID/Sudan SISP Capacity Building Program – Task Order 8, Quarterly Progress Report Q3 2009, June 30 2009 35. The Louis Berger Group: USAID/Sudan SISP Capacity Building Program – Task Order 8, Quarterly Progress Report Q4 2009, September 30 2009 36. The Louis Berger Group: USAID/Sudan SISP Capacity Building Program – Task Order 8, Quarterly Progress Report Q1 2010, December 31 (2009) 37. The Louis Berger Group: Request for Task Order Proposal 8, September 26 2008 The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 66 ANNEX VII – PERFORMANCE INDICATORS Proposed preliminary indicators include:  Number of States Ministries of Infrastructure operational and established.  Number of new business activities coming up in urban centers along the road  Number of IDP and refugee returnees in urban centers in the vicinity of the road  Average cost per km of delivery of foodstuff for humanitarian assistance Specific indicators are:  Kms of roads built through funds provided by US Government.  Number of people benefiting from transportation projects as a result of USG assistance  Has the Government adopted improved transport transportation related policies or plans this year as a result of USG assistance.  Number of people receiving USG supported training in the transportation technical field.  Number of people receiving USG supported training in transportation management systems.  Number of people receiving USG supported training in transportation policy and regulatory system.  Number of transportation infrastructure agencies that are operational as a result of USG assistance.  Number of private local contractors in business as a result of USG Assistance. The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 67 ANNEX VIII – GETTING TO ANSWERS (DATA INSTRUMENTS USED) Evaluation Methods for Data Collection Questions Type of Answer/ Evidence Needed Method Data Source Sampling or Selection Approach (if applicable) Data Analysis Methods (description; comparison; cause and effect) and notes on special requirements (e.g., frequency distributions, trend analysis, cross-tabulations, content analysis) 1. Are there any issues with respect to project design and assumptions (documented or implied) that should be reconsidered based on experience to date? Addressed in 1a thru 1i below. Addressed in 1a thru 1i below. Addressed in 1a thru 1i below. Addressed in 1a thru 1i below. Addressed in 1a thru 1i below. 1.a An operating SSRA (Southern Sudan Roads Authority) has been considered a critical institutional piece for a functioning roads system. Can the project have long￾term impact in the absence of a SSRA? Description of the role of SSRA in project Capacity of stakeholders in absence of SSRA— physical, human, financial, institutional Description of institutional framework in govt for road construction Other Road Authority descriptions Document review Interviews— group with ministry; group with local govt.; group with contractor personnel; key interview w/donors; key interviews with LBG; key interviews with USAID Project Document LBG proposal and work plans Draft legislation Ministry/USAID/LBG Multilateral/bilateral donors; States; Local governments Contractors Road Authority documents from Successful and worst states/contractors/locations Content pattern analysis Compare/contrast The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 68 Evaluation Methods for Data Collection Questions Type of Answer/ Evidence Needed Method Data Source Sampling or Selection Approach (if applicable) Data Analysis Methods neighboring countries (Kenya, Uganda) 1.a.i What is the likelihood of the SSRA becoming a reality? Same as above except Timeline of Road Authority legislation Level of commitment from GOSS Description of process of legislation Same as above Key informant interviews Document review Same as above GOSS or US of Transport and Roads Minister of Legal Affairs/lawyer consulting with ministry—Bill Kosar Minister’s comments on draft legislation USAID LBG States Content pattern analysis 1. b Medium- and long-term financial sustainability would seem to be dependent on an operational financing mechanism, such as a Roads Fund. What is the likelihood of this becoming a reality? Same as 1.a.i Progress and timeline on roads fund Description of reasons for not wanting a road fund Description of roads fund framework Group/key informant interviews Document review Ministry of Finance Ministry of Legal Affairs World Bank Institutional framework for roads fund USAID LBG Content pattern analysis 1.b.i What are USAID Connection between Document MTR 5-year strategy Content pattern analysis The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 69 Evaluation Methods for Data Collection Questions Type of Answer/ Evidence Needed Method Data Source Sampling or Selection Approach (if applicable) Data Analysis Methods and GOSS options in the absence of a functioning financing mechanism or to develop one? roads fund and GOSS strategy—current plans Description of financing available—current plans review Interviews plan GOSS and donor financing strategies GOSS and donors— Ministry of Finance, World Bank, USAID DG Roads and Bridges—James Alam (Chief Engineer) LBG 2007 Transport Sector Policy Final Report on predecessor project 1.c How sustainable are the efforts under SICBP (and its predecessor project)? Description of labor￾based versus mechanized technology approach Description of financing arrangements Contractors’ access to funding & equipment Effectiveness of capacity building List of contractors and roads they’re working Document Review Interviews Surveys—one for old contractors and one for new 2009 Project Report for Labor-based Contractors (LBG) April 2010 Monthly project report LBG State USAID MTR Contractors—as many as possible Content pattern analysis Trend analysis Frequency distributions Cross-tabulations Unit-cost calculation The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 70 Evaluation Methods for Data Collection Questions Type of Answer/ Evidence Needed Method Data Source Sampling or Selection Approach (if applicable) Data Analysis Methods on Contract costs vs kilometers produced Contractors Data from 1.f Training contracts 1.d Many individuals are working for LBG under SICBP with the understanding that they will transition to government after the SICBP. How will those individuals be retained and operate after this project? Description of original planning process for this strategy Description of expectations of SICBP Sudanese staff Description of MTR staff requirements (positions available) and plan for LBG staff— original plans and current thinking/plans Description of similar examples of this strategy USAID has employed Levels of funding available for hiring staff Salary levels at LBG vs. govt. (pay scales) Comparison—staff capabilities vs. ministry Document review Interviews Survey of staff – Sudanese, LBG SICBP contract LBG Concept notes and proposal SICBP documents LBG Sudanese staff LBG MTR USAID Ministry salary scale Roads Authority bill (for description of recruitment strategy) Opportunistic Content pattern analysis Frequency distribution Compare/contrast The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 71 Evaluation Methods for Data Collection Questions Type of Answer/ Evidence Needed Method Data Source Sampling or Selection Approach (if applicable) Data Analysis Methods needs Description of ministry recruitment process 1.d.i Is there potential for this to be a successful approach? Same as above Speed of recruitment processing Same as above Same as above Same as above Same as above 1.e How were the roads under SICBP selected? What criteria (if any) were used? Description of selection criteria Description of selection process Document review Interviews Surveys (contractors) Project design documents LBG staff— management and engineers Meeting reports—LBG and states on site selection MTR USAID State officials— Directorate of Roads and Bridges Local communities Contractors Assessment reports State work plans Purposive—new and old contractor sites; labor-based versus mechanized; weaker versus stronger projects Content pattern analysis Frequency distributions The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 72 Evaluation Methods for Data Collection Questions Type of Answer/ Evidence Needed Method Data Source Sampling or Selection Approach (if applicable) Data Analysis Methods 1.f The project has attempted to address capacity constraints for a diverse set of stakeholders, including local contractors, GOSS staff, and state officials. How has this been done? Description of trainings Number of people trained Description of skills lacking, skill levels, capabilities Description of training strategy Funding levels Document Review Interviews Surveys Training reports Project design documents Training Needs Assessment Training narrative and budget Work plan Budgets USAID Capacity Building Baseline Survey for MTR LBG USAID MTR Contractors State officials Content pattern analysis Frequency distributions 1.f.i Has this strategy been effective in terms of capacity built? Same as above plus Performance of the contractors Number of kilometers per contractor— comparison over time Interviews Document review Surveys Same as above plus Road contracts completion reports Completion certificates Training evaluations— All of contractors Supervisors in states visited—opportunistic Random, stratified sample of MTR-level, based on training subjects Content pattern analysis Frequency distributions Cross-tabulations Mathematical (averages) The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 73 Evaluation Methods for Data Collection Questions Type of Answer/ Evidence Needed Method Data Source Sampling or Selection Approach (if applicable) Data Analysis Methods Number of contractors with work outside of training contract Number of states with work plans and budgets and can identify their road network and are conducting road supervision with LBG￾provided vehicles Description of benefits to MTR from capacity building Skill-levels of trainees Examples of independent actions in Ministry and states based on training in training reports State work plans and budgets Compare/contrast Trend analysis 1.g Is the scope of the road safety program appropriate based on the needs identified? 1.g.i Should it be expanded or reduced? Description of needs identified Traffic safety data Description of road safety training Description of road safety awareness work Percent of project budget devoted to this Interviews Document Review Draft Road Traffic and Safety bill SICBP budget Baseline and follow-up surveys LBG Road safety materials, products from SICBG Road safety awareness Content pattern analysis Statistical analysis Frequency distributions from LBG survey The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 74 Evaluation Methods for Data Collection Questions Type of Answer/ Evidence Needed Method Data Source Sampling or Selection Approach (if applicable) Data Analysis Methods work Survey data Hospital data report—February monthly report MTR Ministry of Internal Affairs—Traffic Police USAID Safety stickers Follow-up impact survey from LBG (if done in time) 1.h Is the IT solution for communication among states and Juba scaled appropriately? Description of IT and communications system Financing plans Comparison of TO8 budget and financing plans with MTR budget and financing plans Description of training provided Visual data—computers provided, generators Document review Interviews Direct observation Site visits MTR budgets for current and future TO8 budgets Website design Training plans LBG System design documents USAID MTR States Sites See above for selection of locations Content pattern analysis Compare/contrast Logical analysis The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 75 Evaluation Methods for Data Collection Questions Type of Answer/ Evidence Needed Method Data Source Sampling or Selection Approach (if applicable) Data Analysis Methods 1.h.i Is it sustainable and/or cost￾effective? Use of computers Availability of trained IT people in states and MTR Financing plans Comparison of TO8 budget and financing plans with MTR budget and financing plans Type of technology Potential to accommodate other users Description of plans for expansion UNOPS project description Description of GOSS communications system Observation Interviews Document review LBG IT trainer States MTR MTR budgets for current and future TO8 budgets Technology selection System design documents (Ministry of Health) UNOPS Content pattern analysis Compare/contrast 1.i Are there any gender issues that have been or should be addressed? Number of female employees Number of women trained Number of MTR employees Observation Document review Interviews Hiring Tracker—LBG monthly reports MTR Gender reports from World Bank, USAID, NGOs Content pattern analysis Compare/contrast Comparative statistical analysis Trend analysis The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 76 Evaluation Methods for Data Collection Questions Type of Answer/ Evidence Needed Method Data Source Sampling or Selection Approach (if applicable) Data Analysis Methods Description of gender strategy Gender targets Comparison with SISP gender strategy Description of incentives/approaches to address gender equity Statistics on women in various departments Number of women￾owned companies Indications of level of political will Descriptions of regional gender strategies for road projects; regional statistics People days of activity—men vs women (income) Ministry of Gender and Human Rights LBG project design documents SISP project design documents Constitution of South Sudan—gender clause Roads authorities in Uganda and Kenya (Kisii) Contractors—Gbudue (Yambio) 2. SICBP is a large, complex project with many activities and moving parts. Have the IPs succeeded in maintaining quality Data from 2.a-c; Evaluation of program results by stakeholders Document review Interviews Survey MTR States Contractors The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 77 Evaluation Methods for Data Collection Questions Type of Answer/ Evidence Needed Method Data Source Sampling or Selection Approach (if applicable) Data Analysis Methods over the various pieces while maintaining production at an appropriate cost? LBG USAID 2.a How effective is the training? Covered by 2ai thru iv. 2.a.i How highly do trainees rate the quality of the assistance they have had in terms of appropriateness, timeliness and quality? Trainee (MTR, States, Contractor) ratings of training Document review Interviews Survey Post-training evaluations; MTR States Contractors LBG USAID Contractors-as many as possible States-as many as possible MTR-as many as possible Content pattern review Compare/contrast Frequency distribution Mathematical 2.a.ii Are the right people being trained? Selection criteria for trainees; Description of LBG evualation process Current job responsibilities; Job performance post￾training Document review Interviews Surveys Solicitation for contractors Selection criteria documents Evaluation process documents MTR States Contractors-as many as possible States-as many as possible MTR-as many as possible Content pattern analysis Compare/contrast Frequency distribution Mathematical The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 78 Evaluation Methods for Data Collection Questions Type of Answer/ Evidence Needed Method Data Source Sampling or Selection Approach (if applicable) Data Analysis Methods Trainer rating of trainees Contractors LBG USAID KISII Uganda Road Authority 2.a.iii Are they applying the training to their work? Contractors performance ratings Contractor description of business operations Other road construction data or statistics Current job responsibilities; Description of promotions, accomplishments post training; Document review Interviews Surveys Contract Completion Report; Performance Indicator Reference Sheet; MTR States Contractors LBG USAID Contractors-as many as possible States-as many as possible MTR-as many as possible Content pattern analysis Compare/contrast Frequency distribution Mathematical 2. a.iv. Does the training offered meet the needs identified Description of needs identified; Document review Training Needs Assessment Content pattern review Compare/contrast The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 79 Evaluation Methods for Data Collection Questions Type of Answer/ Evidence Needed Method Data Source Sampling or Selection Approach (if applicable) Data Analysis Methods in the Training Needs Assessment and by the Ministry of Transport and Roads? Trainee (MTR, States, Contractor) ratings of training; Description of training courses Interviews Surveys Project Documents (LBG project design documents, work plan) Training modules MTR LBG USAID 2. b Do the roads being build meet standard specifications? Description of standard specifications Description of contract specifications; Quality assessment of completed roads Document review Interviews Observations Signed Contract Contract Completion Report; MTR States LBG USAID Donors (e.g. World Bank) Content pattern analysis Compare/contrast Frequency distribution Mathematical 2.c Do they meet appropriate environmental Description of environmental standards; Document review Environmental Guidelines for Roads; Content pattern analysis Compare/contrast The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 80 Evaluation Methods for Data Collection Questions Type of Answer/ Evidence Needed Method Data Source Sampling or Selection Approach (if applicable) Data Analysis Methods standards? Description of project design; Assessment of completed roads Interviews Observations Signed Contract Contract Completion Report; MTR States LBG USAID Donors (e.g. World Bank) Ministry of Environment (time allowing) 3. Are there ways in which the effectiveness and efficiency of SICBP could be improved? Description of program objectives; Description of performance indicators and results Program costs Data from 1 and 2 Document review Interviews Survey Observations LBG Project Documents MTR States LBG USAID Donors (e.g. World Bank) Contractors-as many as possible States-as many as possible MTR-as many as possible Content pattern analysis Compare/contrast The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 81 Evaluation Methods for Data Collection Questions Type of Answer/ Evidence Needed Method Data Source Sampling or Selection Approach (if applicable) Data Analysis Methods Contractors Chamber of Commerce Local Governments 4. SICBP developed contract management and administrative systems and is engaged in a significant level of procurement and oversight. Are these systems working in a way that is effective, transparent, and designed to minimize corruption? Description of procurement process; Description of national, regional, and international tendering practices; Description of concerns, issues, and incidences of corruption within project and in road sector; Anecdotes on corruption within GOSS Document review Interviews Surveys Procurement Documents; MTR States LBG USAID Donors (e.g. World Bank) Contractors Minister of Legal Affairs/lawyer consulting with ministry—Bill Kosar; Committee on Contractors-as many as possible States-as many as possible MTR-as many as possible Content pattern analysis Compare/contrast Cross tabulation The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 82 Evaluation Methods for Data Collection Questions Type of Answer/ Evidence Needed Method Data Source Sampling or Selection Approach (if applicable) Data Analysis Methods Corruption 5. Are SICBP human resources sufficient to achieve the project’s goals? Data on LBG staffing (number and type of positions, position descriptions); Description of staffing issues and concerns; Description of LBG workload; Data on overtime; Data on other USAID Southern Sudan capacity building projects Document review Interviews Group Interview Project Design Documents; LBG (Human Resources, local staff, etc.) LBG HR Reporting Documents USAID MTR USAID Capacity Building Project Documents Content pattern analysis Compare/contrast 6. The GOSS relies on SICBP for support on a number of levels at the same time that SICBP is trying to encourage its government partners to become self-sufficient. How well is LBG balancing these Description Description of program objectives regarding self-sufficiency; Description of original planning process for this strategy; Data on GOSS staffing; Document review Interviews MTR HR Reports Project Design Documents; Project Documents; Transition plans; MTR LBG Content pattern analysis Compare/contrast The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 83 Evaluation Methods for Data Collection Questions Type of Answer/ Evidence Needed Method Data Source Sampling or Selection Approach (if applicable) Data Analysis Methods competing pressures? Data on LBG staffing; Description of division of responsibilities between SICBP and LBG; Description of transitioning of responsibilities from LBG to SICBP USAID 7. What gender issues have been addressed during the implementation of SICBP? Number of female employees; Number of women trained; Number of MTR employees Description of gender strategy Gender targets Comparison with SISP gender strategy Description of incentives/approaches to address gender equity Statistics on women in Observation Document review Interviews Hiring Tracker—LBG monthly reports MTR Gender reports from World Bank, USAID, NGOs Ministry of Gender and Human Rights Ministry for Public Service LBG project design documents SISP project design documents Constitution of South Content pattern analysis Compare/contrast Comparative statistical analysis Trend analysis The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 84 Evaluation Methods for Data Collection Questions Type of Answer/ Evidence Needed Method Data Source Sampling or Selection Approach (if applicable) Data Analysis Methods various departments Number of women￾owned companies Indications of level of political will Descriptions of regional gender strategies for road projects; regional statistics People days of activity—men vs women (income) Sudan—gender clause Roads authorities in Uganda and Kenya Contractors—Gbudue; USAID 8. Assess SICBP communication and coordination with project partners. Description of communication and coordination strategy and planning; Description of meeting schedules; Description of LBG and MTR communication and coordination practices, issues and concerns. Observation Document review Interviews Project Design Documents; Project Documents; MTR LBG USAID States; Kisii, UNRA, Local Government Community; legal Affairs; Min. Internal Affair; Environment Content pattern analysis Compare/contrast The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 85 Evaluation Methods for Data Collection Questions Type of Answer/ Evidence Needed Method Data Source Sampling or Selection Approach (if applicable) Data Analysis Methods 9. Is there evidence that local contractor capacity has been improved? Selection criteria for trainees; Description of LBG evaluation process Contractors performance ratings; Contractor description of business operations; Opinions on capacity improvements Examples of capacity improvements Document review Interviews Surveys Solicitation for contractors; Selection criteria documents; Evaluation process documents; Contract Completion Report; Performance Indicator Reference Sheet; MTR States Contractors LBG USAID Content pattern analysis Compare/contrast 9.a How broadly and how equitably among states? Contractors performance ratings; Contractor description of business operations; Opinions on capacity improvements; Examples of capacity Document review Interviews Surveys Solicitation for contractors; Selection criteria documents; Evaluation process documents; Contract Completion Content pattern analysis Compare/contrast The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 86 Evaluation Methods for Data Collection Questions Type of Answer/ Evidence Needed Method Data Source Sampling or Selection Approach (if applicable) Data Analysis Methods improvments Report; Performance Indicator Reference Sheet; MTR States Contractors LBG USAID 10. What evidence is there that state, regional (Southern Sudan), and national institutions necessary for good and effective implementation of transportation infrastructure development activities in Southern Sudan have been strengthened? Description of program objectives; Description of strategy for achieving program objectives; Description of original needs identified; Observations regarding initial versus current strength and capabilities of institutions (i.e. supervision, policy development, procurement) Document review Interviews Project Design Documents; Project Documents; MTR State LBG USAID Donors (e.g. World Bank) Contractors Content pattern analysis Compare/contrast The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 87 Evaluation Methods for Data Collection Questions Type of Answer/ Evidence Needed Method Data Source Sampling or Selection Approach (if applicable) Data Analysis Methods 10.a How sustainable are any notable improvements? Available budget Observations regarding staffing retention; Description of plans to meet future requirements; Document review Interviews MTR Planning Documents; MTR; MTR HR Staffing Documents; LBG USAID Donors (e.g. World Bank) Content pattern analysis Compare/contrast 11. What evidence is there to suggest that broader economic and social impacts are being achieved? Testimony regarding economic and social improvements; Interviews Surveys Chamber of Commerce Local Govts; Community Members (Chiefs, Hospital, Traders, Drivers) States USAID MTR LBG Content pattern analysis Compare/contrast 12. How is SICBP supporting other Details on other USAID projects and objectives; Document review USAID Content pattern analysis The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 88 Evaluation Methods for Data Collection Questions Type of Answer/ Evidence Needed Method Data Source Sampling or Selection Approach (if applicable) Data Analysis Methods USAID projects and objectives? Project design documents; Interviews LBG Predecessor Evaluation Project Documents USAID Strategy Statement 12.a What has been the overall impact on governance and CPA support? Program objectives; Details on GOSS ability to work in, access and communicate with States and local and governments; Date from 11; Document review Interviews USAID LBG Project Documents USAID Strategy Statement; MTR Content pattern analysis 13. Based on learning from the evaluation, is a shift in resource allocation among project components called for? Special attention should be given to the state level. Description of program strategy; Details on current resource allocation (i.e. staff, budget) and modifications made during implementation; Opinions regarding resource allocation; Resource allocation versus performance (results and impacts for Document review Interviews Document review USAID LBG Project Documents USAID Strategy Statement; MTR Performance Indicator Reference Sheet; States Content pattern analysis; Cross tabulation The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 89 Evaluation Methods for Data Collection Questions Type of Answer/ Evidence Needed Method Data Source Sampling or Selection Approach (if applicable) Data Analysis Methods each project component) 14. What capacity building approaches appear to be particularly productive? Description of capacity building approaches; Resource utilization of each approach (Data from 13) Data from 2a. Performance from each approach. Interviews Group Interview (Contractors) Document review Contractors; USAID States LBG Project Documents USAID Strategy Statement; MTR Performance Indicator Reference Sheet; Content pattern analysis; Compare/contrast Cross tabulations The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 90 SAMPLE QUESTIONNAIRE FOR GOSS MINISTRY OF TRANSPORT & ROADS Interview Guide – Ministry of Transport and Roads Name of respondent: Date: Organisation: Time: Position: Interviewer: Introduction: Thank you for taking the time to meet with us today. This interview should take about an hour to complete. We are part of an evaluation team charged with evaluating the Sudan Infrastructure Capacity Building Program (SICBP), funded by the U.S. Agency for International Development (USAID) and the Louis Berger Group (LBG). This interview is being conducted to collect data for that evaluation and will contribute to learning aimed at making improvements in the planning, management and operations of SICBP and USAID’s infrastructure work in South Sudan in general. To make the evaluation as useful as possible to USAID and its partners/stakeholders, we would appreciate you providing frank and direct answers to the interview questions. Your identity will be kept confidential. Unless we first ask for and receive your permission, your comments will not be shared with USAID or any of its partner organizations. Do you have any questions before we begin? General Profile 1. Please describe your position, principal functions, and responsibilities at this organization Project Design [1a(i)]: An operating SSRA (Southern Sudan Roads Authority) has been considered a critical institutional piece for a functioning roads system. Can the project have long-term impact in the absence of a SSRA? Can you describe the role of the SSRA in the Task Order 8 Project? What do you see as the impact of the SSRA on road construction & maintenance? What is the current timeline of Road Authority legislation How would you describe the level of commitment from GOSS Description of process of legislation Can there be an effective road maintenance program without a SSRA? 1(b) Medium- and long-term financial sustainability would seem to be dependent on an operational financing mechanism, such as a Roads Fund. The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 91 Does MTR think that a Roads Fund is a good idea? What is the likelihood of this becoming a reality? What is the progress and timeline for the roads fund? Can you provide a description of the roads fund framework? What are the options in the absence of a Roads Fund? [1c] How sustainable are the efforts under SICBP (and its predecessor project)? Do you think that MTR currently has the capacity to manage a road maintenance program for the region? In which areas does MTR require additional capacity? Do you think the labor-based versus mechanized technology approach is the most appropriate? How effective has the capacity building been so far? What is the greatest limitation for Sudanese contractors in their work? What is the greatest limitation for MTR in its work? [1(d) Many individuals are working for LBG under SICBP with the understanding that they will transition to government after the SICBP. How will those individuals be retained and operate after this project? Description of original planning process for this strategy Description of MTR staff requirements (positions available) Levels of funding available for hiring staff Salary levels at LBG vs. govt. (pay scales) Comparison—staff capabilities vs. ministry needs Description of ministry recruitment process [1(e)] Can you describe how the roads for rehabilitation were selected? [1(f)] What training have you received from the project? When did you get this training? How long was it? How relevant to your day-to-day work was it? Are there any additional items relevant to capacity building you would have liked to be covered in the training? On a scale of 1 – 10 with 10 being the highest, rate the following aspects of the training under this program Appropriateness Timeliness The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 92 [2ai] [2aii] [2aiii] Quality [1f(i)] Has the capacity building strategy been effective? Can you give me specific examples on how you have applied the training in your work? (i.e. work plans, budgets, supervision, etc.) Performance of the contractors How important do you think the contractor training is to the overall program? Can you describe the benefits to MTR from capacity building How has the skill-level of the trainees improved? Are there any specific examples of how the capacity building has improved MTR’s performance in terms of road maintenance? [1(h)] – IT-SOLUTIONS What current communication systems do you have? What will you use the new system for? What are the plans for financing the maintenance and upkeep of the system? How much will it cost to maintain the system? Have you budgeted for this? Is there an agreement with LBG/USAID that MTR will finance this upkeep and bandwidth costs? What IT training have you received or undertaken on your own? Do you have any plans for expanding the system? 1.g How would you rank road safety as a priority of the MTR: a. the top priority b. one of the main priorities c. not a priority for MTR. Is the scope of the road safety program appropriate based on the needs identified? Should it be expanded or reduced? How have you incorporated road safety in the improvement or maintenance of the roads? ...of the bridges? 2.b Do the roads being build meet standard specifications? What is the most common problem you have found on your completed roads? 2.c Do the roads meet appropriate environmental standards? Description of environmental standards; The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 93 Description of project design regarding environmental standards. 3.i Are there any gender issues that have been or should be addressed? What is the number of female employees at MTR? Number of women trained from MTR?? Number of MTR employees? Is there a gender strategy for the project? Gender targets? Comparison with SISP gender strategy Description of incentives/approaches to address gender equity Statistics on women in various departments 4 SICBP developed contract management and administrative systems and is engaged in a significant level of procurement and oversight. Are these systems working in a way that is effective, transparent, and designed to minimize corruption? Can you describe the procurement process for this program? In what areas of procurement have you had problems or concerns? What have you done about those problems? What has MTR done to prevent corruption within the roads sector? 5 Do you have suggestions on how this program can be improved? 6. The GOSS relies on SICBP for support on a number of levels at the same time that SICBP is trying to encourage its government partners to become self-sufficient. How well is LBG balancing these competing pressures? Description of program objectives regarding self-sufficiency; The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 94 Description of original planning process for this strategy; Data on GOSS staffing; Description of division of responsibilities between SICBP and LBG; Description of transitioning of responsibilities from LBG to SICBP 8. Assess SICBP communication and coordination with project partners. Please describe communication and coordination strategy and planning; Please describe meeting schedules. Description of LBG and MTR communication and coordination practices, issues and concerns. 9 Is there evidence that local contractor capacity has been improved? How important is capacity building for Sudanese contractors to the road maintenance sector? How has the program affected local contracting capacity? – Give examples How many firms have contracts with you now? How many had contracts before the project started (i.e. before LBG) Have any contractors received new contracts subsequent to finishing the LBG contract? 10. What evidence is there that state, regional (Southern Sudan), and national institutions necessary for good and effective implementation of transportation infrastructure development activities in Southern Sudan have been strengthened? Description of program objectives; Description of original needs identified Description of strategy for achieving program objectives How sustainable are any notable improvements? Available budget Observations regarding staffing retention; Description of plans to meet future requirements; The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 95 11 What changes has the completion of specific roads brought to those areas? 12.a What has been the overall impact on governance and CPA support? Are roads considered an important part of the peace dividend? Details on GOSS ability to work in, access and communicate with States and local and governments; Data from 11; 13. Based on learning from the evaluation, is a shift in resource allocation among project components called for? Special attention should be given to the state level. Details on current resource allocation (i.e. staff, budget) and modifications made during implementation; Opinions regarding resource allocation 14. What capacity building approaches appear to be particularly productive? 15. The current program is scheduled to end in 2011. Does your organization have any plans post 2011? What are your organization’s plans for continuation of the programs achievements after 2011? The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 96 ANNEX IX – DATA AND TABLES FROM ANALYSIS Table 1 – SMPI’s Perceived importance of VSAT system Table 2 – SMPI’s Perceived sustainability of labor-based construction technolgy The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 97 ANNEX X – SUDAN TRIBUNE ARTICLE ON SICBP Tue, Jun 29, 2010 22:36 UT GOSS minister lauds USAID for boosting infrastructure in road maintenance Friday 18 June 2010. By Richard Ruati June 16, 2010 (JUBA) — Anthony Lino Makana, Minister of Transport and Roads has praised USAID’s Sudan Infrastructure Services project (SISP) for its concerted efforts over the past four years to boost road maintenance and capacity building in select state ministries in ten states across the would-be South Sudan country. Mr. Makana was speaking at the graduation of 17 new southern Sudanese road contractors at the Ministry of Transport and Roads’ Yei Road Facility in Juba on Monday June 14, 2010. The event marked the completion of a 10-week training course, and the local contractors will be awarded small contracts for the construction of feeder roads, which will be monitored and evaluated by USAID partner site engineers in their local areas. The graduates will also be presented with small, mechanized equipment to support their efforts. "This is yet another milestone in improving rural roads and the underdeveloped transport system in Southern Sudan, promoting economic recovery and employment." The occasion was attended by senior USAID Economic Growth officials among others, Camelita Manes, Eng Boutros Magaya, George Waggwa and Mary Aurupai and high States Director Generals plus their directors of Roads and Bridges in the ten states of Southern Sudan as well as the Louis Berger Group Sudan Chief of Party (COP) Dave Little. The Minister encouraged the local contractors to take improvement of roads upon their shoulders, “you must take your hoes and shovels to show good results to the American people, who continue to develop your capacities." The graduated contractors have joined forces with 13 previously trained labor-based contractors, some of whom were present at the event. USAID capacity-building training has resulted in 30 fully-trained, road maintenance contractors working in all 10 states of southern Sudan. The Transport Minister highlighted some of the major successes of the project – the development of human resources, which is now being used improve deplorable feeder roads, VSAT communication, which will allow State Directorates of Roads and Bridges access to real time data to drive improvements in road asset management. SudanTribune article : GOSS minister lauds USAID for boosting infrastructure in road maintenance http://www.sudantribune.com/spip.php?page=imprimable&id_article=35423 1 of 2 6/29/2010 6:39 PM The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 98 ANNEX XI – REPRESENTATIVE PROJECT PICTURES GOSS Minister of Transport & Roads Anthony Makana hands a certificate to one of two graduating lady contractors. In the background are some of the six USAID vehicles to be donated to the states under SICBP. The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 99 This Congolese-registered truck on Ngangura-Nabiapai Road demonstrates the need for a mechanism for axle load controls on the improved roads (Note: It took the truck 6 hrs to move the 13 km from the DRC border to here) Old VSAT dish sits idle outside the Western Equatoria Ministry of Physical Infrastructure in Yambio. Sustainability of new VSAT system is critical The Sudan Infrastructure Capacity Building Program (SICBP) – Mid-Term Evaluation Report 100 This contractor in Rumbek imported truckloads of rock to fill a bad section of his road, and was surprised to learn all this work was not in the contract. All parties agree on need for more contract interpretation in the training curriculum.