END-OF-PROJECT REVIEW SOUTHERN SUDAN TECHNICAL ASSISTANCE PROJECT 29 SEPTEMBER 2011 This publication was produced for review by the United States Agency for International Development. It was prepared by Nancy Allen and Genzo Yamamoto, Management Systems International. END-OF-PROJECT REVIEW SOUTHERN SUDAN TECHNICAL ASSISTANCE PROJECT Contract No. DFD-1-00-05-00251-00, Task Order No. 2 Services Under Program and Project Offices for Results Tracking (SUPPORT) DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. END-OF-PROJECT REVIEW i CONTENTS ACKNOWLEDGEMENTS ......................................................................................................... iii ACRONYMS ................................................................................................................................. iv EXECUTIVE SUMMARY .......................................................................................................... vi 1. INTRODUCTION...................................................................................................................... 1 1.1. Country Context ........................................................................................................... 1 1.2. Program Description .................................................................................................... 3 2. Purpose and Methodology of the Review ................................................................................. 6 2.1. Purpose and Scope of the End-of-Project Review ....................................................... 6 2.2. Review Methodology ................................................................................................... 6 3. Major Findings and Conclusions .............................................................................................. 9 3.1. Strategic Alignment ..................................................................................................... 9 3.1.1. Strategic Alignment with GoSS Strategic Frameworks ............................... 9 3.1.2. Strategic Alignment with USAID Mission and Education Strategy .......... 11 3.2. Implementation of MTE Programmatic Recommendations ...................................... 13 3.2.1. General Discussion .................................................................................... 13 3.2.2. Areas of Strong Progress ........................................................................... 15 3.2.3. Areas of Modest, Stable,or Mixed Progress .............................................. 16 3.2.4. Areas of Minimal Progress ........................................................................ 19 3.3. Sectoral Coordination and Relationships ................................................................... 24 3.3.1. Nature and quality of relationships between TAP and local partners at the state, county and community levels ................................ 24 3.3.2. Facilitation of SMoE and donor partner/NGO relationships ..................... 25 3.3.3. Nature and quality of relationships between TAP and other NGOs and donor partners .................................................................................. 25 3.3.4. Facilitation of relationships between SMoEs and GoSS MoE .................. 27 3.4. Program Achievements .............................................................................................. 29 3.4.1. Extending the gains and accomplishments identified in the MTE ............. 29 3.4.2. Increasing Possibility for Sustained Capacity Gains ................................. 30 3.4.3. Challenges and Threats to Sustainability of Capacity Gains ..................... 31 4. Implications for Future Strategy and Programming ............................................................ 32 4.1. Initial Recommendations ........................................................................................... 32 4.2. Strategy and Program Management ........................................................................... 33 4.3. Coordination and Relationships ................................................................................. 37 4.4. Human Resources and Organizational Structure ....................................................... 38 Annex 1: Scope of Work for TAP Evaluation ........................................................................... 41 Annex 2: TAP EOP Review, List of Interviews ......................................................................... 49 Annex 3: Job Descriptions for STAs and POEs ........................................................................ 51 Annex 4: List of Documents Consulted ...................................................................................... 55 Annex 5: Institutional Development Framework ..................................................................... 56 Annex 6: Draft Memorandum of Understanding ..................................................................... 61 Annex 7: TAP Indicators ............................................................................................................. 65 END-OF-PROJECT REVIEW ii PROJECT DATA TABLE Program Name EQUIP2 Southern Sudan Technical Assistance Program 2 (SSTAP2) Agreement Information Cooperative Agreement # 650-A-00–07–00004–00 Leader with Associate Award (LWA) Modifications: #1, 2, 3, and 5 Office/Region/Country USAID/Sudan Technical Office: Education Office—Juba Africa/South Sudan Program Funding Total estimated amount (2005–2011): $17,624,752 (As of March 2009): US$17,624,752 (As of June 18, 2007): US$8,499,534 (As of August 6, 2007): US$7,999,534 (As of March 2006): US$1m (As of September 2005): US$0.5m Prime Managing Contractor Academy for Educational Development (AED) Other Implementing Partners International Rescue Committee (IRC) Key Program Dates TAP Phase 1 Program Beginning and End Dates September 2005 to June 17, 2007 TAP Phase 2 Program Beginning and End Dates June 18, 2007 to September 17, 20111 TAP Program Midterm Evaluation Mission Period June 21 to July 11, 2009 TAP Program End-of-Project Review Mission Period June 15 to July 1, 2011 Evaluation Provider Management Systems International (MSI) EOP Team Members: Dr. Nancy J. Allen (External Evaluator/Team Leader) Dr. Genzo Yamamoto (External Evaluator) Victor CholDut (Ministry of Education, GoSS) Jane Namadi (USAID) Peter Muyingo (AED) End-of-Project Mission Site Visits State Ministry of Education, Wau, Western Bahr el-Ghazal, State Ministry of Education, Aweil, Northern Bahr el-Ghazal State Ministry of Education, Juba, Central Equatoria 1SSTAP documentation cites different start and ending points for Phase 2. The USAID Associate Cooperative Agreement dated August 6, 2007 specifies the time period of June 18, 2007 to September 17, 2011 on the signature page. However, page 17 of the ACA notes the ―timeframe for implementation‖ as being from July 1, 2007to September 30, 2011. Modification #5 dated April 28, 2010 revised the program end date to September 30, 2011. END-OF-PROJECT REVIEW iii ACKNOWLEDGEMENTS MSI‘s Services Under Program and Project Offices for Results Tracking (SUPPORT) project contracted with independent evaluators Dr. Nancy J. Allen and Dr. Genzo Yamamoto to conduct an end-of-project review of the United States Agency for International Development (USAID) Mission‘s Southern Sudan Technical Assistance Program Phase 2 (SSTAP2). The SSTAP2 program, led by the Academy for Educational Development (AED) began in 2007 and has been implemented in all 10 states in South Sudan. The external evaluators were joined by three Juba-based team members: Peter Muyingo of AED/TAP, Jane Namadi of USAID, and Victor DutChol of the government of South Sudan‘s Ministry of Education (MoE). Their knowledge and support in the field was deeply appreciated. The report could not have been written without regular consultation with AED‘s Monitoring and Evaluation Advisor, Peter Muyingo. The end-of-project review team would also like to thank the government of South Sudan, in particular the staff of the State Ministries of Education in Northern Bahr el-Ghazal, Western Bahr el Ghaza,l and Central Equatoria states, who generously shared their time and experience. Critically important in the preparation for this review and in the writing of the final report was the work done previously by the midterm evaluation team. The team would like to thank the AED staff: the outgoing Chief of Party, Chris Fagueret; the incoming Chief of Party, Stephen Backman; the Senior Technical Advisors; and the current and former Project Officers for Education. The thoughtfulness and candor of the AED staff greatly improved the quality of this review. The team would like to thank the staff of Management Systems International for their able assistance, especially M&E consultant Simone Holladay, Evaluation Advisor Ingrid Orvedal , and Ms. Ami Henson (Chief of Party, SUPPORT). END-OF-PROJECT REVIEW iv ACRONYMS AED Academy for Educational Development AIR American Institutes for Research AOTR Agreement Officer‘s Technical Representative AP Annual Plan AWP Annual Work Plan APS Annual Program Statement CA Cooperative Agreement CBO Community-Based Organization CBTF Capacity Building Trust Fund CDC Curriculum Development Center CEC County Education Center CEQ/CES Central Equatoria State COP Chief of Party COTR Contracting Officer‘s Technical Representative CPA Comprehensive Peace Agreement CSO Civil Society Organization DCOP Deputy Chief of Party DFID Department for International Development (UK) DG Director General DO Development Objective (within a USAID strategic framework) DPC Development Partner Coordinator DSTP Diaspora Skills Transfer Program DTA Diaspora Technical Assistant EDC Education Development Center EEQ/EES Eastern Equatoria State EMIS Education Management Information System EQUIP Education Quality Improvement Program ERDF Education Rehabilitation and Development Forum ESP Education Strategic Plan FGD Focus Group Discussions FMIS Financial Management Information System FTTP Fast Track Teacher Education and Training Program FY Fiscal Year GEE Gender Equity through Education Project GoSS Government of South Sudan IDF Institutional Development Framework IDP Internally Displaced Person IR Intermediate Result (contributing to a USAID strategic goal) IRC International Rescue Committee JDT Joint Donor Team LTTA Long-Term Technical Assistance M&E Monitoring and Evaluation END-OF-PROJECT REVIEW v MDTF Multi-Donor Trust Fund MoEST Ministry of Education, Science, and Technology MoE Ministry of Education MoU Memorandum of Understanding MoFEP Ministry of Finance and Economic Planning MSI Management Systems International MTE Midterm Evaluation NBG Northern Bahr el-Ghazal State RSS Republic of South Sudan PMP M M Performance Management Plan SMoE State Ministry of Education SMoFEP State Ministry of Finance and Economic Planning SOW Scope of Work/Statement of Work SPLM Sudanese Peoples Liberation Movement SSIRI South Sudan Interactive Radio Instruction SSTAP South Sudan Technical Assistance Program STA Senior Technical Advisor/State Technical Advisor STTA Short-Term Technical Assistance TAP Technical Assistance Program TOT Training-of-Trainers UNICEF United Nations International Children‘s Education Fund WBG Western Bahr el-Ghazal State WEQ Western Equatoria State USAID United States Agency for International Development USG United States Government END-OF-PROJECT REVIEW vi EXECUTIVE SUMMARY I. Introduction 1.1 Country and Sectoral Contexts After half a century of struggle, South Sudan today stands at a historical threshold. On July 9, 2011, with the formal declaration of independence, the Republic of South Sudan (RSS)2 became the world‘s newest nation. The preceding decades of war and underdevelopment—as well as frequent drought, floods and famine—had a devastating impact on social, political, and economic structures in South Sudan. Delivery of basic services by government is extremely limited, hampered by a cavernous gap in human and institutional capacity at all levels of government. The capacity building gains that must take place across all sectors to enable the country to establish itself as an independent state are enormous. In the education sector, the magnitude of challenges facing the government is particularly daunting. On almost every indicator for educational access, South Sudan is at or near the bottom of international rankings. Primary and secondary school enrollments are growing rapidly, and yet only half of all school-age children attend school. Forty percent of teachers have only a primary school education, and the majority of learning spaces are open-air or semi-permanent structures. The ratio of pupils to teachers is extremely high, reported as 200:1 in Jonglei State and averaging 52:1 across all ten states. 1.2 Program Description In September 2005, USAID established the EQUIP2 Southern Sudan Technical Assistance Program (TAP) to build the capacity of the newly created Ministry of Education, Science and Technology (MoEST), TAP has progressed in two distinct phases. Phase 1 (2005–2007): National Focus: In its first phase, SSTAP focused on establishing the ―machinery of government‖ necessary for the functioning of the newly created central-level MoEST (subsequently renamed the Ministry of Education (MoE)). SSTAP worked primarily through short-term technical assistance (STTA) in this first phase. Phase 2 (2007—2011): State Focus: In this second phase, SSTAP2, the project shifted its focus to capacity building at the state level, now operating primarily through long-term technical assistance (LTTA). Capacity building efforts were consolidated in FY 2010 into five key areas: planning and budgeting; financial management; communication and coordination; human resources; and school and educational materials. It is this second phase that is the subject of this review Although SSTAP2 is technically the most accurate acronym for the program, it is commonly referred to simply as TAP. This review henceforth refers to the program as TAP, unless there is a need to make a distinction between the first and second phases. 2 Throughout the report, the term – ―Republic of South Sudan‖ or RSS – is used when describing actions/events from July 9, 2011 onward. When referring to past actions/events (i.e. prior to July 9, 2011) the term ―Government of South Sudan or GOSS is used. END-OF-PROJECT REVIEW vii The TAP capacity building model follows a mentoring approach that supports on-the-job training and learning-by-doing. TAP advisors are embedded directly in SMoEs, working side by side with ministry staff to build their professional skillsets and, by extension, strengthen the institutional capacity of the MoE and SMoEs. This approach is supported by a large number of formal trainings and workshops. In this second phase, Senior Technical Advisors (STAs) and/or Diaspora Technical Assistants (DTAs)—later revised to become Program Officers for Education (POEs)—were to have been placed in all ten states. The program, for a variety of reasons, has never achieved the complete staffing ideal of an STA and a POE in each state. II. Purpose and Methodology of the Review The purpose of the SSTAP2 end-of-project (EOP) review was: ―to assess performance in implementing programmatic changes in accordance with findings, conclusions, and recommendations reported in the midterm evaluation of SSTAP, conducted in 2009.‖ In support of this purpose, a series of questions were posed in the Scope of Work (Annex 1) focusing on three areas: Strategic Alignment, Implementation of MTE Recommendations, and Sectoral Coordination and Partnership. The review was conducted from June 15, to July 1, 2011. The methodology was fundamentally qualitative, with intensive review of program documents followed by site visits in three of the ten states and included interviews with over 50 people (see Annex 2). III. Major Findings and Conclusions 3.1 Program Achievements The EOP review found that TAP has extended the gains and accomplishments identified in the MTE, namely (i) increased professionalism of the SMoEs in general; (ii) increased productivity among individual staff members; (iii) improved capacity of SMoEs to interact with development partners; (iv) increased capacity in planning, understanding, and use of education data; (v) acceptance of the electronic payroll system; and (vi) more confident use of computers and the internet. The EOP team concludes that in the Planning and Budgeting Department, these capacity gains extend to multiple individuals, and the knowledge embedded in the Education Management Information System (EMIS) and electronic payroll systems improves the chances for sustained capacity improvements. Additionally, TAP has begun to expand its role as a facilitator of national-state policy dialogue and implementer of national policy initiatives. While extreme challenges continue to exist and threaten capacity gains, the EOP review team concludes that capacity gains are deepening in a manner that will permit TAP or future CB programs to begin shifting program emphases to new areas, while consolidating gains made in areas such as planning and budgeting. END-OF-PROJECT REVIEW viii Internally, the most significant constraints to program effectiveness, including the implementation of MTE recommendations, have been challenges of recruitment and retention. Externally, the greatest threat to continuing accomplishment is the absence of government funding that would enable SMoEs to put acquired skills into practice. Ministry staff may be content for some time to train with limited opportunity to exercise their new skills, but eventually professional integrity is undermined and corruption invited if funding to support service delivery does not materialize. 3.2 Strategic Alignment The design and objectives of TAP operate in very close alignment with the strategic frameworks of both the GoSS and USAID, both of which strongly emphasize international partnership, capacity building, and institutional strengthening for the delivery of education services. The value of that alignment ultimately depends, however, on how well the strategy functions in practice: namely, how effective has TAP been in contributing to the strategic goals of both USAID and the GoSS? The remainder of this document focuses (through examination of MTE recommendations and analysis of partnership and sectoral coordination) on the degree to which TAP has improved its contribution to institutional capacity building in Sudan and, by extension, to the strategic goals of USAID and the GoSS. 3.3 Implementation of MTE Recommendations The EOP review team found that few recommendations were implemented precisely as specified by the MTE. The team chose to assess the effectiveness of the implementation not by the rigor, specificity, or completeness by which it was being or had been implemented, but rather by the progress being made in the underlying problem area. Therefore, the MTE recommendations are reclassified below to illustrate progress being achieved by TAP in various areas. The EOP review team concludes the following: Strong Progress was made in the following area: (1) Program Management and Monitoring & Evaluation. TAP implemented a program logical framework named the System Strengthening Framework (SSF). The value of the SSF went well beyond M&E improvements, contributing directly to the possibility of strengthened and sustained capacity building impact. (Recommendation 8) Modest, Stable or Mixed Progress was made in the following areas: (1) Operations Management. The post of Deputy Chief of Party, Operations (DCOP) was created and filled in May 2010. The new post served (as intended) to relieve the burden on the COP, affording the incoming COP in June 2010 with the opportunity for improved strategic attention to harmonizing TA activities and processes as well as for strengthening the working relationship with GoSS at the Ministry of Education. The recommendation to create a full-time STA position at the MoE has not yet been fully implemented. (Recommendation 3) (2) Partnership and Sectoral Coordination. TAP has played an increasingly instrumental role in fostering communication and coordination between the GoSS and SMoEs, as well as facilitating and building END-OF-PROJECT REVIEW ix capacity for improved relations between state ministries and other donors. TAP made an effort to conclude discussions and signing of a Memorandum of Understanding (MoU) between the GoSS and AED, but the effort was stalled at the MoE. (Recommendations 1 and 2) (3) Conflict-sensitive programming. Clear progress has been made in assessing risk and planning for potential conflict-driven disruptions to work from the viewpoint of staff safety and security. Less progress has been made in assessing risk and responding to the limitations conflict imposes on capacity building efforts and educational outcomes. (Recommendation 6) Limited Progress was made in the following areas: (1) Gender Equity Strategy. There is profound recognition amongst TAP staff of the challenges of improving gender equity. However, few solutions have been proposed beyond references to possibilities for ―gender mainstreaming.‖ TAP has proposed hiring a gender specialist on an STTA basis to assist with creative thinking in this area. (Recommendation 7) (2) Exit and Sustainability Strategy. Little formal or informal discussion of exit criteria seems to occur among AED staff, most of whom remain highly focused on the still enormous needs for continued capacity building at SMoEs. The FY 2011 work plan suggests some possibilities for building a sustainable TA capacity at MoE, and these and other ideas should be pursued. (Recommendation 9) (3) Recruitment and Retention. Limited progress has been made in the critical area of recruitment and retention, with some unintended and negative consequences resulting from implementation of the recommendation to reposition DTAs as full-time POEs. The redefinition of the DTA role as POE (along with change in payment structure from US$ to Sudanese pounds) reduced compensation for all Sudanese staff and contributed to a disruptive loss of over half of TAP staff. Strongly negative perceptions of AED‘s intent persist among TAP staff and at the MoE, with ongoing damage to working relationships.(Recommendations 4 and 5) 3.4 Sectoral Coordination and Partnership TAP advisors are embedded directly in SMoEs placing the program in a unique and distinctive position vis-a￾vis its multiple partners. The relationship between TAP and state ministry staff has been, with few exceptions, strong and productive. Solid and respectful relationships between advisors and ministerial staff have been foundational to capacity gains. TAP actively facilitates the relationship between multiple partners and the SMoEs, contributing directly and significantly to improved sectoral coordination. TAP has played a significant role in building capacity at the state level to coordinate donor relations. Cooperation between TAP and other NGOs and donor partners has been instrumental to sectoral success for a number of projects—most notably, facilitation of the UNICEF-funded EMIS and the Booz & Co.–led electronic payroll system. The collaborative advantages of cooperation across USAID-funded projects as noted above (i.e., BRIDGE, GEE, and SSIRI) are more limited than suggested by the MTE and for this and other reasons, less cooperation has emerged. END-OF-PROJECT REVIEW x IV. Implications for Future Strategy and Programming Initial Recommendations Despite constraints, variability across states, and even missteps, there is evidence to suggest that the capacity building gains are real and deepening. The team therefore concluded that it was in the best interest of the continuity of the capacity building effort in South Sudan to provide recommendations that would (1) serve TAP should USAID decided to extend the current program; and (2) incorporate lessons learned from the TAP experience into future design considerations for the period after the new Mission strategy begins to be implemented. The following 12 recommendations follow from the initial two above. Strategy and Program Management Recommendation 1. If USAID decides to continue TAP, the program should increase its support for teacher management capacity building activities (within the Schools and Materials Management component) during the transition period and assess the viability of these efforts for future programming. Recommendation 2. If continued, TAP should explore means and financial requirements for increasing activity at the county level. Recommendation 3. USAID should consider including funds for a micro-grant or similar program in future CB projects. Recommendation 4. USAID should encourage SMoE contributions to the development of the Education Strategy for South Sudan (ESSP) with TAP facilitation, if the program is extended. Recommendation 5. TAP, if continued, should renew its efforts at improving gender equity. Future programming should include specific plans and devise indicators for measuring gender equity and equality gains. Recommendation 6. Further steps should be taken to strengthen M&E: in particular, methods for making the IDF tool more reliable for comparative purposes should be investigated. Any future program should establish a Program Logical Framework at the outset of the program for both M&E and program management purposes. More specifically, USAID should shift focus from reliance on standard output indicators and encourage the use of outcome indicators in future CB programs. Recommendation 7. If the program is continued, current TAP advisors should work with SMoE staff to identify three or four firm 1-year and 2-year ―exit targets‖ using the IDF instrument. USAID should request that any future CB program have in place an exit strategy drawn up in the context of the long-term nature of capacity building in post-conflict settings. END-OF-PROJECT REVIEW xi Coordination and Relationships Recommendation 8. Any future CB program should negotiate with the RSS MoE for signature of an MoU to increase the efficacy of capacity building efforts by nurturing dialogue and establishing expectations. Recommendation 9. USAID should support TAP, if continued, and future programs in establishing an STA position at the RSS MoE level. Human Resources and Organizational Structure Recommendation 10. TAP (if continued) or future programs should make even greater efforts to employ best recruitment and retention practices in the transition period; that future efforts consider various possibilities for recruitment adjustments (detailed below); and that TAP revisit its personnel and contract decisions for Sudanese STAs and POEs. Recommendation 11. Future CB programs should consider alternative structural configurations for the efficient and effective deployment and pairing of scarce STA and POE resources. Recommendation 12. Future CB programs should make internal capacity building of its current and future staff an explicit and formal part of their programming. END-OF-PROJECT REVIEW 1 1. INTRODUCTION 1.1. Country Context After half a century of struggle, South Sudan today stands at a historical threshold. Since 1955, with only a decade of troubled peace between 1972 and 1983, the South Sudanese people have suffered from two prolonged periods of civil war between the largely Arab and Islamic north and the predominantly black, Christian, and animist south. In 2005, the disputing parties signed the Comprehensive Peace Agreement (CPA), which called for a six-year period of semi-autonomous government in the south to be concluded with a referendum on independence. In January 2011, the South Sudanese voted overwhelmingly for the creation of an independent state. On July 9, 2011, with the formal declaration of independence, South Sudan became the world‘s newest nation. An interim constitution provides the government of the Republic of South Sudan (RSS)3 with the basic institutions of governance and will continue to guide the government until at least 2013. The constitution stipulates three levels of decentralized government: national, state, and local. Accordingly, South Sudan is decentralized into ten states: Western Equatoria, Central Equatoria, Eastern Equatoria, Jonglei, Lakes State, Upper Nile, Warrap, Western Bahr el-Ghazal, Western Upper Nile, and Northern Bahr el-Ghazal. The states are further subdivided into counties, payams (district), and bomas (village). Population dynamics in South Sudan are highly fluid as mobile pastoralist communities and large refugee and IDP flows merge within sometimes ambiguous borders. As a result, population estimates for South Sudan differ enormously, ranging from 9 to 14 million. Referring to the 2008 census, the Statistical Yearbook for Southern Sudan, 2010 puts the total population in South Sudan at 8,260,490.4 More than three-quarters of the population depends on crop farming and/or animal husbandry as their main livelihood. The decades of war and underdevelopment—as well as frequent drought, floods, and famine—have had a devastating impact on social, political, and economic structures in South Sudan. The nation currently has among the lowest human development indicators in the world: only 27 percent of the population (over 15- years old) is literate, and over 90 percent of people in South Sudan live on less than $1 per day. One out of every six women who become pregnant will die in (or as a result of) childbirth, and one in six children die before their first birthday. Although some progress has been made in fostering social, political, and economic stabilization since the signing of the CPA, numerous challenges persist. Independence has created high expectations among citizens for tangible peace dividends, transparent sharing of power and wealth, and an end to insecurity. However, the delivery of basic services such as education, health, and infrastructure by government is extremely limited, hampered by a cavernous gap in human and institutional capacity at all levels of government, as well as by continuing security concerns. The capacity building gains that must take place across all sectors to enable the 3 Throughout the report, the term – ―Republic of South Sudan‖ or RSS – is used when describing actions/events from July 9, 2011 onward. When referring to past actions/events (i.e. prior to July 9, 2011) the term ―Government of South Sudan or GOSS is used. 4Southern Sudan Center for Census, Statistics and Evaluation (SSCCSE), Statistical Yearbook for Southern Sudan, 2010 (2011). END-OF-PROJECT REVIEW 2 country to establish itself as an independent state are enormous. Capacity building is taking place across many, if not all, sectors, but there is little doubt that long-term national development must ultimately rely on the nurturing of educated citizens to fill essential roles in society. Education Sector Status5 In the education sector alone, the magnitude of challenges facing the government is enormous. From 2005 to 2009, student enrollment doubled to 1.3 million counting only primary schools, with enrollments increasing also in secondary and vocational schools. Still, more than half of primary school age children do not attend schools; most who enter do so late and do not complete the full primary cycle. Girls face particularly extreme disadvantages, entering school in fewer numbers, entering later in the primary cycle, and dropping out earlier than boys. On almost every indicator for educational access, South Sudan is at or near the bottom. Less than one-quarter of primary schools offer the full primary cycle through Grade 8. Forty percent or more of the teachers have had only primary school education. While an estimated 90 percent of teachers are competent in Sudanese Arabic, only about 20–30 percent have even rudimentary language skills in English, which is deemed the medium of instruction from Grade 4 onwards. The average pupil-teacher ratio is 52:1, and is as high as 200:1 in Jonglei state. The learning environments, moreover, are inadequate. Despite increasing classroom numbers, the three median states (Eastern, Western, and Central Equatoria) have almost 70 percent of learning spaces described as tents, roof-only structures, open-air spaces, and semi-permanent classrooms. In states such as Unity, this can be as high as 86 percent. The Sudanese People‘s Liberation Movement (SPLM) Secretariat of Education expended much effort to create and implement a strategy for providing basic education and life skills training during the most recent war and in the immediate post-conflict period. Since the signing of the CPA, the fundamental building blocks of the sector have been put in place, specifically: (i) the establishment and staffing of the newly established Ministry of Education, Science, and Technology (renamed the Ministry of Education in 2010) and the State Ministries of Education (SMoEs); (ii) the development of a number of necessary legislative, policy, and other documents including the draft Southern Sudan Education Act (drafted 2008); (iii) the establishment of the ―educational cycle or ladder‖ of eight years of primary (basic) education for learners aged six to thirteen (but also including non-formal education), four years of secondary education, and four years of tertiary education; and (iv) the establishment of sector planning and development mechanisms such as the Education Rehabilitation and Development Framework meetings and thematic working groups. The education sector, however, is still in its infancy in terms of its capacity to manage education service delivery to the current primary education student population and its capacity to expand the provision of quality education to other South Sudanese interested in learning. Funding for education remains insufficient. In 2010, for example, 7 percent of the national budget was directed toward education, amounting to just 315 million Sudanese pounds; in the past, over three-quarters of those funds have been spent on salaries, leaving 5 Statistics for the education sector are drawn from two sources: (1) UNESCO (2011). Education For All Global Monitoring Report 2011: Building a Better Future for Education for an Independent South Sudan; and (2) Fast Track Initiative/World Bank (June 2011). Education in South Sudan: Status and Challenges for a New System Draft Final. END-OF-PROJECT REVIEW 3 very little for other administrative costs. Just as necessary as funding for effective service delivery and long-term development is institutional management capacity. Capacity for evidence-based planning—and for the management of finances, donors, human resources, and schools—is lacking. In such circumstances in the past, many development partners have largely provided their own standalone solutions to education service delivery, at best only informing SMoEs of their activities and, at worst, bypassing the government altogether. This scenario exacerbates the already daunting challenges facing the new ministries. The training of administrators and managers then is of key importance. In most cases administrators, despite having genuine interest in developing the area under their care, have not had the luxury of training in such areas. While there is evidence of strong leadership in the education sector in some states, these leaders also tend to be vulnerable to the changing political scene including wide-scale, sweeping changes of administrators in some SMoEs. These displacements create significant challenges to building the strength of the sector. The challenges of seeking to nurture the education sector in this environment cannot be overstated. 1.2. Program Description In September 2005, USAID established the EQUIP2 Southern Sudan Technical Assistance Program (SSTAP or TAP) to build the capacity of the newly created Ministry of Education, Science, and Technology. SSTAP built on an already substantial investment in education by USAID in South Sudan. The progress of SSTAP has occurred in two distinct phases. Phase 1 (2005- 2007): National Focus In the first phase, SSTAP focused on establishing the ―machinery of government‖ necessary for the functioning of the newly created central-level Ministry of Education, Science, and Technology (MoEST). SSTAP worked primarily through short-term technical assistance (STTA) in this first phase. During this time, the project was engaged in four distinct sets of activities: (1) providing ongoing advice and technical assistance to the newly appointed minister of education and his undersecretary in areas that would build MoEST structure and capacity; (2) providing planning and support and technical assistance to new MoEST Department Directors in areas of quality and innovation, planning, administration, and finance; (3) providing technical assistance to other key program initiatives; and (4) facilitating communications and collaboration with USAID and other donor education programs.6 Phase 2 (2007—2011): State Focus In its second phase, SSTAP2 shifted its focus to capacity building at the state level, now operating primarily through long-term technical assistance (LTTA). Capacity building efforts were consolidated into five key areas in FY 2010: Planning and Budgeting Financial Management Communication and Coordination 6 USAID, AED TAP Award Document, August 6, 2007. END-OF-PROJECT REVIEW 4 Human Resources School and Education Material It is this second phase that is the subject of this EOP review. Although SSTAP2 is technically the most accurate acronym for the program, it is commonly referred to simply as TAP. This review henceforth refers to the program as TAP, unless there is a need to make a distinction between the first and second phases. The TAP capacity building model follows a mentoring approach that supports on-the-job training and learning-by-doing. In this model, state advisors—a Senior Technical Advisor (STA) and/or a Diaspora Technical Assistant (DTA), subsequently redefined as Project Officer for Education (POE)—are fully integrated into SMoEs and work side by side with ministry staff to build their professional skill sets, and by extension, the institutional capacity of the MoE and SMoEs. (See Annex 3 for Job Descriptions for STAs and POEs.) In most of the SMoEs, the desk of the STA and/or POE sits in the Directorate of Planning and Budgeting. Such intensive administrative support was deemed necessary if rapid expansion of educational services was to be implemented in less than five years. In addition, TAP conducts and facilitates a large number of formal trainings and workshops and assists with the preparation of SMoE and MoE laws, guidelines, and documentation. The following TAP figure diagrams the logic of improving capacities in each of the five domains in order to strengthen ministerial capacity to deliver quality educational services: END-OF-PROJECT REVIEW 5 In this second phase, an STA and DTA/POE were to be placed in all ten states. The program has never achieved the complete staffing ideal of an STA and a POE in each state. SSTAP2 currently is managed under a Cooperative Agreement with a cumulative budget of just over $17 million; The Academy for Educational Development (AED) is the lead implementing agency, partnering with the International Rescue Committee (IRC) in the state of Northern Bahr el-Ghazal. TAP is scheduled to end on September 30, 2011. END-OF-PROJECT REVIEW 6 2. PURPOSE AND METHODOLOGY OF THE REVIEW 2.1. Purpose and Scope of the End-of-Project Review The purpose of the TAP end-of-project (EOP) review was: ―to assess performance in implementing programmatic changes in accordance with findings, conclusions, and recommendations reported in the midterm evaluation of TAP, conducted in 2009.‖ In support of this purpose, a series of questions were posed in the Scope of Work (Annex 1): Implementation of MTE Programmatic Changes  Describe the changes in programmatic direction and scope implemented in response to the recommendations of the MTE.  What are the key internal and external opportunities/constraints that TAP has leveraged/suffered from in responding to the MTE recommendations?  What has been the impact of the implementation of these recommendations on the overall effectiveness of the TAP approach in building and sustaining institutional capacity development?  What are the implications of implementation constraints/opportunities for any future programming modeled on the TAP approach? Strategy Alignment  How well does the TAP model/approach fit within USAID‘s Education Strategy, generally, and the Mission‘s education strategy in particular?  How well does the TAP design and objectives align with the South Sudan Development Plan‘s program objectives for education?  How should any similar future USAID investments be designed/implemented to enhance alignment and contributions of USAID assistance to the achievements of key education targets in the SSDP? Coordination and Relationships  What is the nature and quality of the relationships between TAP and its local partners, communities, other USAID/USG cooperating agencies, other NGOs and donor partners? The findings and conclusions established in the course of answering these questions were intended to document lessons learned and provide recommendations to inform evidence-based future programming. 2.2. Review Methodology The methodology of the review was fundamentally and predominantly qualitative, designed for the examination of a specific set of questions focused (as noted above) on (1) strategic alignment, (2) implementation of MTE recommendations, and (3) the efficacy of coordination and relationships, all with an eye toward future programming. As such, it was not designed to be an impact evaluation as commonly understood. The limitations for such an evaluation include factors noted as common in post-conflict settings END-OF-PROJECT REVIEW 7 (e.g., ―foreshortened timeframes‖ requiring unanticipated strategic shifts)7 as well as those common to measurement of individual and institutional capacity building (e.g., instrument subjectivity).8 Despite these limitations, the review sought maximum coverage within the constraints of the time and travel allowed. Salient perspectives were gathered at the national level in Juba, as well as at the state level through visits to three of the ten states. The limitations imposed by the team‘s inability to travel to additional states was offset to a large extent by the decision to bring STAs and POEs from the states not visited to Juba for interviews and group discussions. The team interviewed over 50 individuals during its two weeks in South Sudan (Annex 2). This methodology is appropriate given the purpose and scope of the review in the three areas noted above. The review was conducted from June 15, 2011, through July 1, 2011. The methodology and subsequent analysis was heavily grounded in the findings and recommendations of a comprehensive MTE completed in 2009. Additional data was gathered through: (1) Intensive examination of internal and external program-related documents. (2) Key informant and focus group interviews in Juba, including: a. Government officials at the Ministry of Education, b. AED and South Sudan Interactive Radio Instruction program staff, including all but one STA, and c. Representatives of collaborating organizations. (3) Site Visits in Western Bahr el-Ghazal, Northern Bahr el-Ghazal, and Central Equatoria states during which the team9 : a. Held exploratory and focused discussions with ministry staff at the director, deputy and inspector levels, b. Conducted focused interviews with ministers, directors general, and directors, and c. Observed office space and computer usage patterns. (4) Review of relevant literature in a number of areas, including: a. Development in conflict-affected areas, b. Education in post-conflict settings, and c. Evaluation of technical assistance and capacity building. The evaluation process was guided by Management Systems International‘s (MSI) quality assurance guidelines, which stipulated a team planning meeting (TPM) at the outset of the evaluation. This TPM was facilitated by MSI and attended by the full team of external evaluators, a GoSS representative, a USAID representative and an AED staff representative. The SOW was finalized at the TPM and was followed by the development of interview protocols and other evaluation guidelines prepared for fieldwork and for ongoing analysis. 7USAID, ―Monitoring and Evaluation in Post-conflict Settings,‖ (March 15, 2006). 8 USAID, ―Government of Southern Sudan: Strategic Capacity Building Study‖ (June 15, 2010) 9 WBG and NBG were selected prior to the arrival of the team, largely for logistical reasons and provided extreme and useful contrasts. Interviews at the CEQ SMoE were added on request of the review team after arrival in Juba, recognizing the opportunity presented by the physical location of the SMoE for CEQ in Juba. END-OF-PROJECT REVIEW 8 On return from the final field visits, the evaluation team conducted an abbreviated ―Findings, Conclusions, and Recommendations (FCR)‖ workshop in preparation for debriefs of the review to USAID (June 29, 2011) and the GoSS (June 30, 2011). This process concluded with the report that is presented here. END-OF-PROJECT REVIEW 9 3. MAJOR FINDINGS AND CONCLUSIONS 3.1. Strategic Alignment An effective partnership between USAID and the GoSS necessitates that programs and projects funded by USAID be well aligned with the strategic frameworks of the GoSS. The programs themselves are most productively coordinated within the Mission portfolio when they align well with the USAID Education and Sudan Mission Strategy. Accordingly, the SOW asked two important questions: (1) How well does the TAP design and objectives align with the South Sudan Development Plan‘s (SSDP) program objectives for education? (2) How well does the TAP model/approach fit within USAID‘s Education Strategy, generally, and the Mission‘s education strategy in particular? 3.1.1. Strategic Alignment with GoSS Strategic Frameworks As of the writing of this review, the 2011 South Sudan Development Plan (SSDP) was the most recent public document informing South Sudan‘s educational strategy.10 Here within the ―Social and Human Development Pillar,‖ the SSDP described the current low levels of education and laid out goals for greater primary and secondary school enrollment and for the furthering of human and national capacity to address these needs. Specifically, capacity building is to take place at three levels: (1) improving institutional systems, administrative management, and organizational processes—accomplished through planning and financing, implementing programs, developing institutional and legal frameworks, and quality assurance and manager training; (2) improving the qualification of social service providers through formal and in-service training and task shifting, exposure to other country experiences, and coaching; and (3) building the capacity in communities to empower households to manage their livelihoods, claim their rights and ultimately realize their full potential. The ―SSDP Medium-Term Capacity Development Strategy‖ further emphasizes the importance of working with international partners in addressing critical institutional capacity needs to make SSDP succeed and to make statehood viable in the immediate post-independence period (2011–13) in a coherent and effective manner.11 It focuses on five key need areas: (1) strengthening institutional and organizational frameworks, (2) strengthening research and planning mechanisms, (3) establishing mechanisms to enable oversight and regulation, (4) improving the qualifications of social service providers, and (5) building community capacity. 10Government of Southern Sudan, ―Southern Sudan Development Plan, 2011‖ (2011); Attention must be paid to current discussions taking place to formulate an updated Education Sector Strategic Plan (ESSP): in contrast to the SSDP, this document incorporates input from the states and seems to have received favorable attention. TAP has facilitated on this issue as well, but no drafts of the document were as yet available at the time of this review. 11 Government of Southern Sudan, ―SSDP Medium-Term Capacity Development Strategy,‖ Third Draft, May 17, 2011. END-OF-PROJECT REVIEW 10 TAP programming aligns well with the emphases of the SSDP. TAP‘s focus on institutional management across a variety of areas—institutional systems, administrative management, and organizational processes as they relate to planning, financing, program implementation, institutional frameworks, and quality promotion and manager training—fall well within the strategy framework of the SSDP. Likewise, the TAP model, which focuses not on direct delivery but on capacity building, seeks to improve the qualification of social service providers through the learning-by-doing approach through dialogue, modeling, and facilitation of collaboration with other donor partners. Indeed, capacity building in the coordination of multilateral partners has been a significant aspect of the TAP program. This approach is in line with SSDP guidelines. Also, TAP programming in the area of planning and budgeting, procedures of oversight and regulation, and quality promotion align well with the SSDP guidelines. On the ground, strategic priorities in education in South Sudan are regularly and almost universally rephrased around three fundamental needs: (1) Qualified teachers (2) Improved learning spaces (3) Adequate materials Indeed, it is these three priorities that animate discussion both among GoSS officials and officials at state ministries. In turn, AED staff regularly raised the issue of how TAP might more directly contribute to these critical needs by increasing engagement in teacher management activities, necessitating greater involvement at the county level. Strategic Focus 1 Develop national vision for institution building drawing on priorities identified in national frameworks Strategic Focus 2 Develop framework for maximizing impact and sustainability of capacity development efforts Pillar 4. Social & Human Development 1. Education Sectoral Strategy Attributes 1. Creating enabling environments: Policy & legal frameworks 2. Implementation: Institutional structures & mechanisms 3. Performance: Human resources development Other Pillars 1. Governance 2. Economic Development 3. Conflict Prevention & Security Other Sectors within Education Pillar 2. Health 3. Social Protection Five areas for development 1. Strengthening institutional and organizational frameworks 2. Strengthening research and planning mechanisms 3. Establishing mechanisms for oversight and regulation 4. Improving qualifications through formal and in-service training 5. Building capacity in communities South Sudan Development Plan (2011) Mission Accelerate universal access to basic social services by building necessary institutional, human, social and economic capacity Programme Areas 1. Enhancing Education Quality 2. Increasing access and improving efficiency of the educational system 3. Enhancing literacy and alternative education 4. Enhancing institutional and human capacity 5. Expanding access to quality higher education END-OF-PROJECT REVIEW 11 3.1.2. Strategic Alignment with USAID’s Mission and Education Strategy The principles and focus areas of the 2011 ―USAID Education Strategy, 2011–2015‖ includes the desire to maximize ―impact and scale‖; a focus on Sub-Saharan Africa; a focus on crisis and conflict-affected countries; and collaboration and coordination with bilateral and multilateral donor partners.12 The strategy document further emphasizes: (1) greater reliance on host country leadership through engagement of its governments and civil society contributions; (2) a greater emphasis on communication, collaboration, and coordination with donors, host country governments, and other in-country partners; (3) the use of innovative technology; and (4) the implementation of enhanced evaluation practices. A number of goals and intermediate results (IRs) are relevant to the TAP. For example, IR 1.2 notes enhanced capacity in the supply, distribution, and use of learning materials. Most obviously relevant is IR 3.3 under Goal 3, ―Institutional capacity to provide services strengthened.‖13 Illustrative activities include strengthening the school system, monitoring and evaluation, and recruitment and compensation systems. For the achievement of any of these goals, efficient and effective governance and management infrastructure for education is essential. Among the key development objectives in the ―USAID Transition Strategy for South Sudan, 2011–2013‖ are ―Effective, Inclusive, and Accountable Governance Strengthened‖ (D02), and ―Essential Services Developed and Maintained‖ (D03).14 Under the first category is a focus on institutions—―capacity building . . . focus[ing] on core governance institutions that are vital to effective government‖ (IR2.2). This seeks to support ―national institutions and processes of governance, particularly decentralization efforts.‖ These efforts include the establishment or improvement of legislative and policy frameworks, streamlining of institutional structures for effectiveness, strengthening of key planning and decision-making processes, and improvement of managerial, technical, and basic skills of managers and staff. A second broad area touches on systems and functions—―strengthening specific components of essential service delivery‖ (IR3.1). Under this 12 USAID, ―USAID Education Strategy: Opportunity Through Learning, 2011–2015,‖ (February 2011). 13 Discussion of how contributions to IR 3.3. are measured is discussed in conjunction with M&E analysis. 14 USAID, ―USAID Transition Strategy for South Sudan, 2011–2013,‖ (May 2011); This document explicitly notes the consonances of this strategy with the three-year development plan being worked on by GoSS with its four pillars touching on governance, economic development, social and human development, and conflict prevention and security. USAID Education Strategy, 2011-2015 END-OF-PROJECT REVIEW 12 category is improving and expanding delivery of essential services; under this, in turn, is the enhancement of the professional capacity of service providers, and nurturing collaboration with public-private partnerships. Yet another area under this category is the strengthening of planning and management capacity, as well as development of sector strategies and policies based on analysis (IR3.2). The TAP model and approach fulfills the vision of both USAID‘s Education and Mission Strategies well. By embedding STAs and POEs in the ministries of education at the national and state levels, TAP relies on host country leadership and begins to build institutional capacity at the national, regional, and state levels to affect systemic changes. TAP programming has involved working closely with directors general, directors, deputy directors, and inspectors to build capacity including in the areas of planning and budgeting, financial management, facilitating the implementation and practical use of technologies such as EMIS and the electronic payroll system, developing capacity for donor coordination, facilitating communication between national and state level ministries, quality promotion through school inspections, educational material management, and enhanced evaluation practices to enable the measurement of ministry capacity across a wide number of indices. This model, which focuses on capacity building to affect systemic changes, seeks sustainable development in educational infrastructure with hope of achieving impact and scale. Finally, the extension of activities to the state level furthers the goal of decentralization, empowering the states to function effectively as part of national governance. In conclusion, by design TAP operates in very close alignment with the strategic frameworks of both the GoSS and USAID. The value of that alignment ultimately depends, however, on how well the strategy functions in practice: namely, how effective has TAP been in contributing to the strategic goals of both USAID and GoSS? The remainder of the document addresses that question, among others, in its examination of the MTE recommendations and analyzing partnership and sectoral coordination for a better understanding USAID Transition Strategy for South Sudan, 2011-2013 END-OF-PROJECT REVIEW 13 of the degree to which TAP has improved its contribution to institutional capacity building for educational service delivery in Sudan. 3.2. Implementation of MTE Programmatic Recommendations 3.2.1. General Discussion Midterm evaluations provide one of the best opportunities in the evaluation cycle to directly improve the implementation of projects and programs. This EOP review completes this important cycle by assessing whether the achievement of project goals was improved as a result of implementing the recommendations offered in the MTE. In this case, the MTE of TAP was conducted in July/August 2009, and nine carefully delineated recommendations were finalized in three areas: Strategy, Human Resources, and Impact. Table 1. Midterm Evaluation Recommendations. Strategy Recommendation 1: Request AED TAP to review and refine the draft Memorandum of Understanding. Recommendation 2: Request AED TAP to articulate its approach (including an action plan) to progressing the anticipated partnerships between State Ministries of Education and donor/development partners and between TAP and other USAID nominated activities (BRIDGE, GEE and SSIRI) and ensure that TA personnel are familiar with and supportive of the directions proposed. Human Resources Recommendation 3: Request AED TAP-Washington to reassess the level of TAP Hub (central program office) resourcing and to specifically ‗uncouple‘ the Chief of Party (COP) role from the Senior Technical Advisor role, with both becoming separately staffed, full-time positions. Recommendation 4: Request AED TAP to resolve outstanding personnel well-being and contractual issues, including revisiting and refining position descriptions. Recommendation 5: Request AED TAP to identify and implement a strategy to support current TA personnel on the job. Impact Recommendation 6: Request AED TAP to immediately draft: (i) a sustainability and exit strategy, taking into account TAP‘s ending date of September 2011, as well as the long-term considerations associated with a realistic capacity building timeframe (conservatively 10–20 years); and (ii) a standard risk management matrix. Recommendation 7: Request AED TAP to incorporate conflict-sensitive analysis into its programming, utilizing one of the many tools available, and to report on the cumulative amount of time TA personnel are kept off or are limited on the job due to insecurity, since this factor has significant implications for TAP impact. Recommendation 8: Request AED TAP to introduce a program Logical Framework or Results Framework that grounds qualitative findings in empirical data and sound research methods and improves monitoring and evaluation to take into account (i) a clear M&E approach (and, ideally, a framework) and (ii) analyses and meta-analyses of data and inclusion of these findings in reports. Recommendation 9: Request AED TAP to articulate a gender and development support strategy in collaboration with GEE that is informed by the work being undertaken by GoSS bodies, other development partners, and projects. END-OF-PROJECT REVIEW 14 The EOP team was asked to: (1) describe the implementation of these recommendations; (2) identify the constraints and opportunities presented in responding to these recommendations; and (3) assess the likely effect that these recommendations had on improving human and institutional capacity. The MTE recommendations were designed to address identified problems and perceived program weaknesses, putting TAP on track for greater impact in its final two years of implementation. These recommendations represented the informed judgments of an evaluation team with strong project development and education sector experience. It is incumbent upon program implementers to fully consider all recommendations; however, implementation partners may have good reasons for modifying the parameters of particular recommendations, or, in their more complete understanding of the local context, may seek other means to address identified problems. The EOP team found that few recommendations were fully implemented as specified by the MTE. Some recommendations were not entirely realistic in their expectations (discussed below). Additionally, TAP experienced a number of significant constraints that contributed to delays or limited the intensity of the efforts to implement the MTE recommendations. The time available to implement recommendations was somewhat limited. Upon completion of the final draft of the MTE report, lengthy discussions ensued between the evaluation team, USAID mission staff, and the implementing partner, AED. As a result, the final MTE report was not formally released until 10 months after the initial fieldwork, leaving AED less than 12 months to formally implement some of the recommended changes. In that year, a new COP was assigned to TAP, beginning his one-year appointment in June 2010. Recommendations were, of course, known to the TAP leadership before the official release of the report, allowing them to proceed at a more informal level to implement them. However, it also appears that the delay and imminent departure of the outgoing COP may have reduced the commitment to some of the more paperwork-intensive recommendations, such as the drafting of a gender equity strategy or an action plan for ―progressing‖ partnerships. The other matter of some significance is that, worldwide, AED was placed on formal suspension by USAID in December 2010, restricting its ability to make substantive changes or contract for STTA support. It is commonly understood that AED‘s assets and ongoing project commitments will be assumed by another agency at some point in the near future.15 This agency will most likely have a strong working relationship with USAID but perhaps will have different strategic commitments and human resource practices. Finally, the January 2011 referendum focused government attention away from sectoral concerns for a number of months, while security concerns surrounding the referendum period limited staff ability to perform their responsibilities. In the final analysis, the team chose to assess the effectiveness of the implementation not by reference to the rigor, specificity, or completeness by which it was being (or had been) implemented, but rather by the progress being made in the underlying problem area. Therefore, the MTE recommendations are reclassified below to illustrate to what extent TAP is making progress in identified problem areas. From that analysis, the EOP team draws lessons learned for future program design, and in the final section of the paper incorporates 15 FHI Development 360 (FHI 360) has acquired the programs, expertise, and assets of AED as of June 30, 2011. END-OF-PROJECT REVIEW 15 those broad lessons and some more specific findings into a series of recommendations, should funding continue past the formal end date of September 2011. 3.2.2. Areas of Strong Progress Program Management and Impact Program Logical Framework The MTE requested AED TAP to ―introduce a program Logical Framework or Results Framework that grounds qualitative findings in empirical data and sound research methods and improves monitoring and evaluation‖ (Recommendation 8). This resulted in the completion of a Performance Management Plan (PMP), including a ―Systems Strengthening Framework‖ that has had a profound effect on how TAP operates, greatly increasing the possibility of sustained improvements in individual and institutional capacity building. The value of this recommendation far exceeded its intent to improve M&E systems. The System Strengthening Framework (SSF) was intensively disseminated among staff and actively supported by the new COP in all planning and cross-state meetings. The framework rationalized 18 activities into five coherent components. Additionally, it has the virtue of mapping well onto the SMoEs‘ new organizational structure, as components closely parallel the departmental structure of SMoEs. This has served to harmonize technical assistance (TA) activities across states, and is of particular value to POEs who are operating without END-OF-PROJECT REVIEW 16 STA support in a state. Additionally, in at least some states, it is regularly used as a systems map for ministry officials to think about their own capacity building needs and objectives. The SSF also very clearly improved M&E practices at TAP. The SSF was accompanied by the development of a program Logical Framework (see figure on page 4) and a comprehensive PMP comprising USAID standard indicators, custom indicators, and a set of project-level process indicators. One of the customized indicators for capacity building measures institutional development through the use of a tool called the Institutional Development Framework (IDF) (Annex 4). The IDF is fundamentally an organizational development tool and, as such, is not a fully objective instrument that allows for comparison of capacity achievements across states, nor does it yet allow for comparison of capacity over time. However, it acts as an important capacity building tool and holds some promise for comparative use in the future. The IDF has been administered twice in most states. Such efforts are a testimony to strengthened M&E and will provide TAP and future evaluators with important information on capacity building efforts. STAs and POEs uniformly reported that state ministerial personnel were able to assess their capacity more honestly (and effectively) the second time around, producing the possibility that a downward trend in the index could mean greater capacity. Such willingness to acknowledge that initial estimates of capacity were overstated is evidence in its own right of the increasing sensibilities of state ministry staff. Lessons Learned for Future Program Design: The value of a program Logical Framework can go well beyond its contributions to M&E. This is especially true in the case of CB projects or programs such as TAP, where TAs have highly variable skill sets, are working in places of highly variable absorptive capacity, are working at great distances from one another, and regularly experience competing demands on their time and attention from their local counterparts. Any future program should establish a program Logical Framework at the outset of the program for both M&E and program management purposes. 3.2.3. Areas of Modest, Stable, or Mixed Progress Operations Management Role of Chief of Party The MTE concluded strongly that then-COP—who was acting both as COP with responsibility for programs and operations, as well as providing STA services at the GoSS and itinerantly in other states— was severely over-tasked. Accordingly, the MTE requested that: ―AED TAP-Washington reassess the level of TAP Hub (central program office) resourcing‖ and that it specifically ‗uncouple‘ the COP and STA roles, with both becoming separately staffed, full-time positions (Recommendation 3). This (and a related) recommendation received considerable attention at the time of MTE, and eventually became reinterpreted as a recommendation to create three positions: COP, Deputy Chief of Party (DCOP), and STA at the national level. This recommendation could not be implemented without significant budget realignment, and funding was allocated for the newly created position of Deputy Chief of Party (DCOP) for operations. This position was finally filled in May 2010 (approximately six months after the formal release of the MTE) and, accordingly, END-OF-PROJECT REVIEW 17 the operations management burden for the COP decreased. Logically, this allowed the COP to focus greater attention on strengthening program effectiveness. The second COP (who began his tenure in June 2010) appears to have been able to focus more strategically on program development, including active commitment to the SSTF as well as a number of other initiatives, including formal consultation with state ministry staff as part of the TAP FY 2011 work planning effort; creative attention to both the content and process of technical assistance in education; and on improving working relationships at GoSS, which had become strained during the tenure of the outgoing COP. There has been some disagreement across COPs and among USAID staff regarding the value of the STA position at the Ministry of Education, GoSS level. The question of whether a STA at the national level is warranted and, indeed, whether such an advisor could be effective is discussed in the next section of this report in the context of TAP‘s facilitation of national-state coordination. Lessons Learned for Future Program Design: Management of a technical advisory program with personnel of highly variable qualifications scattered across 10 states in South Sudan is an enormous managerial challenge, and program and operations budgets for effective management should be fully analyzed in considering any future funding. Conflict-Sensitive Management The MTE requested AED TAP ―to incorporate conflict-sensitive analysis into its programming, utilizing one of the many tools available, and to report on the cumulative amount of time TA personnel are kept off or are limited on the job due to insecurity since this factor has significant implications for TAP impact‖ (Recommendation 7). Queried about this recommendation, most STAs and POEs expressed some confusion, one asking very plainly, ―what is a conflict-sensitive analysis?‖ The EOP team did find that TAP planning reflected strong awareness of event-driven risks of conflict. Careful planning was conducted in preparation for the 2011 referendum and again in anticipation of July 9, 2011 Independence Day celebrations.16 Quarterly reports indicated some reflection on the experience of the April 2010 election period, on state ministry staff attendance, and on the impact on TAP‘s work, but the primary focus has generally been on implications and risk for staff security and not on implications for capacity building efforts. No reporting appears to have been done on the cumulative loss of staff time due to conflict preparedness, but it appears to have been significant. At the time of this EOP review, all STAs and POEs were due to gather in Juba for a period of a week or more in preparation for possible Independence Day conflict and the possible need to evacuate personnel from South Sudan. 16The FY 2011 Work Plan considered four possible conflict scenarios for the 2011 referendum: (1) very minimal or no disruption; (2) some civil unrest or some sporadic outbreaks of violence; (3) substantial civil unrest and widespread outbreaks of violence; (4) extensive civil strife and a return to civil war. The plan continued, ―[a]lthough plans will be drawn for each scenario (in conjunction with other Juba INGOs), scenarios 2 and 3 should receive the most attention.‖ The comprehensive contingency plan to cover TAP within AED‘s larger security plan will be ready by October 31, 2010, and will include two sections: one on safeguarding TAP‘s technical integrity and one on ensuring its members‘ security. END-OF-PROJECT REVIEW 18 With regard to the impact of conflict on sectoral programming, it was clear that STAs and POEs in conversation with state ministry staff were regularly confronting the challenges of education in a post-conflict setting, particularly those challenges associated with educating large numbers of IDPs and returnees, responding with sensitivity to the conflict dynamics of the regions in which they worked. There does not seem to be any formal recognition of differences in conflict dynamics across states, although there is clear awareness in conversation about the difficulties of operating in more heavily conflicted border states. Lessons Learned for future Program Design: Planning for interruption to work schedules in anticipation of possible conflict is useful, but in itself does not address the consequences of time lost on capacity building outcomes. Partnership and Coordination Memorandum of Understanding with GoSS The MTE requested the AED team review and refine a draft Memorandum of Understanding (MoU) between TAP and GoSS and present it to GoSS for discussion and signature (Recommendation 1). The reasons for this recommendation were identified as follows: (1) It will help ensure that AED is not put in the difficult roles of being both a watchdog and an advocate. (2) It will help ensure that the responsibility for the success of a development assistance program ultimately rests with the receiving government and its personnel. (3) It will help eliminate any lack of clarity or confusion regarding which organization is responsible for what. (4) It will allow for a clear pathway to identifying (and rectifying) any instances where one or another of the signatory organizations does not live up to its commitment(s). (5) It makes all parties‘ responsibilities visible and clear. TAP did indeed submit a revised MoU (Annex 5) and began discussions with GoSS on the topic. The effort stalled, however, at the Ministry of Education; the chief reason provided by the ministry for the delay was that an NGO Act is working its way through the legislative process, and until such time as the legislation is fully promulgated, government is not prepared to enter into any formal agreements with an NGO such as AED. There was broad agreement among AED staff and other donor agencies that an MoU is worth pursuing, even as this was qualified with the recognition that solid personal relationships often serve to solve outstanding issues that a ―piece of paper‖ cannot. It is likely that the need for such MoUs will become even more important as the government of the Republic of South Sudan takes on greater budgetary responsibilities as an independent state. END-OF-PROJECT REVIEW 19 Lessons Learned for Future Program Design: While there is no certainty that an MoU will ensure that each party to the agreement fulfills its obligations, it will increase that possibility and remains valuable for encouraging commitment from the RSS. The MoU discussion process should begin at the soonest possible moment upon receipt of a funding award. Partnership Action Plan The MTE requested AED TAP to ―articulate its approach (including an action plan) to progressing the anticipated partnerships between State Ministries of Education and donor/development partners and between TAP and other USAID-funded activities (BRIDGE, GEE and SSIRI) and ensure that TA personnel and some personnel are familiar with and supportive of the directions proposed‖ (Recommendation 5). No action plan was developed in direct response to the recommendation, but sectoral coordination and partnership absolutely remains a central focus of TAP‘s activities and, in some respects, the area of greatest impact. (Sectoral coordination and partnership is addressed in greater depth in the next section of the report.) In short, great progress has been made with respect to relationships between TAP and SMoEs, including the facilitation of communication between SMoEs and donor partners. For various reasons, there has been less progress in coordinating across USAID-funded projects (e.g., BRIDGE, GEE, and SSIRI). In contrast, excellent cooperation with the EMIS project (formerly funded by USAID and now funded by UNICEF) and with the electronic payroll program (funded by the CBTF) has been instrumental to the success of both those projects. Lessons Learned for Future Program Design: Cooperation and coordination across USAID activities is difficult to achieve and may require greater support and encouragement from USAID program staff. Cooperation is most likely to emerge and be most effective where collaborative advantage (as in the case of EMIS) is clear. 3.2.4. Areas of Minimal Progress Gender Equity Few discussions of education in South Sudan proceed without reference to the enormous gender equity challenges. The most obvious and acute gender issues are those having to do with girls‘ school attendance: primary school attendance among girls is low to begin with and drops off precipitously in the last years of primary school (see figure below from the UNESCO 2011 report, ―Building a Better Future: Education for an Independent Sudan‖). Given this situation, it is not surprising that the percentage of female teachers is low (see figures below, also from the UNESCO 2011 report). The challenge of identifying a sufficient number of women for training is a major problem for capacity building projects in all sectors in South Sudan. END-OF-PROJECT REVIEW 20 In light of these challenges the MTE recommended that AED TAP ―articulate a gender and development support strategy in collaboration with GEE that is informed by the work being undertaken by GoSS bodies, other development partners, and projects‖ (Recommendation 9). No gender equity strategy was formally developed. However, there is profound recognition amongst TAP staff and, indeed, within the SMoEs of the need for and challenges to improving gender equity. Few solutions have been proposed beyond some informal discussion of possibilities for ―gender mainstreaming.‖ In general, staff acknowledge that limited attention has been provided to the directorates of gender at SMoEs, except in some states where the director of gender proved unusually receptive to TAP and GEE initiatives. TAP has proposed hiring a gender specialist on an STTA basis to assist with creative thinking in this area, but the terms of reference for such a specialist have not yet been finalized. END-OF-PROJECT REVIEW 21 Lesson Learned for Future Program Design: Addressing gender equity will not be resolved at TAP or other projects without creative and determined effort. An explicit gender equity strategy would inform and support ongoing efforts. Exit and Sustainability Strategy Remarkably little formal discussion of exit or sustainability criteria has occurred among TAP staff, most of whom remain highly focused on the still enormous needs for continued capacity building at SMoEs. SMoEs and GoSS staff resist discussions of project exit, often quite emphatically. Staff, when pressed, do believe it would be possible and wise to identify some clear exit objectives, usually defined around basic functions at the state ministry. In the FY 2011 work plan, the outgoing COP shared some thoughts on minimal policy requirements and capacities that must be demonstrated by SMoE and GoSS officials in order to sustain an ongoing TA capacity within the ministry. The difference between establishing basic capacity standards for an orderly exit and a broader strategy for creating a sustainable TA capacity in education in South Sudan are fundamentally different propositions. Clearly, TAP has much additional work to do to develop a coherent, actionable exit or sustainability strategy. The recommendation as phrased by the MTE was perhaps unrealistic at the midpoint of a program, as there was as yet little understanding of what was meant by a ―standard risk management matrix‖ or how one might set about developing an exit strategy that was both short-term in focus and simultaneously considered a capacity building timeframe of 10 to 20 years.17 Lesson Learned for Future Program Design: The absence of an exit or sustainability strategy creates a sense of dependence on TAP which interferes with the identification and measurement of independent capacity at SMoEs. Exit, when it inevitably happens, is likely to be perceived as abrupt and poorly executed. An exit strategy developed in consultation with SMoEs could encourage phased a hand-off of responsibilities. Developing a best￾practice exit strategy is a challenging proposition requiring considerable time and analysis that is best conducted at the onset of a program, not at midterm. Recruitment and Retention The most significant constraint to program effectiveness, including the implementation of MTE recommendations, has been challenges of recruitment and retention: turnover remains high and is likely to continue to be high. There are multiple reasons for limited progress on this issue, not the least of which is the fundamental challenge of first hiring and then retaining individuals working in very demanding jobs, in extremely difficult living circumstances at great distances from colleagues and management support. Most development projects in South Sudan suffer from high turnover, even those being implemented in the far less 17 Request AED TAP to immediately draft: (i) a sustainability and exit strategy, taking into account TAP‘s ending date of September 2011, as well as the long-term considerations associated with a realistic capacity building timeframe (conservatively 10–20 years); and (ii) a standard risk management matrix. END-OF-PROJECT REVIEW 22 challenging environment of Juba. TAP deserves special recognition for the challenges it has overcome in recruiting and retraining staff in this environment. At no point in time has TAP ever reached ideal staffing levels of one STA and one POE in each state.18 Currently, in the final months of operation, only three states enjoy the services of both an STA and a POE; two states operate with only an STA; while in five states only a POE remains to provide capacity building services. This represents a low point in staffing; three months earlier, eight states were fully staffed with both an STA and POE. USAID-imposed restrictions on new hires since March 2011 have constrained TAP. Still, as illustrated in the figure below, TAP has often experienced large gaps in staffing Senior Technical Advisors covering two or more states Senior Technical Advisor covering a single state Diaspora Technical Advisors Project Officers in Education 18 It should be noted here that TAP was not responsible for providing an STA in the states of Warrap, Lakes, or Jonglei. Those positions were to have been funded by the MDTF, administered by GoSS. The GoSS failed to fulfill its commitments in this and other MDTF-funded obligations. END-OF-PROJECT REVIEW 23 Two recommendations were made to address this challenge: (1) request AED TAP to identify and implement a strategy to support current TA personnel on the job; and (2) request AED TAP to resolve outstanding personnel well-being and contractual issues (for DTAs), including revisiting and refining position descriptions. Unfortunately, the recommendations of the MTE as they came to be implemented had some decidedly negative consequences. The MTE intended in the latter recommendation to improve the situation of DTAs. DTAs at the time were employed on a six-month contractual basis as consultants, and many had apparently expressed dissatisfaction with their employment circumstances. In recommending that the position of DTA be redefined as a full-time position (and renamed POE), the expectation was that Sudanese staff in POE positions would be better situated for advancement to a STA position and would enjoy the assumed benefits of a full-time position. However, in conjunction with the redefinition of the positions, and in response to USAID policy directives, the salaries for POEs were to be denominated in Sudanese pounds rather than US dollars, as had previously been the case for DTAs).19 This policy decision reduced overall compensation for Sudanese employees, making it particularly difficult for diaspora Sudanese to provide hard currency to families who, in most cases, were living outside Sudan. These changes were not well anticipated, fueling dissatisfaction. Additionally, incumbent DTAs interested in becoming POEs were required to apply for the new positions, and those not selected became publicly disgruntled. While such application requirements are not unusual in organizations, they are commonly (and often correctly) perceived to be a means of weeding out staff members. A number of staff members, as well as GoSS officials, interpreted AED‘s decision and actions as having a pejorative intent. The results were disruptive to the work program, with more than 50 percent of staff (including DTAs as well as others) departing in the final months of 2010. Not all staff departures can be attributed to this particular HR change, but it was certainly a contributing factor to many of them. Unfortunately, these negative perceptions persist, with deleterious consequences for interactions between POEs and STAs and between GoSS officials and AED. As many diaspora South Sudanese hired into POE positions hold dual citizenship, and most have families living outside South Sudan, the issue of payment in Sudanese pounds remains a troubling issue for new hires as well. Lessons Learned for Future Program Design: International human resources management (IHRM) in South Sudan is complicated by the fact that many, if not most, diaspora South Sudanese have families living outside the country and a large number hold dual citizenship with the US, Canada, or other countries. Many expatriate staff members are citizens of neighboring countries (e.g., Kenya, Uganda, Rwanda), and some receive full international remuneration packages while others receive specially designed third-country national pay packages. IHRM principles and practices must be reviewed and managed with utmost care both for the well￾being of staff, but also to manage the very sensitive issue of employment of South Sudanese nationals. 19 While USAID policies do allow an organization to request a waiver in order to pay national staff in USD, these requests are not always accepted and the progress is lengthy (including approval from the USAID Contracting Officer). It was not evident to the review team that AED/TAP requested this waiver from USAID. END-OF-PROJECT REVIEW 24 3.3. Sectoral Coordination and Relationships TAP advisors are embedded directly in SMoEs, placing the program in a unique and distinctive position vis-a￾vis its multiple partners (no other CB program in South Sudan enjoys a similar presence at the state ministerial level). TAP cooperates directly with a number of partners, including, of course, SMoEs. More importantly, TAP actively facilitates the relationship between multiple partners and the SMoEs, contributing directly and significantly to improved sectoral coordination. 3.3.1. Nature and quality of relationships between TAP and local partners at the state, county, and community levels The relationship between TAP and state ministry staff has been, with a few exceptions, strong and productive. TAP support is valued highly by state ministry staff, and declarations of that appreciation are sincerely voiced. Some of the appreciation rests on the still vital operational assistance TAP provides in internet-based communications, use of the TAP vehicle, and other logistical support. However, the deepest levels of trust and respect—even affection—emerge when discussing the contributions that individual STAs and POEs have made to individual capacity building. Indeed, the positive nature of those individual relationships is instrumental to TAP‘s ability to build capacity at the organizational and institutional levels. Inevitably, some relational challenges have occurred on both sides. Some officials, most often in the Directorate for Finance and Administration, have been resistant to TAP involvement, and other officials, typically older inspectors, have been unable to meet expectations for changed behavior. Some STAs and POEs have been received more warmly than others; some have been more competent than others. The political, economic, and logistical realities of South Sudan have sometimes made consistent programming difficult. Rapid advisor turnover—as well as high turnover among ministry staff—has been detrimental to sustained capacity building. For these and other reasons, enhanced capacity levels are not consistent across all states, but in general solid and respectful relationships between advisors and ministerial staff have been foundational to capacity gains. There has been limited engagement at the county level, even though job descriptions for STAs and POEs continue to identify work in the counties, and work with County Education Commissioners, as an important area of program activity. There are certainly occasions where STA and/or POEs with their state ministry counterparts have traveled to the counties to run workshops, facilitate school inspections, assist with EMIS data collection and the like. But the large distances involved, and the demands such activities impose on TAP personnel time, have made a strategic or systemic focus at the county level impractical. Indeed, there does not appear to be an explicit strategic commitment to county-based work in Phase 2 documentation. Nonetheless, almost unanimously, STAs, POEs, and some counterparts called for expanded effort at the county level, highlighting the need for further teacher quality assurance efforts and school management training—activities which occur primarily at the county level and which are not yet capably managed by county education staff. The extension of TAP management work down to the county level strikes many as a necessary and logical extension of the work being done at the national and state levels. There are strong arguments to support this position. END-OF-PROJECT REVIEW 25 The draft ―Southern Sudan Education Act of 2008‖ in Chapter 3, Section 16 (Governance and Management Responsibilities of State Ministries of Education) outlines multiple responsibilities of SMoEs that require active involvement at the county level. Action by SMoEs at the county level is potentially most important at this early stage of systems development when, for example, school inspection methods are under development, criteria for teacher performance assessment are still being established, and school inventory systems are all still being developed. There is every reason to expect that SMoE staff would benefit from (and even need) capacity building support and on-the-job-training from TAP advisors while in the field (i.e., in counties), as much as they do sitting in state ministry offices. Vitally, this additional responsibility would permit advisors to monitor the extent to which improved SMoE capacity is contributing to improved delivery of educational services in schools and communities. 3.3.2. Facilitation of SMoE and donor partner/NGO relationships TAP has played a significant role in building capacity at the state level to coordinate donor relations. The new organizational structure of the SMoEs includes a position called ―Development Partner Coordinator (DPC),‖ and the GoSS MoE maintains an office for Development Partner Coordination. STAs, POEs, and state ministry staff speak with great pride of the capacity gains achieved in this area. In all states (continuously supported with a TAP advisor) Development Partner Coordination meetings are now held on a quarterly basis and are attended by an increasing number of NGO and donor partners. Review of the minutes of these meetings suggest that the outcome of these meetings varies from pro forma information reporting by government officials and donor participants in some states to active participation in question-and-answer sessions in other states. These meetings, once closely facilitated by TAP advisors, are increasingly managed by the DPC, from the development of the agenda to recording of minutes. State ministry staff report that they are increasingly able to develop and submit funding proposals to development partners. Donor representatives often do not have the authority to make decisions locally, but SMoEs are encouraged by their increasing participation at quarterly meetings. TAP has assisted the DPC with the development of partner profiles, and these profiles are now more regularly updated and in some states are being managed in a computer database, allowing the states to more easily share this information with the MoE DP Coordination Unit. Relationships between the SMoE and donor partners have been enabled by the increasing professionalism of ministry staff. The benefits run both ways: SMoEs feel increasingly empowered to seek information about donor funding and plans in counties and states, and donor partners feel more confident in directly contacting SMoE officials to share possible funding decisions and opportunities. 3.3.3. Nature and quality of relationships between TAP and other NGOs and donor partners Cooperation between TAP and other NGOs and donor partners has been instrumental to sectoral success for a number of projects. Notable and successful cooperation has occurred in the following partnerships: UNICEF-EMIS. TAP’s ongoing support to implementation of the originally USAID-funded (now UNICEF-funded) Education Management Information System (EMIS) has been vital to the success of that project. TAP has supported the implementation of EMIS at multiple points, from facilitation of workshops on data collection to use of the data in annual work plans and budgets. TAP‘s role in EMIS is END-OF-PROJECT REVIEW 26 lauded by TAP advisors and acknowledged positively by the respective partner. While UNICEF-EMIS might have been able to implement some components of the system (i.e., data collection and data processing) without TAP support, it is highly unlikely that EMIS would have been able to stimulate and support the active use of that data in planning and budgeting, as is now becoming routine in many of the state ministries. The excitement and pride with which some directors describe their use of the data (e.g., planning for rising numbers of IDPs in certain districts and counties) holds great promise for improved planning and budgeting. The implementation of the EMIS system could well prove to be one of the lasting and important legacies of donor support in the development of South Sudan‘s education system, and cooperation between EMIS and TAP was vital to that success. This experience also provides an important lesson about the process whereby capacity is instilled and in which areas it is most likely to be sustained. Improved utilization of data required close support of TAs who were available on a regular basis to instruct and encourage. In turn, the knowledge gained in that process has a far greater chance of being sustained because of the systems nature of EMIS. UNICEF Training Funds and Education Materials Support. The partnership with UNICEF at the state level has been mutually beneficial to both parties. UNICEF is one of very few sources of regular funding for training and workshops available to SMoEs and has limited staff to facilitate the expenditure. TAP advisors have worked closely with ministry staff to assist them in preparing and submitting proposals to UNICEF for such workshop funds. UNICEF has a long history of involvement in the education sector in South Sudan, and has for some time been the primary provider of educational materials, including textbooks. As UNICEF continues to scale back its presence at the state level, TAP has supported ministry staff to step in and perform technical functions in support of the distribution of educational materials. The most recent activity of that nature has been support to SMoEs in first establishing and eventually, it is planned, maintaining inventories of educational materials and other assets. Booz & Co./CBTF Electronic Payroll System. In March 2008, the MoEST began a teacher payroll management reform program with direct support from the Ministry of Finance and Economic Planning (MoFEP), implemented by the CBTF-funded Booz & Co. project. This electronic payroll system has been widely credited with eliminating from the payroll large numbers of ―ghost teachers,‖ reducing the total number of teachers on the payroll from an estimated 35,000 to 24,000. According to the MTE, ―ministry officials speak of the payroll reform and the clarification of the nominal roll as heavily indebted to the facilitation of TAP advisors.‖ TAP staff and documentation also attest to TAP‘s active support of this important effort. TAP appeared less active in this area at the time of the EOP review, in part because the electronic payroll system is maintained by State Ministry of Finance and Economic Planning (SMoFEP) personnel seconded to SMoEs, rather than directly by SMoE staff. TAP needs to encourage SMoE staff to coordinate with the SMoFEP in order to ensure that the integrity and usefulness of the payroll system is maintained. The value of the payroll system as a database for M&E of teacher training and screening efforts is immense and not yet exploited. UNDP/SMoFEP Financial Management Systems. TAP has collaborated with UNDP in a number of states on a range of financial management topics and training efforts. TAP reported on one such collaboration in NBG, in which UNDP and the SMoFEP presented on budget ceilings and the use of SMoFEP planning and budgeting formats and tools; the report asserts that, ―as a result, during a meeting of all directors general in December 2010, the SMoE was the only ministry that presented its work plan and budget within the set ceilings and tools prescribed by the SMoFEP.‖ Much of the cooperation with UNDP and SMoFEP has been ad hoc, occurring in only some states, reflecting the relative variability of TA capacity in the financial management area. In an effort to formalize cooperation, TAP, UNDP, SMoFEP, and SMoE in CEQ entered into a formal working agreement to pilot training across a range of financial management topics, with plans if successful to extend to the training the other nine states in South Sudan. END-OF-PROJECT REVIEW 27 Cooperation with USAID-funded projects has been less successful: The MTE strongly encouraged TAP to ―progress‖ partnerships with BRIDGE-Winrock, Winrock-GEE, and SSIRI. All three projects present some opportunities for collaboration and shared learning, but these are less robust than was suggested by the MTE. While GEE and TAP share a commitment to improving gender equity in education, GEE‘s focus has been on secondary education and scholarships for teacher training rather than systemic support to the sector. SSIRI likewise engages in a very different set of activities than TAP. BRIDGE-Winrock provides the greatest opportunity for collaboration, as BRIDGE operates in three counties in NBG, three counties in Warrap, and nine counties in Unity, fielding a relatively large staff of education specialists and community development officers and presenting an opportunity for extending the reach of TAP to the county level or, at a minimum, sharing lessons learned about community-driven and county-based capacity building efforts. Collaboration between TAP and BRIDGE has been discussed among the leadership and has emerged most successfully in Warrap State, where TAP‘s STA resided on the BRIDGE compound. The collaborative advantage between TAP-EMIS and between TAP-Booz is far stronger than in the case of BRIDGE, GEE, or SSIRI. Although not a significant factor, it should also be mentioned here that a focus on attribution may impede collaboration across some USAID-funded training projects. For example, if a workshop or training is shared, only one organization under USAID rules can count the number of individuals trained. 3.3.4. Facilitation of relationships between SMoEs and GoSS MoE Facilitation of the relationship between the national MoE and SMoEs is becoming an increasingly important contribution of the TAP program. The MoE is seeking to introduce greater harmony in the decentralized education system via various national policy initiatives, while state ministries are developing the capacity to engage in the national dialogue. The need for such facilitation is likely to grow in the post-independence period as the MoE and its state-based partners come together to consult with other stakeholders in the development of the ESSP. Similarly, it can be expected that the National Education Act, which has languished in parliament since 2008, will be redrafted and submitted for legislative and parliamentary action. These interactions can be organized in three categories: Dissemination of National Policy Initiatives. TAP has played an essential role in supporting GoSS policy initiatives at the state level. The primary example is TAP‘s advocacy for, and facilitation of, the new organizational structure advanced by the MoE in late 2008. TAP facilitated the implementation of the new organizational structure at state ministries, which reduced the number of directorates from as many as 27 in one state to just five, covering five broad functional areas: planning and budgeting, finance and administration, general education, quality promotion, and gender and social change (see organogram below).20 The new organizational structure was proposed by GoSS as a matter of national policy as part of an effort to improve organizational efficiency and effectiveness, while at the same time creating structural consonance 20A sixth directorate, the directorate for alternative education systems, has recently been authorized, and addresses the needs of a wide range of learners, including adults, the disabled, children in mobile nomadic communities, etc. END-OF-PROJECT REVIEW 28 across states and with GoSS at the national level. By design, the new organizational structure sought to impose greater executive responsibility on the director general (thereby decreasing the burden placed upon the state minister of education); to provide for the efficient handling of issues by directors responsible for a functional area; and to establish clear lines and areas of authority. Additionally, and not insignificantly, the GoSS also sought, via structural elimination of positions, to reduce the number of individuals receiving salaries and allowances at the director level. Beginning in 2009, TAP has worked with ministerial staff to develop associated job descriptions. The organizational structure has been well accepted in some states, but in others TAP advisors continue to meet extreme resistance. Where this new structure is being successfully implemented, TAP will leave a structural legacy with the possibility of significant and lasting impact at the state and national level. Facilitation of State Input into National Policy Dialogues. TAP has facilitated the flow of information and influence in the other direction as well, facilitating state input into national policy dialogue. Two examples are suggestive: First, TAP has facilitated state participation in the annual Education Reconstruction and Development Forum (ERDF), and that participation has become increasingly professional. Second, TAP has begun to facilitate state input into the current discussions taking place regarding the Education Sector Strategy Plan (ESSP) in a few states (see Annex 6). END-OF-PROJECT REVIEW 29 Facilitating Direct Communication. Finally, TAP continues to perform a critical communications liaison function for the GoSS MoE and the SMoEs. Facilitation of the larger projects noted above succeeded, in part, because TAP advisors played a seemingly simple, yet essential, technical function by providing e-mail and internet access, as well as by developing and maintaining cellphone contact lists. It is to be hoped that this role will diminish as technology permits greater direct communication between units and officials at the state and national levels. 3.4. Program Achievements This review was guided by a SOW that focused the team‘s attention on the implementation of MTE recommendations and on the implications for future USAID strategy and programming. As such, the review team was not charged with conducting an evaluation that would allow a definitive assessment of program outcomes and possibilities for sustainable impact. Still, the team felt it was necessary to investigate and provide some summary statement of program achievements and discuss the likelihood that these achievements will be sustained at the SMoEs. These findings and conclusions were critical in preparing the subsequent discussion of implications for future strategy and programming. 3.4.1. Extending the gains and accomplishments identified in the MTE TAP‘s achievements are formally measured using three standard USAID indicators: (1) Number of administrators and officials trained; (2) Laws, policies, regulations, and guidelines developed or modified to improve equitable access to the quality of education services; and (3) Number of hours of on-the-job training. These standard indicators are supplemented by two additional ―customized indicators‖: (4) Institutional Development Framework Assessment scores for the MoE and SMoEs; and (5) Number of hours of on-the-job training. These data indicate that AED TAP had met or exceeded targets on all indicators for previous years and is on track to do so again in FY 2011 (Annex 7).21 TAP receives strong praise from donors for the positive and important contributions it has been making to the education sector. DFID and JDT, for example, both spoke well of the work that TAP has been doing; both noted the value of the model that involves embedding in GoSS and the SMoEs; both acknowledged the challenges that come with it; and both noted its instrumental value for other projects and other agencies. 21The EOP team reviewed the ―data quality‖ assessment provided by MSI and accepts that the three standard indicators as reported by AED‘s monitoring and evaluation unit accurately reflect achievement in those areas. MSI reviews M&E procedures to determine data validity, reliability, timelines, precision, and integrity. END-OF-PROJECT REVIEW 30 In 2009, the MTE reported that: ―TAP was assisting SMoEs to (i) increase the professionalism of SMoEs generally, (ii) take a leadership role with development partners, and (iii) increase the productivity of those partner government personnel who interact most consistently with STAs and DTAs. Benefits accruing include increased capacity in planning, understanding, and using education data, using the new payroll system, working collaboratively with colleagues and development partners, using computers and the internet, and ―self￾management.‖ The EOP team found that, in most areas, TAP has extended those gains and accomplishments. For example, where the MTE found evidence of ―improved working habits,‖ the EOP team encountered improved understanding of job responsibilities beyond simply reporting to work on time. Similarly, the MTE identified ―improved writing skills‖ as a capacity gain, while the EOP team was regaled with proud reports (by TAs, some staff, and even GoSS officials) of individuals from state ministries who are now able to prepare reports using EMIS data and to make formal presentations at national fora on education (e.g., ERDF). Where the MTE cited ―improved planning skills,‖ the EOP team found staff increasingly able, and even eager, to use EMIS data to inform those plans and budgets. In some states, annual work plans and budgets are being produced for the second and even third time. Still, as we note below, these gains are threatened by the absence of any significant budget financing. 3.4.2. Increasing possibility for sustained capacity gains The most significant achievements, in the view of the EOP review team, are those capacity gains that exhibit patterns of increasing sustainability. Most of these have already been identified in earlier sections of this report, but deserve to be highlighted again in this section: (1) TAP‘s facilitation of EMIS and the electronic payroll was instrumental to the success of those efforts. Both EMIS and the payroll, as systems, embody knowledge that is more deeply established and more easily transferred the longer and more repeatedly the system is used. (2) TAP‘s support to Development Partner Coordination at SMoEs is well-established across all states in the form of quarterly partner meetings, maintenance of partner profiles, roll-up to the GoSS Development Partner Coordination Unit, and the increasingly confident and comfortable interactions between SMoEs and donors. (3) TAP has promoted substantial communication and coordination between GoSS MoE and SMoEs, with increasing possibilities for consistent implementation of national policy across states. In directorates of planning and budgeting, where TAP has been most active, there may now be a DPC, two EMIS focal points, one or more inspectors, the director, and a deputy—all of whom have received training and coaching. Where EMIS system and payroll system are well-integrated, it seems increasingly possible that capacity will survive the departure of TAP advisors and even one or two members of the unit. Although STAs and POEs continue to identify enormous capacity building challenges, they are also eager to discuss successes, and conversation turns easily to addressing new areas of activity in teacher and school management. These conversations suggested some foundational capacity in planning and budgeting, even as they insisted there was still much work to be done. In one state, for example, the STA could state quite confidently that, were the program to end and she to leave, while some capacity would be lost, capacity across a number of areas, such as planning and budgeting, would remain. Were she to be given an extra year or two, she believed the ministry could be brought to a certain capacity level that would be sustainable. In all three END-OF-PROJECT REVIEW 31 states visited by the review team, ministry officials corroborated this sense of progress, often with a sense of accomplishment. 3.4.3. Challenges and Threats to Sustainability of Capacity Gains This report has presented a very positive picture of TAP‘s achievements. This positive portrayal is justified on the grounds that TAP is moving the capacity of SMoEs forward in important and meaningful ways. However, it is important to also acknowledge some internal weaknesses, as well as substantial challenges and threats to the sustainability of these capacity gains. First, these capacity gains are clearly not uniform across states, although it was not possible on the basis of this review to make definitive statements about differences across all states. Eventually, more even progress will be necessary to establish sectoral success. Likewise, it is important to identify areas of weaknesses, notable among them a relative lack of progress in the area of financial management. Two factors contribute to this weakness: directorates of finance have generally been more reluctant to accept support from TAP, and most TAP advisors acknowledge that financial administration is generally not an area in which they have strong skills. Improvements in financial management and related efforts to mitigate corruption will be essential to sustained delivery of educational services. Planning and budgeting will remain paper exercises if financial systems are not in place to accurately and sustainably fund planned activities. Finally, it is essential to discuss the threats and constraints to continuing accomplishment and sustained capacity. Internally, the most serious constraint derives from the extremely high turnover of TAP staff. The TAP model is deeply dependent on the competence of its TAs and the quality of relationships they develop with their counterparts at SMoEs. High turnover is disruptive to the learning process at many levels. Some of the uneven progress across states is directly attributable to this high turnover. This has been highlighted at various points in this report and a number of recommendations to address this are provided in the final section of this report. Externally, TAP faces the same broad and persistent challenges faced by all capacity building efforts in South Sudan. These include high turnover amongst ministry staff, creating a constant need to train a new set of individuals; a lack of computers, even where these are essential to the performance of the job; older staff members who block efforts at change; and a lack of transport to conduct essential tasks. Additionally, vast distances, seasonal inaccessibility, and persistent security concerns make work at the county level where schools are managed very difficult. The most serious threat, however, derives from the near absence of operational funds at the state ministry level. The revenue base for states and counties is almost entirely dependent on GoSS transfers that meet basic payroll commitments with little or no operating funds remaining to conduct the business of the ministry. Learning-by-doing—upon which capacity building at TAP is premised—is not possible if educational services are not actually being delivered. Ministry staff may be content to train with limited opportunity to exercise their new skills for a limited time, but eventually professional integrity is undermined and corruption invited if funding to support service delivery does not materialize. END-OF-PROJECT REVIEW 32 4. IMPLICATIONS FOR FUTURE STRATEGY AND PROGRAMMING 4.1. Initial Recommendations An EOP review is an excellent opportunity to take stock of lessons learned for use in future USAID portfolio and programming decisions, as well as to provide the implementing partner with recommendations should they decide to seek renewed funding or submit a future proposal. Accordingly, the SOW asked questions to elicit recommendations about how a similar future USAID investment might best be designed: 1. How should any similar future USAID investments be designed/implemented to enhance alignment and contributions of USAID assistance to achievements of key education targets in the SSDP? 2. What are the implications of the constraints and opportunities—faced in implementing the MTE—for any future programming modeled on the TAP approach? 3. Is the TAP approach replicable across other USAID-supported sectors—Health, Economic Growth, and Governance? Despite constraints, variability across states, and even missteps, there is evidence to suggest that the capacity building gains are real and deepening. TAP advisors are uniquely positioned to reinforce capacity gains in a learning-by-doing, one-on-one, coaching format not available to projects and programs operating through STTAs or Juba-based itinerant teams. One of the fundamental contributions of TAP is the role it plays in extending the gains of other capacity building programs in the states (e.g., EMIS). The presence of TAP advisors in the states increases the possibility that GoSS officials will engage actively with TAP personnel at the national level—a benefit not enjoyed by some GoSS-based national policy advisors. A more complete evaluation of the efficacy of the ―TAP model‖ would be to compare the capacity building gains of TAP with state-directed CB projects that do not embed advisors in state ministries. The question then becomes, are the capacity gains attributable to TAP presence in the ministries worth the added expense and managerial challenges of state-based STAs and POEs, which brings with it enormous challenges of recruitment and retention? Such an analysis was beyond the scope of this review, even if such comparative evidence was available. Despite the absence of such an analysis, we conclude that the TAP model is worth extending into the future and broad lessons learned perhaps even extended to other sectors. The team therefore concluded that it was in the best interest of the continuity of the capacity building effort in South Sudan to provide recommendations that would (1) serve TAP should USAID decided to extend the current program; and (2) incorporate lessons learned from the TAP experience into future design considerations for the period after the new Mission strategy begins to be implemented. Implicit in this phased approach is the recommendation that, should a no-cost extension be extended, additional funding should be invested in a future technical advisory program in education management built on lessons from TAP. Investment in a future program on the model of TAP should occur concurrently with increased support from the RSS development partners in matters of finance, civil service stabilization and the like. If, however, state ministries continue to operate without any funding to deliver services beyond NGO-funded workshops, USAID should seriously reconsider its investment in state-based capacity building. At a minimum, any future program should be subjected to a risk analysis built on different scenarios for RSS support. END-OF-PROJECT REVIEW 33 The recommendations below, unless otherwise noted, are ones in which small shifts in focus or some additional activity can begin in the transition period, while greater strategic and programmatic efforts can be considered more carefully and implemented in a future program. The recommendations are organized into three categories: Strategy and Program Management, Coordination and Relationships, and Human Resources and Organizational Structure. 4.2. STRATEGY AND PROGRAM MANAGEMENT (1) Increase Attention to Teacher Management If USAID decides to continue TAP, the program should increase its support to ―teacher management‖ activities at the state and county level (within the Schools and Materials Management component) during the transition period and assess the viability of those efforts for greater emphasis in future programming. Going forward, as USAID considers an education portfolio that almost certainly will extend funding to teacher training and development, creative consideration should be given to how a CB program with embedded TAs might best complement and extend those efforts. The viability of a capacity building project such as TAP relies on the project‘s alignment with stated strategic goals; likewise, its continuing effectiveness relies on appreciation of emerging strategic priorities. The call for increased attention to improving teacher quality was nearly universal among national and state ministerial staff as well as among AED staff. This recommendation rests also on the recognition that teacher training has stalled generally in South Sudan for a number of reasons. TAP‘s mandate is directed toward teacher and school management activities (e.g., school inspection, teacher examination, and screening) and should not be conflated with ―teacher training‖ as commonly understood. Should it become clear that some future USAID-funded program or project intends to address these same issues, collaboration with TAP should be explored with a view to how embedded TAs might best complement and extend such efforts. END-OF-PROJECT REVIEW 34 (2) Extend Activities to the County Level USAID should consider placing additional resources to extend the reach of any follow-on program to the county levels in future programming. If continued, TAP should be asked to explore options and costs during the transition period, experimenting with how such an effort might be implemented. (For example, one suggestion has been for POEs alongside an SMoE counterpart to spend one week per quarter working at county education offices). While institutional capacity building at the national and state levels are absolutely necessary for a functioning national educational system, the improved capacity needs to be able to effect change and results monitored at the county level, where schools are managed. STAs/POEs do, in fact, currently reach out to the county levels, but this has not been a major focus of attention. Given the logistical challenges of working in counties (i.e., distances, seasonal inaccessibility, transport) additional resources may indeed be necessary in a future program. Expanded monitoring of SMoE activities and evaluation of MoE policy impacts is likely to be included among future goals and objectives of ministerial capacity building efforts. Such M&E efforts cannot be effected without more regular presence of SMoE staff at the county level. (3) Increase Attention to Opportunities for Learning-by-Doing USAID should consider including funds for a micro-grant or similar program in future CB projects. This funding should in no way obviate the MoE from providing operating funds, but act as supplement until such time as improved funding streams develop. It is well-established among Organizational Development practitioners that capacity building is best achieved in government when combined with learning through real service delivery. The clearest capacity gains at SMoEs have occurred in those areas where ―learning-by-doing‖ has been possible (e.g., EMIS). The experience of other projects in South Sudan should be examined for cautionary lessons and positive guidance on how such a program might be administered. By way of example, below is a sample list of LGDF grants (administered through BRIDGE-Winrock) that were in process for the education sector in late 2010. Lower cost projects involving educational materials or supplies might be considered before involvement with school rehabilitation and reconstruction, which, aside from any cost considerations, involve complex USAID environmental compliance processes. Project Name State BRIDGE Estimated Cost Est. In-Kind Government Contribution Est. In-Kind Community Contribution Total Project Cost Classroom Expansion Liech Primary School Rubkona Unity $80,000 $17,000 $0 $97,000 Renovation of 6 Classrooms Kuajok Boy‘s Primary School Gogrial West Warrap $57,083 $0 $8,333 $65,416 END-OF-PROJECT REVIEW 35 Project Name State BRIDGE Estimated Cost Est. In-Kind Government Contribution Est. In-Kind Community Contribution Total Project Cost Mangar Primary School Roof Gogrial West Warrap $32,900 $0 $9,020 $41,920 Ameth School Improvement Gogrial West Warrap $19,660 $11,960 $0 $31,620 Renovation of 9 Classrooms Kuajok Girl‘s Primary School Warrap $110,600 $12,500 $0 $123,100 Pariang Primary School Project Gogrial West Warrap $29,881 $0 $6,540 $36,421 Maluil-tit Primary school expansion Gogrial West Warrap $29,610 $0 $6,540 $36,150 Lukluk Primary School Rehab. Project Gogrial West Warrap $57,410 $0 $12,830 $70,240 (4) Facilitate Strategic Engagement of SMoEs with the RSS MoE USAID should encourage SMoE contributions to the development of the Education Strategy for South Sudan (ESSP) with TAP facilitation, if the program is extended. Similarly, if TAP is continued, USAID should encourage advisors to consult with one another on how best to support state ministries‘ participation in the ESSP development process. USAID should also consider the implications of new strategic developments in future program design. There is every indication that great effort has been made by USAID to establish Mission goals that are in line with the strategic goals of RSS. While the emphases between USAID and the RSS goals may differ somewhat, the overlap among them is considerable and significant. Mission strategy documents have been written in collaboration with RSS, broadly speaking, and alignment with the SSDP, more specifically, in mind. The SSDP offers an ambitious and comprehensive vision of what the government wishes to achieve, and the ESSP may offer a more focused strategy on how it wishes to achieve it. Moreover, among the SMoEs, there is evidence of growing interest in the ESSP as the latter is being framed with input from the states. This input from the states should be facilitated and encouraged. END-OF-PROJECT REVIEW 36 (5) Increase Attention to Gender Equity TAP, if continued, should renew its efforts at improving gender equity. Future programming should include specific plans and devise indicators for measuring gender equity and equality gains. STAs and most ministry officials take the issue of girls‘ education quite seriously but are at some loss as to how to deal with the enormous challenge faced in South Sudan. Some within the TAP program have suggested ―mainstreaming‖ gender into activities and goals of all directorates in the ministries. This should certainly be considered as an option, but more focused capacity building in the Directorate for Gender and in the soon-to-be established Directorate for Alternative Education might also be fruitful. As noted earlier, TAP‘s FY 2011 work plan calls for developing a TOR for a STTA to consult creatively on this issue. The results of such a consultancy might also be used to inform future program design, and if TAP is unable to hire such an STTA, USAID might wish to consider hiring such a consultant for input to the broader education sector program. (6) Continue to Strengthen Monitoring and Evaluation Further steps should be taken to strengthen M&E: in particular, methods for making the IDF tool more reliable for comparative purposes should be investigated. Comparison across states and across time requires increased external validation, which might be achieved by cross-state facilitation of the instrument by STAs/POEs from other SMoEs or perhaps through the use of a single external facilitator across all states. Some variables/indicators in the IDF are more readily and reliably compared than others, and such external efforts at comparison might best focus on those indicators. Such methods may still not result in statistically valid data but are likely to improve TAP‘s ability to define exit criteria and sharpen the Results Framework so that improving capacity indeed enables improved performance. The value of the IDF tool is likely to improve with each assessment and shared discussion of results across states. For future program design, consider carefully whether to continue to refine and use the IDF tool or research other capacity measurement tools that may have been used and/or validated for M&E purposes. The authors of this report acknowledge that there are few, if any, such instruments readily applicable to TAP‘s embedded CB approach. There is, however, a growing literature on measuring capacity building that should be consulted. Any future program should establish a Program Logical Framework at the outset of the program for both M&E and program management purposes. More specifically, USAID should shift focus from reliance on standard output indicators and encourage the use of outcome indicators in future CB programs. This last recommendation was put forward by TAP management, and the EOP review team both endorses the recommendation and commends FHI for seeking to raise the required standard of evaluation. TAP has identified approximately 22 indicators that serve to measure five broad outcomes in their Results Framework. Some careful selection and refinement of these and similar indicators would be necessary in determining a final set of outcome indicators that a future implementing partner would wish to designate and customize for USAID evaluation purposes. (7) Develop an Exit Strategy If the program is continued, current TAP advisors should work with SMoE staff to identify three or four firm 1-year and 2-year ―exit targets‖ using the IDF instrument. For future programs, USAID should require the submission of a phased exit strategy along with the project proposal. The exit strategy should have a clearly defined model for sustainability (e.g., CB complete and sustainable, TA focal points at SMoE or permanent TA Service at the MoE etc.). Though progress differs depending on the state in question, there is evidence to END-OF-PROJECT REVIEW 37 suggest that officials in some states are at points where they can commit to achieving certain goals within a designated time period. Likewise, in some states, STAs agreed that arriving at certain sustainable levels of expertise may be possible within an established timeframe. Work that has been done in the last two years on the IDF may provide the beginnings of an index useful for setting mutual goals between TAP advisors and the MoE/SMoEs. 4.3. Coordination and Relationships (1) Negotiate a Memorandum of Understanding An MoU should be put forward to the RSS immediately upon award for any future CB programs. For any gains to be effective and sustained, capacity building requires both parties to be working in concert. It is urgent for the continued success of the program that the RSS MoE and TAP commit to nurturing both relationships and competence in the personnel they field at the national and state levels, as well as that each commits adequate resources. The MTE recommended that an MoU be signed to frame the relationship between the GoSS and TAP. In the absence of certain transition funding, however, it is unlikely that there is much to be gained by continued efforts to negotiate an MoU. However, in any future program an immediate effort to negotiate a MoU would be useful both to establish dialogue and to establish certain expectations of both parties. (2) Facilitate National-State Policy Dialogue and Action USAID should reconsider the possibility of establishing an STA position at the national MoE level. A full-time STA appears warranted in light of the increasing level of state-national dialogue that is already occurring and of the essential need for state participation in development of the ESSP and in a likely redrafting of the National Education Act. Such engagement between the state and the center will only become more salient as South Sudan faces the task of establishing policy for a new, independent nation. Such an STA should come with deep experience in educational policy at a national level and should be capable of providing expert advice on the alignment of RSS priorities with those of USAID. There are a number of senior technical and national policy advisors already at the MoE, and there is some suggestion that these advisors have not been very successful in gaining the ear of senior ministry officials. The EOP team believes that a STA at MoE would have a comparative advantage in influencing policy and facilitating implementation, as MoE officials are cognizant of the value of these advisors in disseminating policy at the state level. Additionally, the establishment of a close, working relationship with RSS MoE requires substantial attention that is best provided by a dedicated staff member. END-OF-PROJECT REVIEW 38 4.4. Human Resources and Organizational Structure The biggest internal constraint to the effectiveness of TAP has been the high levels of staff turnover and insufficient staffing across states. Recruitment and retention will remain ongoing challenges and must be addressed at multiple levels of human resource management currently and in future program design. These must include concerted efforts to: (1) Employ best practices and innovative approaches in recruitment and personnel management; (2) Develop an internal capacity building program to extend capacities of existing staff and future staff; and (3) Consider, and perhaps experiment with, alternative structural solutions to most effectively and efficiently deploy what are likely to remain limited human resources. (1) Employ Best Practices for Recruitment and Retention (a) TAP, if continued, should make even greater efforts to employ best recruitment and retention practices in the transition period. Any future programming should also consider the following suggestions: (i) Consult with NGO leaders in South Sudan for creative solutions to recruitment and retention challenges; (ii) Review current candidate profiles to determine whether it might be possible to relax certain criteria to widen the pool of STA candidates (in other words, assess whether a higher degree in education is actually essential or instead just desirable); (iii) Gather the collective wisdom of experienced STAs and POEs to create a profile of a successful TA; (iv) Consider greater participation of STAs on search and selection panels, especially in interviews of POE candidates with whom they might be paired. (b) TAP, if continued, should revisit its personnel and contract decisions for Sudanese STAs and POEs. Despite the best intentions of the midterm evaluation that pinpointed the need to address this issue, the changes that were implemented resulted in reduced remuneration for national staff. TAP should examine whether the current compensation package denominated in Sudanese pounds is competitive in the South Sudanese NGO market. Additionally, the current COP must address the discontent and lingering perception that AED favors expatriate staff by championing Sudanese staff for promotions, in negotiations with headquarters, and in team-building across STA and POE positions. (2) Consider Alternative Structural Solutions and Configurations TAP should consider alternative structural configurations for the efficient and effective deployment and pairing of scarce STA and POE resources. This is led by three realities: (1) the difficulty of finding sufficient number of candidates for the STA positions leading to assignment gaps; (2) the strong possibility of increasing pressure to hire only South Sudanese nationals into STA positions, limiting the pool of candidates; and (3) the call for the extension of TAP‘s reach to the county level for training in teacher and school management. END-OF-PROJECT REVIEW 39 The figure below presents some options. These are not intended as fully developed structural options, but are presented to underscore the need to creatively explore staffing challenges through a variety of means. Only some of the circles visible in the figure are discussed below: For example, Circle 4 suggests the use of both itinerant STA experts traveling to places where specific expertise meets an identified need, and regional STAs that travel but remain focused on a particular region. Circle 5 suggests a model whereby experienced STAs who have established a strong record of success in their respective states leverage their insights and experience for impact in a second state. The ―one STA/two states‖ model was originally attempted and has generally been seen as a failure in the past. However, two experienced STAs suggested that the original failure was due to the relative inexperience of STAs in any single state, hence entry into two states simultaneously proved problematic. Should strong relationships be established in one state, as is clearly the case with some STAs, it may be possible that those STAs can begin to have impact elsewhere without losing the influence in the original state. It may be an opportune time to revisit this model, but only for experienced STAs with proven skills and a strong foundation in a single state. Circle 6 explores a means to extend the reach of TAP to build teacher/school management capacity at the county level by hiring additional POEs. The current STA/POE staffing configuration is likely not sufficient for expanding TAP‘s reach into all counties in those states with a large number or counties. END-OF-PROJECT REVIEW 40 (3) Develop Formal Internal Capacity Building Program TAP, if extended, should focus on building the capacity of its current staff. Any future programming should make internal capacity building a central feature of the project, including explicit professional development planning for POEs where necessary. There is little evidence that the recruitment of competent STAs/POEs will get easier; political circumstances and financial limitations may well require hiring individuals without the full set of STA or POE competencies, necessitating substantial internal capacity building. Quarterly meetings and staff workshops have involved an explicit capacity building component, and these have already played an important role. Also valuable in this process is the creation of knowledge resources that STAs/POEs can have at their disposal. Making documents and resources available online, facilitating access to STAs/POEs with especially recognized strengths, and creating STA teams with special expertise in mentoring may all be possible considerations. Such efforts may not be enough and, depending on the gaps, professional development planning may be necessary. END-OF-PROJECT REVIEW 41 ANNEX 1: SCOPE OF WORK FOR TAP EVALUATION DRAFT Scope of Work Management Systems International (MSI) Support Project with USAID/Sudan22 Evaluation of the Technical Assistance Project (TAP) Implemented by Academy for Education Development (AED) (Estimated start date: June 2011) 1. Program Details Program Identification: Name: Technical Assistance Program (TAP) Contracting Vehicle: Leader with Associate Award (LWA), CA# 650-A-00–07–00004–00 Program Funding: Total estimated amount: $17,624,752 Program Beginning/End Dates: June 18th, 2007—September 30th, 2011 Key Agreement/Contract Modifications: mod #1, 2, 3, and 5 Implementing Partners(s): AED and IRC USAID/Sudan Technical Office: Education Office—Juba Purpose 22MSI has a 3-year contract to provide Mission-wide support to USAID/Sudan in program and project evaluation and designs, MIS management, translation services, facilities management, VIP hosting, and research. An in-country team, based in Juba, will provide these services, supplemented by short-term technical assistance. END-OF-PROJECT REVIEW 42 This Scope of Work (SOW) is designed for the purpose of conducting end-of-project review of the Southern Sudan Technical Assistance Project (TAP) implemented by the Academy for Educational Development (AED) under USAID/Sudan‘s education portfolio. The review will assess both implementation performance and impact, is intended to document lessons learned, best practices and provide recommendations to inform evidence-based future programming. The specific objectives of the TAP end-of-project review are: 1. To assess the effectiveness of the TAP approach in terms of its overall impact on institutional systems strengthening and organizational/individual Capacity development in the South Sudan context. 2. To assess TAP progress to date in meeting the deliverables of the AED/TAP Cooperative Agreement (including amendments and modifications to the original agreement. 3. Make recommendations for and identify: a) Project components that could be scaled up or phased out for the greatest impact. Replicable components that will inform further Mission investments in similar activities. b) Longer-term dynamic follow-on programs which respond to the needs to provide concretepositive outcomes to the learners in South Sudan in support of GoSS/MoE priorities and coordinate efforts with other sectors such as Health, Economic Growth, and Governance. Background The Academy for Educational Development (AED) leads the USAID-funded Southern Sudan Technical Assistance Program (TAP) with EQUIP2 partners the International Rescue Committee (IRC) and American Institutes of Research (AIR).23 The USAID/EQUIP2 Associate Award for TAP Phase I began in August 2005. TAP Phase II began in September 2007, and will continue for four years until September 2011. Between August 2005 and September 2007, TAP worked to build capacity at the Government of Southern Sudan, Ministry of Education (GoSS-MoE) in Juba. The program expanded in FY08 to nine State Ministries of Education (SMoEs) and in FY09 to all ten. TAP was designed to address policy and planning needs of the nascent government of South Sudan Ministry of Education (MoE) through institutional systems strengthening and individual capacity development as well as development of the capacity of the State Ministries of Education to plan and implement programs and policies of government at various levels. This is in line with the USAID/Sudan strategy of 2006/2008 as well as the current Transition Strategy for South Sudan. These activities consolidate, build upon, and expand accomplishments under previous USAID education programs. Specific program objectives are strengthening capacities at both GoSS and state levels to develop the policies, plans, programs and implementation capabilities that increase equitable access to and the quality of education in Southern Sudan. 23 AIR, which had been scheduled for a limited amount of short-term technical assistance in TAP, withdrew from participation in the program during discussions preceding the cooperative agreement modification. END-OF-PROJECT REVIEW 43 TAP is a labor intensive effort comprising primarily of short- and long-term technical assistance. In FY09 the State Advisors Program expanded so that one Senior Technical Advisor (STA) and one ―Diaspora Technical Assistant‖ (STA) now known as Project Officer for Education were placed in each state, in addition to the MoE Senior Advisor who sits in Juba. Short-term specialized support is also being provided on the basis of the needs identified with MoE. TAP‘s capacity building model incorporates learning-by-doing, on-the-job training, and mentoring by qualified professionals who are embedded in the ministries. TAP does not have budget for a comprehensive set of formal training activities (i.e. workshops, materials development, or study￾tours), therefore advisors work alongside ministry officials to leverage other donor and NGO funds for implementing activities. Having the advisors and assistants fully integrated into ministry structures working side by side with education officials on a daily basis allows maximum interface and knowledge transfer in an actual working environment which ultimately leads to behavior change such as improved work ethics. The TAP designed an implementation framework that guides all activities and provides the structure for demonstrating progress and measuring results. This framework ensures a consistent approach and a set of core activities across all states while at the same time allowing for responsiveness to the priorities and unique needs of individual states. TAP comprises of two closely related program areas: 1) Education Management and 2) Program Implementation. Program Activities and Beneficiaries A. Education Management activities support: (1) systematic annual planning and budgeting; (2) effective financial management; (3) a transparent and payroll system; (4) thorough documentation and record keeping; (5) rationalized approaches to task management; (6) appropriate organizational structures; (7) effective staffing and institutionalized performance evaluation; (8) a computer literate staff; (9) coordination of development partners; (10) gender equity across the education sector; and (11) enactment of the Education Act. B. Program Implementation activities are designed to build capacity to manage education projects effectively. In 2007—2008, efforts focused on launching teacher training and alternative education programs supported by the Multi-Donor Trust Fund (MDTF). These projects stalled, however, so were dropped by TAP at the end of 2008 in favor of a variety of other program areas, including: (1) state-supported fast track teacher training initiatives; (2) building community support for education; (3) enhancing school supervision; (4) improving textbook distribution; and (5) improving examination protocols. TAP‘s direct beneficiaries are MoE and SMoE education officials and systems, indirect beneficiaries include all education staff and, ultimately, students in both formal and non-formal education programs. In the 2009, a midterm evaluation was conducted (to be attached) which came up with a number of findings and recommendations that informed the implementation strategies for the remainder of the Life of the TAP project. Some of these findings and recommendations will form part of the planned end-of-project review to ascertain exactly to what extent the project performance has been improved because of the midterm evaluation findings and recommendations. END-OF-PROJECT REVIEW 44 The End-of Project Review Main Tasks Under the guidance of MSI, the Consultants are to: 1. Prior to arrival in South Sudan: a. Review major documents including the program description of the Cooperative Agreement (CA), quarterly reports and other relevant program implementation reports (to be provided by USAID Education Team); i) Quarterly reports ii) Annual reports iii) Cooperative Agreement and amendments iv) Midterm evaluation v) Annual work plans vi) PMP documents vii) USAID strategy(2006/8, Transitional strategy 2011/13) and MoE Interim Strategy, Education Status Report b. Review the report of the midterm evaluation of the program conducted in 2009 and identify major findings and recommendations for the current evaluation; c. Prepare draft methodology (Answering Evaluation Questions and data collection tools) and an noted report outline 2. On arrival, finalize methodology and evaluation work plan (in collaboration with additional team members) 3. Based on interviews with TAP staff, GoSS officials at all levels and field visits to the beneficiaries, conduct an assessment addressing the questions listed in the following section. Issues and Questions to Be Addressed The key questions (with illustrative examples) and issues to be addressed are: I. Strategy questions Question A: How does the design and objectives of the project align with the Government of South Sudan, Ministry of Education Interim strategy? Question B: What is the nature and quality of the relationships between TAP and its local partners, communities, other USAID/USG cooperating agencies, other NGOs and donor partners? Question C: Are the technical areas of focus and current approaches appropriate for USAID/Sudan‘s continued investment? How should any similar future USAID investments be implemented and/or refocused? II. Program-related questions Question D: Assess the overall impact of the TAP Program to date:  Assess program performance and progress towards achieving program results in all the key program areas as measured against targets established in the Cooperative Agreement, annual implementation plans and the Performance Management Plans. Are the results commensurate to the USAID investment in the program? END-OF-PROJECT REVIEW 45  What impact has the program had in developing human and institutional capacity in the MoE/SMoE?  To what extent is the program having an impact on improved educational service delivery in South Sudan through an improved GoSS capacity? Question E: Lessons Learned: What are the key lessons and best practices from the program that should inform designs of future education programs within the South Sudan context. III. Program Management Questions: Question F: Assess the quality and performance of AED in managing the implementation of the program. IV. Cross cutting issues: Question G: Gender being the visible development challenge in most post-war environments, what strategies has the program adopted in order to bridge the gender gap in the education sector in South Sudan? Question H: What is the project exit/sustainability strategy?  Can the project be sustained beyond the end of USAID current funding? If yes what is the strategy that will make it happen and if no, what are the impediments and how could these impediments be addressed Deliverables 1. Evaluation Methodology (Answering Evaluation Questions) Annotated Outline 2. Work plan for the evaluation 3. PowerPoint presentation on preliminary findings to USAID/AED/MOE officials to solicit feedback and comments 4. Draft Report: Prior to the departure of evaluators a draft report on the findings, conclusions and recommendations of the Evaluation Team will be presented to the MSI which will in turn share with USAID/Sudan Education Program/or Education Team Leader. The document, in MS Word, will not exceed 40 pages (excluding fact sheets, data charts and graphs). The report should include:  Executive Summary  Introduction  Background  Methodology  Findings  Conclusions END-OF-PROJECT REVIEW 46  Recommendations  Lessons Learned  Annexes: Vital source documents consulted and any other relevant materials that cannot be part of the body of the report. 5. Final Report: Following the USAID Mission‘s expedited internal review (within 10 working days) and comments on the draft report, the Consultants will incorporate the feedback and comments in a final report. The final report will be submitted to MSI 10 work days after the Consultants‘ receipt of USAID‘s final written comments on the draft report. The Mission will receive electronic copies of the final report after formatting; cover design work, etc. by MSI. This final report will be reviewed and approved by USAID to be uploaded to the DEC. Illustrative Itinerary Projected Level of Effort (LOE) and Timeline Tasks (Both External Evaluators, unless otherwise noted) Work Days (6-day weeks in Sudan; 5 in USA) Timeline for Completion Initial Preparation Review advance background documents and SUPPORT Project‘s Evaluation and Special Study Guide, make travel preparations, and travel days to Juba. 5 June 2011 In-Country Evaluation Initial briefings, meetings, field visits, draft report preparation and debriefings. 18 June 2011 Return travel 2 June 2011 Final Report Preparation in U.S. Incorporate collective Sudan feedback, complete final report, and submit to USAID Education Team Leader. 1 each and 4 for Review Team Leader July 2011 Total for each Evaluation Team member 26 Total for Evaluation Team Leader (3 additional days) 29 Team Composition, Activities and Timing The Evaluation Team END-OF-PROJECT REVIEW 47 Team Composition USAID/Sudan is conducting the End-of-Project Review of TAP in a collaborative manner to maximize USAID, GoSS and AED learning opportunities. Accordingly, the team will be comprised as follows:  Two External Evaluators (skill sets detailed below), provided by MSI  One representative of USAID  One representative of GoSS/MoE  One representative of AED Additional inputs may come from other staff from these agencies, as needed, and as coordinated by the respective team member. Team Member Roles and Responsibilities USAID, Goss/MoE, and AED team members will provide historical, contextual and programmatic background information that will inform the assessment. They will be expected to participate in the team planning meeting (TPM), field visits, interviews, brainstorming on Findings, Conclusions, and Recommendations, and in the frequent reflections on evaluation learning, often occurring after a long day of interviews and traveling. These individuals participate as representatives of their respective organizations and are expected to share their learning with their home organizations so that all three key organizations are kept abreast of progress. It may well happen that the External Evaluators will ask USAID, MoE, or AED representatives to be excluded from certain portions of interviews in order to ensure candid responses. During the initial meetings in Juba, the Evaluation Team Lead will present in writing and orally the Team‘s proposed work plan for the entire period of the Team‘s presence in Sudan as well as thereafter with respect to the submission of the final report, i.e., no later than ten work days after receipt of the Mission‘s final comments on the draft report. The work plan also will include a schedule for periodic USAID meetings/progress reports and possible submissions of specific work products, as determined by the two parties. The External Evaluators will take the lead in conducting the evaluation, leading interviews, framing the analysis, facilitating group discussion and consensus building, preparing for the debriefing, and drafting the evaluation report. One of the External Evaluators will serve as the overall Evaluation Team Leader. The Evaluation Team Leader will take full responsibility for managing the team, organizing its work, and ensuring quality control and delivery of a final report acceptable to USAID. Between the two External Evaluators, the following capacities must be brought to the team: 1. Strong skills in assessment and analysis of USAID capacity building and institutional development projects in the education sector; 2. Extensive experience working in Africa and/or similar post-conflict environments; 3. Expertise building effective systems for decentralized provision of education services in a similar— and preferably African—context; 4. Experience leading participatory evaluations, or at least evaluations where evaluation teams include critical stakeholders as active participants, or significant experience as a facilitator; and END-OF-PROJECT REVIEW 48 5. Experience arranging and facilitating meetings, setting up travel schedules for field visits, reporting on meeting outcomes, and generally managing the logistics of the review (although significant logistical assistance will be provided by the SUPPORT team in Juba). While subject to change with the acceptance by both parties, it is envisioned that the two Team members will be in South Sudan the entire duration of the evaluation‘s in-country component, i.e., three weeks (six-day work weeks are authorized). Besides travel days, an additional three days are provided for each Team member for purposes of the Team‘s initial out-of-country preparations and an additional one day for any necessary post-Sudan work related to the completion of the final report. The Team Leader, however, will be provided a total of three additional days out-of-country to ensure the completion and transmission of the final report as well as the closure of any outstanding matters. (See above graphic presentation.) Activities and Timing The USAID/Sudan Mission requests that the entire Team arrive in Juba at the same time (i.e., fragmentation of Team member‘s arrivals is unacceptable) for the initial briefings and discussions with USAID‘s Education Office(Education Team Leader , and other Mission officers, as well as AED and GoSS representatives). Subsequently, the Team will commence its field trips and meetings. The location of those trips will be determined prior to the Team‘s arrival and will be handled by MSI, in conjunction with USAID and AED. However, the probable sitesto visit include a selection of the following: A).Central Equatoria State B). Western Bahr el Gazal State C). Upper Nile State E). Warrap State Compliance to USAID Regulations The Evaluation Team will ensure that the evaluation is fully compliant with the terms for Project Evaluations contained in the USAID Automated Directives System (ADS) Series 203 and other relevant regulatory requirements, as may be determined by USAID. Additionally, the Team will utilize MSI‘s ―SUPPORT Evaluation/Special Study Quality Management Guide.‖ The Guide will be presented to the Team members prior to their initial MSI out-of country briefing END-OF-PROJECT REVIEW 49 ANNEX 2: TAP EOP REVIEW, LIST OF INTERVIEWS INTERVIEW LISTING Date Name Gender Position/Title JUBA AND AWEIL INTERVIEWS ONE-ON-ONE INTERVIEWS 17th June Chris Fagueret M Chief of Party, AED 17th June Dr Eva Nderu M Senior Technical Advisor, TAP 17th June Betty Opia F Project Officer, TAP 17th June Shadrack Chol M Director General for Planning, GoSS MoE 17th June Esther Akumu F Director for Development Partners (MDTF), GoSS MoE 17th June Utem Watba M Gender Director, GoSS MoE 17th June Alan Paul M International Rescue Committee 20th June H.E Stephen Chol Ajoga M Minister of Educ. GoSS MoE 20th June Dr. Edreda Tuwangye AED/TAP SMOE Advisor 20th June Mr. Caeaser Atem Baijo M Director General, GoSS MoE 20th June Mr. Jamesco Deng Deng M Partner Education Co-ordinator, GoSS MoE FGD I 20th June Mr. Akok Ngor Kuany M Director for Quality Promotion and Innovation, GoSS MoE 20th June Mr. Jok Aleu Jok M Deputy Director for General Education, GoSS MoE 20th June Mr. Arkangelo Aleu Thiik M Inspector of schools/Standards, GoSS MoE 20th June Mr. Jackuar Garang M Inspector for Girls‘ Education, GoSS MoE 20th June Mr. Elias Ngong Ngong M Director for General Education 20th June Mr. David Kon Martin M In charge Nominal Roll FGD II 20th June Mr. Jamesco Deng Deng M Director Planning and Budgeting, GoSS MoE 20th June Mr. Peter Lual Aleu M Director Planning and Budgeting, GoSS MoE 20th June Mr. Mawien Gok Tuong M Director Finance and Administration, GoSS MoE 20th June Mr. Andrew Matem M Deputy Director for Finance and Administration, GoSS MoE 20th June Mr. Peter Deng Deng M Inspector for Pay roll and EMIS, GoSS MoE FGD III 20th June H.E Stephen Chol Ajoga M Minister of Education GoSS 20th June Mr. Caeaser Atem Baijo M Director General, GoSS MoE 20th June Dr. Edreda Tuwangye M AED/TAP Advisor, SOME FGD IV 23rd June Kanju Yakuma Losukwa M State Technical Advisor, Unity State END-OF-PROJECT REVIEW 50 Date Name Gender Position/Title 23rd June Charles Kanyarusoke M State Technical Advisor, Western Equatoria State 23rd June Stephen Backman M State Technical Advisor, Warrap State 23rd June John Robert Lokong M Project Officer Lakes State 23rd June Gol Ayie Jal M Project Officer, Jonglei State Date Name Gender Position/Title 23rd June Simon Deng Ayub, M Project Officer, Unity State 23rd June James Adiok Mayik, M Former Project Officer, AED, Unity State; Current Position: MSI/Technical Committee Support Advisor to USAID ONE-ON-ONE INTERVIEWS 23rd June William Osafo M Team Leader, Education Team, USAID 23rd June Pia Phillip M Education Project Manager, USAID 23rd June Victor Dut Chol M Office Manager, Directorate of Planning, GoSS MoE, Review team member 23rd June Jane Namadi F Education Specialist, USAID/MSI Peter Muyinga M Monitoring and Evaluation Advisor, TAP 27th June Remigious Otira M TAP/ POE, GoSS MoE 27th June Mr. Stephen Backman M COP—AED/TAP, GoSS-MoE WAU INTERVIEWS ONE-ON-ONE INTERVIEWS 20th June Dr.Adil Athanasius Surur M Minster of Education, Western Bahr El-Ghazal State 21st June Mr. Abdul Hakim M AED/POE, Western Bahr El-Ghazal State 21st June Mr. Gabriel Gok Mawien M Acting Director General, SMoE Western Bahr El-Ghazal State 21st June Mr. Hussein Mohamed M Deputy director for Planning/Acting Development Partners Coordinator, SMoE Western Bahr El-Ghazal State 21st June Mr. Thomas Biajo M Director for Admin & Finance, SMoE Western Bahr El￾Ghazal State FGD I 21st June Mr. John Samba Rabeh M Director for Planning, SMoE Western Bahr El-Ghazal State 21st June Mr. Elias Enriko M Senior Inspector for school inspection and standards, SMoE Western Bahr El-Ghazal State 21st June Mr. Salvatore Akot M Acting Inspector for Secondary, SMoE Western Bahr El￾Ghazal State 21st June Ms.Martha Richards M Director for Gender & Social Change, SMoE Western Bahr El-Ghazal State FGD I 21st June Mr. Mario Alex M Inspector for Basic Education, SMoE Western Bahr El￾Ghazal State 21st June Mr. Clement Mayar M Inspector for Private Schools, SMoE Western Bahr El￾Ghazal State 21st June Mr. John Peter M Deputy Director for Inspections and Standards, SMoE Western Bahr El-Ghazal State 21st June Albert Apai Dimo M Deputy Director Finance & Admin, SMoE Western Bahr El-Ghazal State END-OF-PROJECT REVIEW 51 ANNEX 3: JOB DESCRIPTIONS FOR STAS AND POES Southern Sudan Technical Assistance Progra Ministry of Education Government of Southern Sudan, Juba VACANCY ANNOUNCEMENT Position(s) Project Officer—Education Vacancy Reference: POE-02/11 Duty Location(s): Kuajok—Warrap State Service Duration: 7 Months. Report To: Senior State Education Advisor Application Deadline: February 25, 2011 Background: The Southern Sudan Technical Assistance Program (SSTAP) is funded by the United States Agency for International Development (USAID) and implemented by the Academy for Educational Development (AED). The program places senior education advisors in both government of Southern Sudan‘s Ministry of Education (GoSS-MoE) and State Ministries of Education (SMoE) throughout Southern Sudan. SSTAP works to build and strengthen institutional capacity. Scope of Position: Working under the supervision of the Senior State Education Advisor, the Project Officers will support the scope of work in AED‘s USAID-funded Southern Sudan Technical Assistance Program (SSTAP, or TAP) and will work within the program developed by the GoSS-MoE and SMoEs to strengthen management systems and assist with program implementation at the state and county levels. Each Project Officer will be assigned to a State Ministry of Education. This small technical assistance team will work in a highly integrated and collaborative way with the SMoE to build capacity. Specific Duties: The Project Officers will work with the State Ministry of Education, under the supervision of the Senior State Education Advisors, to accomplish the following:  Work with SMoE counterparts to implement financial and administrative systems, including the development of yearly plans and budgets.  Assist the Senior State Education Advisor and the M&E specialist in conducting monitoring and evaluation activities at the state level.  Assist the Senior State Education Advisor and other TAP staff in carrying out all workshops and meetings.  Train and assist SMoE counterparts in basic computer and internet use.  Assist the Senior State Education Advisor in supporting programs that affect teachers and teacher quality.  Assist the Senior State Education Advisor in supporting the effectiveness and ownership of GoSS-MoE priority programs at the SMoE level.  Assist the Senior State Education Advisor to support the growth and capacity of County Education Offices as needed.  Work to improve gender awareness among all education professionals.  Work with Senior State Education Advisors to leverage funding from donors and development partners. Qualifications and Skills  Bachelor‘s degree in a relevant area such as public administration, educational administration, or business management. Master‘s degree preferred.  Minimum of two years of direct experience in implementation of development activities.  Computer proficiency in MS Office applications, e-mail, and the internet.  Ability to operate in a cross-cultural environment requiring commitment and flexibility.  Strong communication skills in English, with a demonstrated ability to work effectively as part of a team. END-OF-PROJECT REVIEW 52  Well-organized and able to work independently with minimal supervision and competently manage a variety of activities.  Excellent writing and management training skills; the ability to write business communication and technical reports. All applications addressed to: Academy for Educational Development Office, Located at Thong piny Business Center, opposite Glory Hotel and Petro City Enterprises, airport—Ministries road-Juba. Applicants in Warrap can drop applications at the DG Office State Ministry of Education in sealed envelopes. Electronic applications can be submitted to: southsudan_TAP@live.com Only short-listed applicants will be contacted. Only applicants with Sudanese citizenship will be considered. Sudanese females are strongly encouraged to apply. END-OF-PROJECT REVIEW 53 Southern Sudan Technical Assistance Program Ministry of Education Government of Southern Sudan, Juba VACANCY ANNOUNCEMENT-INTERNAL ADVERT Position(s) Senior Technical Advisor Vacancy Reference: STA-11/10 Duty Location(s):Bentiu–Unity State Service Duration: Up to September 30, 2011 Report To: Chief of Party Application Deadline: November 15, 2010 Background: The Southern Sudan Technical Assistance Program (SSTAP) is funded by the United States Agency for International Development (USAID) and implemented by the Academy for Educational Development (AED). The program places senior education advisors in both the government of Southern Sudan Ministry of Education (GoSS￾MoE) and in the State Ministries of Education (SMoE) throughout Southern Sudan. SSTAP works to build and strengthen institutional capacity. Scope of Position: The Senior Program Managers/State Advisors, assigned to SMoEs, work to build capacity, strengthen management systems, and assist with program implementation at the state and county levels. With the leadership of the GoSS-MoE advisor, the State Advisors help to build linkages between the central and state ministries to improve the delivery of quality education services in schools throughout Southern Sudan. The State Advisors work with the SMoEs in the following areas: 1) developing yearly plans and budgets; 2) setting up or strengthening administrative and communication systems within the SMoE; 3) strengthening financial management of programs; 4) designing and implementing systems of monitoring and evaluation of programs; 5) designing and implementing teacher in-service programs within the state; 6) helping to carry out a system of teacher verification and teacher payment; 7) working to strengthen Alternative Education Systems within the state; and 8) working to strengthen the new County Education Centers. The Senior State Advisors will also play an important role in assisting SMoEs to leverage the support of local NGOs for state education programs. Specific Duties: General Capacity Building Approaches Senior State Advisors will work with the state ministries, and under the leadership of the Chief of Party, to accomplish the following: • Work together with the leadership and counterparts in the SMoE to build effective systems of planning, budgeting, implementation, and monitoring and evaluation of programs. • Coordinate the activities of the SMoE with county education departments and assist the state ministries to build capacity in planning and implementation at the county level. • Build capacity by providing direct on-the-job training in the state ministries. • Assist the SMoE in the production of yearly plans, periodic reports, and financial accountability. • Focus on programs that affect teachers and teacher quality: teacher verification, registration and payment systems, teacher hiring and assignment, teacher professional development, and the expansion of in-service programs throughout the state. • Support the effectiveness and ownership of GoSS-MoE priority programs at the state ministry level, using all of these activities as broad-spectrum capacity building programs. • Work to improve gender awareness among all education professionals and to achieve gender balance in all programs. • Work in a proactive manner to leverage additional donor and NGO funding for programs at the state level. General Management Responsibilities END-OF-PROJECT REVIEW 54 • Collaborate with project team members in the development and analysis of project implementation data for reporting purposes, and assist the project team in preparing required reports and deliverables. • Maintain project financial records at the state level and provide these to the Juba office on a monthly basis. • Contribute to AED regional safety and security measures. Communicate with AED management and incident management team (COP, DCOP, RSM) regarding any issues or changes in security situation. • Supervise the Project Officer for Education assigned to the SMoE. • Supervise the driver for the state and monitor safe driving practices. Qualifications and Skills  Masters degree in one of the following or related fields: Education required.  Experience:  Seven years of relevant experience required.  Two years of successful experience with AED in South Sudan a definite advantage. Specific Knowledge Requirements:  At least five years direct experience with basic education programs in Africa, preferably in South Sudan. Experience in a conflict or post-conflict context preferred. Skills:  The Senior State Advisors will have particular skills and experience in an area such as educational planning, finance and budgeting, teacher professional development, curriculum development, or monitoring and evaluation.  Strong interpersonal communications and team collaboration skills.  Familiarity with USAID standards and procedures. Supervisory Responsibilities: Will supervise Project Officer for Education and a Driver/Administrative Assistant Working Conditions including Travel and Overtime: Positions are located in state capitals in Southern Sudan and will require travel around Southern Sudan. All applications addressed to: Application letters (no CV is required) should be submitted to: southsudan_TAP@live.com or to Michael Duku, Program Officer for Administration at the AED Juba Office. Only applicants who fulfill the above conditions will be short-listed and interviewed. Only applicants with Sudanese citizenship will be considered. END-OF-PROJECT REVIEW 55 ANNEX 4: LIST OF DOCUMENTS CONSULTED AED. Southern Sudan Technical Assistance Program (TAP). Map of State Advisors and Diaspora Technical Advisors. 2009. AED.Southern Sudan Technical Assistance Program. USAID Sudan Technical Assistance Program Annual Performance Report 2010. 2010. AED.Southern Sudan Technical Assistance Program (TAP). USAID Southern Sudan Technical Assistance Program Quarterly Progress Report Quarter 1: 1 October, 2010—31 December, 2010. 2011. AED.Southern Sudan Technical Assistance Program (TAP). ANNUAL REPORT: October 2006 to September 2007. 2007. AED. Technical Assistance Program (TAP). USAID Sudan Technical Assistance Program Quarterly Progress and Annual Performance Report 1 January–31 March 2010. 2010. AED. AED Annual Workplan 2010. . AED/ Government of Southern Sudan (GoSS). Academy for Educational Development, Draft Technical Assistance Program (TAP) Memoranda of Understanding (MoU). 2010. Deloitte. Strengthening Institutional Structures Project FY 2010 Annual Progress Report. 2010. ERDF. Summary of Proceedings. 2010. Government of Southern Sudan (GoSS). Interim Constitution of Southern. 2005. Government of Southern Sudan (GoSS). Governement of Southern Sudan Aid Strategy Revision. 2012. Government of Southern Sudan (GoSS). Kiir Inauguration Speech. 2010. Management Systems International (MSI). Constructing an Evaluation Report. . Management Systems International (MSI). Midterm Evaluation of Southern Sudan Technical Advisors Program (SSTAP) Ministry Of Education, Science And Technology (MoEST). 2009. Sudan People‘s Liberation Movement.Secretariat of Education. The Sudan People’s Liberation Movement. The Education Secor Plan for New Sudan 2005–2010. 2004. UNESCO. Building a Better Future: Education for an Independent South Sudan. June 2011. UNESCO. Capacity development in education planning and management in fragile states. 2009. USAID. USAID Service Delivery in Fragile States. 2005. USAID. USAID Guide to Gender Analysis and Integration. 2010. USAID. USAID Fragile States Strategy. 2005. USAID.Planning and Performance Management Unit Office of the Director of U.S. Foreign Assistance. Glossary of Evaluation Terms. 2009. USAID.USAID. Southern Sudan Technical Assistance Program Program Monitoring Plan (PMP). 2010. USAID.Education Team. Education: Opportunity Through Learning. USAID Educaiton Stratgey. February 2011. 2011. USAID/ AED. Associate Cooperative Agreement No. 650-A-00–07–00004–00. 2007. USAID/ AED. Modification of Cooperative Agreement No. 650-A-00–07–00004–00. 2009. END-OF-PROJECT REVIEW 56 ANNEX 5: INSTITUTIONAL DEVELOPMENT FRAMEWORK END-OF-PROJECT REVIEW 57 END-OF-PROJECT REVIEW 58 END-OF-PROJECT REVIEW 59 END-OF-PROJECT REVIEW 60 END-OF-PROJECT REVIEW 61 ANNEX 6: DRAFT MEMORANDUM OF UNDERSTANDING DRAFT Memorandum of understanding CONTEXT The education system in South Sudan is emerging from a prolonged period of civil unrest and will require substantial assistance to develop the capacity to provide adequate education. The overall objective of AED‘s USAID-funded South Sudan Technical Assistance Program‘s (TAP) is to build the capacity of the Government of Southern Sudan Ministry of Education (GoSS-MoE) and the 10 State Ministries of Education (SMoEs) in the areas of education management and program implementation, with a particular focus on building systems that support the continuing expansion and improvement of public education. Capacity building within the TAP program is envisioned in terms of action and performance—what individual professionals and groups of professionals are able to accomplish when working within an institution. In the case of TAP, the ultimate measure of capable professionals and institutions is the ability to support expanding and improving education at the local and school level—teachers teaching and students learning. Improving institutions strive for effectiveness, efficiency, relevance, and sustainability within work cultures that expect and reward a high level of performance. The capacity building model used by TAP is primarily a learning-by-doing, on-the-job training, mentoring model. To achieve its goals, TAP places one Senior Technical Advisor (STA) and one Program Officer in Education (POE) at the GoSS-MoE and at each of the 10 SMoEs. STAs and POEs are embedded into GoSS-MoE and SMoEs, working side by side with senior management. They strive to build professional and institutional capacity by working with ministry officers as they carry out their daily activities. While AED is TAP‘s prime implementer, it operates in direct collaboration with the GoSS-MoE and SMoEs. To that end, MoE directly funds (through MDTF) three STA positions, one each in the states of Eastern Equatoria, Lakes and Jonglei. Except these three, all other STAs and POEs in all states are funded by TAP. VISION MoE and AED/TAP recognize and acknowledge that capacity building works best when all those who are directly involved in program activities work in close cooperation and mutual support; TAP is an objective embodiment of that view. MISSION Through this MoU, both parties agreed to join hands and utilize their respective professional expertise in support of achieving EFA goals throughout Southern Sudan. PARTIES 1. The Ministry of Education, Government of Southern Sudan, as represented by HE Dr. Michael Willy Hussein, Minister of Education, Government of Southern Sudan, acting through Undersecretary, Mr. ___________________ 2. The Academy for Educational Development (AED), a USAID-funded international non￾governmental organization based in Washington, DC (USA), as represented in South Sudan by Dr. Brenda ARRINGTON, acting through the TAP Chief of Party, Dr. Chris FAGUERET. END-OF-PROJECT REVIEW 62 GOALS The goals of this MoU are to: 1. Ensure that the responsibility for the success of a technical assistance program ultimately rests with the receiving government and its personnel. 2. Clarify roles and responsibilities of the two parties 3. Allow for a clear pathway to identifying (and rectifying) any instances where one or another of the signatory organizations does not keep up its commitment(s). SCOPE This MoU does not involve—implicitly or otherwise—any legal or financial obligation on either of the parties. Rather, it is intended to govern the manner in which parties freely agree to manage their partnership in order to best achieve common goals. Parties to this MoU have agreed to their respective responsibilities and obligations in order to ensure that TAP capacity building activities are successful in meeting the beneficiaries‘ priorities throughout the course of the project. IMPLEMENTATION In order to implement TAP, AED has appointed a COP, a DCOP, and a team of national and international specialists. AED may also access a pool of international, regional, and national experts in direct support of specific, highly specialized capacity building activities as may be required from time to time. The COP is the AED/TAP representative and is responsible for the management and oversight of AED staff and consultants. Both parties agree that such protocol has been entered into freely and that, from time to time, revisions may become necessary by one or both parties. Such revisions would then be consigned in an appendix to the present document under both parties‘ signatures. RESPONSIBILITIES OF the GoSS-MoE  The MoE Undersecretary or his/her designate will have the primary responsibility for overseeing the implementation of the Technical Assistance Program (TAP).The Director General is the day￾to-day contact person for TAP.  The MoE Undersecretary will ensure that SMoEs are honoring their commitments under this MoU and, in the event that the state is not honoring its commitments, MoE will confer with TAP COP regarding next steps.  The MoE Undersecretary will identify and act on any element that put the Technical Assistance Program at risk, including any issues associated with the MoE itself.  The MoE Undersecretary will move toward incorporating over time the cost of TAP personnel into its own budget. RESPONSIBILITIES OF GoSS-SMoE  The SMoE Director General or his/her designate will be the contact person for the TAP Project. END-OF-PROJECT REVIEW 63  The SMoE Director General will take the lead in supervising, reporting on inputs, and judging the quality of the work of Senior Technical Advisors and Program Officers in Education.  The SMoE Minister and Director General will ensure that SMoE personnel participate consistently and regularly in learning opportunities being provided through the Technical Assistance Program.  The SMoE Minister and Director General will identify and attend to any elements that may put the Technical Assistance Program at risk, including any issues associated with the SMoE, the MoE, or the broader enabling environment.  The SMoE Minister and Director General will move toward incorporating over time the cost of the TA personnel into its own budget.  The SMoE Minister and Director General will put in place and implement a succession-planning strategy so that opportunities for the future are made available to women. RESPONSIBILITIES OF AED (On behalf of USAID)  Provide technical assistance to the GoSS-MoE and SMoEs through STAs and POEs.  Respond to any issues raised regarding the implementation of the Technical Assistance Program and resolve these with the appropriate parties.  Support the MoE‘s role in its essential activities as a central governmental body.  Endeavor to deploy full staffing of specialist personnel (STA and POE) in all 10 SMoEs, as well as at the GoSS-MoE.  Collaborate as much as possible with other donors in order to avoid duplication and/or redundancy of effort.  Liaise between GoSS-MoE and USAID. SHARED RESPONSIBILITIES Through joint ratification of the present MoU, AED/TAP and the MoE agree to engage in a professional partnership based on mutual respect and with a view to enhancing capacity building initiatives mutually agreed upon and concurred by USAID as part of TAP annual work plan. It is essential that such partnership derives visible, objective, sustainable gains for public education in Southern Sudan. In order to better achieve this aim, and to ensure a fully synergetic approach to capacity building, both parties further agree to:  Share office space in state ministry locations;  Communicate openly on all key issues of interest to both parties by means of regular joint meetings—or any other form of communication deemed appropriate by both parties;  Share all statistical data relevant to areas of TAP intervention;  Engage in joint training activities wherever possible and/or feasible; and  Provide for quarterly reporting of all TAP activities. END-OF-PROJECT REVIEW 64 PARTNERSHIP MANAGEMENT AND FOLLOW-UP The partnership that is being created through this MoU will be managed and executed within the framework of regular meetings and communications between parties. Its Steering Committee will combine representatives from MoE, AED, and USAID as soon as the present MoU has been finalized and signed. Steering Committee decisions (and their publication) shall require full agreement of both parties. Furthermore, the TAP Steering Committee will be responsible for:  Ensuring TAP interventions are duly validated;  Determining how to best achieve project goals;  Examining conditions of implementation of the present MoU and ensure that follow-up actions occur whenever necessary;  Suggesting possible amendments to the present MoU that might make it even more operational; and  Drafting a final report outlining partnership successes and suggesting possible new orientations for future similar agreements between the MoE and other partners. COMMUNICATION Both parties will communicate the content of this agreement to their respective subsidiaries. VALIDITY The present MoU takes effect immediately after signing and expires on September 30, 2011. Any and all technical reports pertaining to the present MoU will be forwarded to both MoE and AED/TAP on or before 30 days after their validation. CONTACTS All formal communications occurring under the present MoU will be in writing and will be sent to the following addresses: For MoE: For AED/TAP: Mr. xxx Dr. Chris Fagueret Undersecretary Chief of Party Ministry of Education TAP—Thongpiny Business Center Juba SS Juba SS COMMITMENT This____th day of ____________________ 2010, Juba Southern Sudan END-OF-PROJECT REVIEW 65 ANNEX 7: TAP INDICATORS