Final Evaluation: Building Recovery and Reform through Democratic Governance (BRDG) - Liberia USAID/DCHA/OTI (2006-2007) SOCIAL IMPACT Submitted by: Martina Nicolls, PhD Susan Kupperstein SOCIAL IMPACT, INC. HDA-I-00-03-00124-00 Task Order 30 24 December 2007 ii CONTENTS ACKNOWLEDGMENTS ................................................................................................... iii MAP OF LIBERIA...............................................................................................................vi EXECUTIVE SUMMARY .................................................................................................vii A. PROGRAM BACKGROUND AND DESIGN .............................................................1 B. EVALUATION APPROACH AND METHODOLOGY .............................................3 B.1 Purpose of Evaluation....................................................................................3 B.2 Scope of Work ...............................................................................................3 B.3 Approach and Methodology ..........................................................................3 C. PROGRAM STRATEGY AND IMPLEMENTATION FINDINGS............................4 C.1 Program Strategy ...........................................................................................4 C.2 Short Term Technical Assistance (STTA).....................................................8 C.3 Programmatic Focus ....................................................................................11 C.3.1 Capacity Building and Public Sector Reform..........................................11 C.3.2 Liberian Identity.......................................................................................14 C.3.3 Anti-corruption and Transparency...........................................................14 C.3.4 Media – the Star Radio Network .............................................................15 C.4 Perception of STTA and Grant Clients........................................................16 C.5 STTA and Grant Synergism.........................................................................19 C.6 Follow-on Activities and Projects................................................................24 C.7 Monitoring Implementation and Evaluating Impact....................................24 D. MANAGEMENT AND OPERATIONS FINDINGS .................................................27 D.1 Management Issues......................................................................................27 D.2 OTI, USAID Mission and US Embassy Views ...........................................29 D.3 Lessons Learned...........................................................................................29 E. LESSONS LEARNED.................................................................................................30 F. RECOMMENDATIONS.............................................................................................32 ANNEX 1: ANSWERS - OTI’S MISSION STATEMENT & BRDG PROGRAM...........34 ANNEX 2: SCOPE OF WORK...........................................................................................38 ANNEX 3: EVALUATION SCHEDULE...........................................................................43 ANNEX 4: PEOPLE INTERVIEWED ...............................................................................44 ANNEX 5: ASSESSMENT OF BRDG-LIBERIA’S IMPLEMENTATION & IMPACT.46 ANNEX 6: STTA/GRANT CLIENTS & BENEFICIARIES SURVEY RESULTS ..........50 ANNEX 7: STTA/GRANT CLIENTS & BENEFICIARIES SURVEY ............................54 ANNEX 8: GOVERNANCE PROGRAM IMPLEMENTERS SURVEY .........................56 Photo on Cover Page: Macami Choir (DAIM042) iii ACKNOWLEDGMENTS The evaluation team sincerely thanks OTI Liberia and OTI Washington for their support with the proposed itinerary, an interview list, and the field visit. In particular, the early access to the grants activity database and program documentation enabled the evaluation team to undertake extensive preparation before commencing stakeholder interviews in Washington DC, and the field review. Due to this preparation, the evaluation team was able to devise an in-country questionnaire that supplemented the field review. The evaluation team would like to thank past and present staff of DAI who readily and openly discussed their implementation activities, giving freely of their time. The evaluation team especially appreciated the assistance of Barb Lauer, Chief of Party. The evaluators warmly thank BRDG-Liberia’s grant and short-term technical assistance beneficiaries, such as local implementing partners and ministerial staff, for their cooperation and time. Special mention is awarded to the Liberian-based organization, Alexa Inc. staff, for their exceptional logistical support and companionship. In particular, the evaluation team thanks Executive Director Axel Addy, Acquyea Addy, Vaanii Kaizolu, and driver Mr. Momo. Their untiring, persistent and creative efforts in disseminating and collecting survey questionnaires were highly commendable and greatly appreciated. Kate Buban, Program Manager within Social Impact, Inc. receives a special mention for her guidance and assistance throughout the mission. iv ACRONYMS ACPA Accra Comprehensive Peace Agreement AITB Agriculture & Industrial Training Board BOB Bureau of the Budget CCN Cooperating Country National CISCAB Liberia Civil Service Capacity Building Project (DFID) COP Chief of Party CPFF Cost Plus Fixed Fee (contract) CSA Civil Service Agency DFID Department for International Development DCHA Democracy, Conflict, and Humanitarian Assistance EC European Commission ECOWAS Economic Community of West African States EGSC Economic Governance Steering Committee EITI PWYP Coalition Extractive Industries Transparency Initiative Publish What You Pay Coalition FAO Food & Agriculture Organization (of the United Nations) FDA Forestry Development Authority FLY Federation of Liberian Youth GEMAP Governance & Economic Management Assistance Program GOL Government of Liberia HOF House of Freedom I&I Implementation and Impact IPRS Interim Poverty Reduction Strategy IQC Indefinite Quantity Contract IRFC Internationally Recruited Financial Controller IRI International Republican Institute KPCS Kimberley Process Certification Scheme LIPA Liberian Institute of Public Administration LISGIS Liberian Institute of Statistics & Geo-Information Services M&E Monitoring and Evaluation MDG Millennium Development Goals MLME Ministry of Land, Mines and Energy MOF Ministry of Finance MOU Memorandum of Understanding MPEA Ministry of Planning and Economic Affairs MRU Mano River Union MYS Ministry of Youth and Sports NDI National Democratic Institute NTGL National Transitional Government of Liberia OAA Office of Acquisitions and Assistance OTI Office of Transition Initiatives PDO Program Development Officer PIP Performance Improvement Plan RFTF Results Focused Transition Framework v SES Senior Executive Service SOW Scope of Work STTA Short-term Technical Assistance SWIFT Support Which Implements Fast Transitions T&M Time and Materials (contract) TCN Third Country National UN United Nations UNDP United Nations Development Program US United States USAID United States Agency for International Development USG United States Government WB World Bank YMCA Young Men’s Christian Association vi MAP OF LIBERIA vii EXECUTIVE SUMMARY The United States Agency for International Development’s Office of Transition Initiatives (OTI) implements cutting-edge, short-term, responsive and innovative programming within a country’s transitional government. The average duration that OTI is present in countries is approximately three years, due to its focus on quick outcomes to stabilize a country emerging from conflict and to evolve the programming to change with the country’s political and security situation. The aim is to respond to immediate needs and create interventions that provide a momentum for positive change that would open the way for eventual future development. USAID/DCHA1 /OTI entered Liberia in February 2004 during the formation of the National Transitional Government of Liberia (NTGL) and commenced its programming with the Liberian Transition Initiative (LTI). Liberia held its first democratic elections in October 2005 and inaugurated President Ellen Johnson Sirleaf in January 2006. OTI ended the two-and-a-half-year LTI program in August 2006. The country’s historic move to a new government was the basis for OTI to consider whether to remain in Liberia or to close operations and to possibly handover any activities to USAID and other U.S. government agencies. In January 2006, OTI undertook four months of internal and informal political analysis before deciding to continue its assistance in Liberia for an additional year and to launch the Building Recovery and Reform through Democratic Governance (BRDG) initiative in September 2006. Initially a $5.9m program, the budget for BRDG-Liberia was increased to $6.4m and the duration of the program was extended from 12 to 15 months,2 ending in December 2007. It was precisely the right ‘window of opportunity’ for OTI to continue in Liberia and to focus on assisting Johnson Sirleaf’s nascent government. Although Liberia had moved beyond the transitional government at the commencement of the BRDG program, the political situation necessitated continued assistance due to the acute deterioration of government systems and the steady decline of human resource capacity caused by widespread and intermittent civil war that swept the country for over 14 years. To provide responsive, quick-impact interventions, OTI awards small grants to local partners, through an Implementing Partner.3 OTI designed the innovative BRDG-Liberia program4 to provide short-term, immediate support to the Government of Liberia. It was experimental and innovative for OTI in two ways: 1. The use of short-term technical assistance (STTA) to complement small grants; and 2. The use of at Time and Materials (T&M) contracting mechanism instead of their more customary Cost Plus Fixed Fee (CPFF) contract. 1 DCHA = Bureau of Democracy, Conflict, and Humanitarian Assistance. 2 The decision to extend the program occurred in June 2007. 3 The Implementing Agency for BRDG-Liberia was DAI. 4 BRDG is a USAID indefinite quantity contract (IQC), however this report refers to BRDG-Liberia as a program. viii OTI implemented STTA and grant interventions in a parallel and coordinated manner. The short￾term technical assistance (STTA) strategy provided a mechanism to strengthen the likelihood of achieving the government’s general country objectives of institutional transparency, accountability and efficiency. Short-term grants supported this approach while serving to implement innovative, pertinent and specific activities to address immediate needs. Through the Implementing Partner DAI, OTI awarded a total of $1,523,5785 to government ministries or agencies for 29 STTAs with an average of $52,537 per STTA to 26 September 2007.6 This surpassed the target of 20 STTAs. In the same period, OTI awarded a total of $1,809,128 to local partners for 60 grants, with an average of $30,152 per grant.7 STTAs were highly regarded by ministerial stakeholders8 due to OTI’s support at a critical time in the development of the country’s institutional reforms. Nevertheless, STTAs were not equally successful in their application. Some institutions supported by the program remain inherently weak as the likely gains from BRDG-Liberia interventions were thwarted by the politics of patronage that continues to characterize most institutional practice in Liberia. A major cultural shift is needed to address this problem. The program also fell short of adequately monitoring the capacity of targeted institutions to not only receive support but also their ability to capitalize upon it to implement public sector reforms and build human resource management and systems capacity. However, the impact of BRDG-Liberia needs to be seen in the context of Liberia’s emergence from one of the most protracted and violent conflicts in Africa’s recent history, which was characterized by a complete breakdown of government services. OTI implemented the program against the background of a steady brain drain of the country’s most educated civil servants, significantly reduced or non-existent technical capacities in some cases, and ruined public buildings without adequate working facilities. BRDG-Liberia clearly contributed towards a foundation for institutional change in Liberia. Its interventions can be characterized as ‘leverage for change’ – catalyzing positive change towards (1) anti-corruption, transparency and accountability of government institutions, (2) work practice changes and improved productivity within government ministries, departments or agencies, and (3) public information campaigns that generated understanding, debate, and comment from the community regarding democratic and governance issues. Factors that characterized highly effective OTI activities were as follows: • Synergistic and coordinated activities that linked grants and STTAs, leading to follow-on activities; • Public campaigns that amplified the messages of a specific grant or STTA; 5 This amount does not cover the full cost of consultants, such as airfares and other operational costs. 6 26 September 2007 was the cut-off date for accessing information from the OTI BRDG-Liberia Activity Database, as determined by the evaluation team. By the end of the program (December 2007), OTI had awarded 36 STTAs. 7 OTI did not set targets for BRDG-Liberia grants due to the addition of the experimental STTA activities. However, the number indicates a reasonable number of specifically targeted grants. 8 Beneficiaries praised BRDG-Liberia interventions during interviews and in written responses to a survey questionnaire. An analysis appears in Annex 6. ix • Interventions that led to changes within a ministry, agency or government department, such as positive work practices and updated processes, procedures, or systems; and • Activities that embedded a lasting memory or impact within the public consciousness. Five important successes of BRDG-Liberia underline its aims and objectives: 1. The Legislative Report Card (LRC); 2. The Kimberley Process Certification Scheme (KPCS) technical assistance; 3. Public information campaigns for GEMAP9 , LRC, KPCS and the birth certificate; 4. The establishment of the Star Radio network of community radio stations; and 5. Youth-related activities. Beneficiaries and the public have already observed the impact of the above activities as indicated by documentation, interviewees and responses to a survey conducted by the evaluation team.10 For example, OTI funded the Catholic Justice and Peace Commission (JPC) to produce and publish a monthly Legislative Report Card recording the legislators’ attendance and performance at sessions over a period of seven months in 2007. Some legislators called JPC admitting that the process improved and strengthened their actions in the National Legislature, pledging to work harder for the betterment of Liberia.11 OTI awarded follow-on grants to amplify the message through radio and television. OTI’s STTA and complementary grants to the Ministry of Land, Mines and Energy to improve its capacity to implement the Kimberley Process Certification Scheme and to provide legal support to miners in regional camps and communities through drama performances was an innovative way to prevent the flow of conflict diamonds.12 By supporting processes and educating miners, it contributed to the development of a system that enabled Liberia to mine and trade in diamonds legitimately. The grants for STAR Radio to develop six new affiliate stations and conduct training enabled the immediate dissemination of reliable and credible information to rural communities, increasing its audience from 100,000 to 180,000 and improving radio programming. The senior senator of Maryland County wrote in a formal communication that ‘the impact of the grant would help foster reconciliation and peace, transfer knowledge, and improve the lives and livelihood of individuals’.13 OTI’s grant to the Liberian Institute of Statistics and Geo-Information Services (LISGIS) for field communication equipment and GIS materials for mapping population census areas boosted insufficient funds from the GOL and the United Nations Population Fund (UNFPA). It was also OTI’s highest funded single grant ($205,651). This enabled LISGIS to complete the first national census since 1984. One example of a youth-related activity is the grant awarded to the Liberian Children Parliament to establish children assemblies in electoral districts within Montserrado County and to train representatives in advocacy, parliamentary processes, financial management, planning and 9 GEMAP = Governance & Economic Management Assistance Program – an anti-corruption program to secure Liberia’s revenues. 10 Refer to Annex 6. 11 DAIM037, 071 & 073 Combined Outcome Evaluation, 12 November 2007. 12 Conflict diamonds are rough diamonds used by rebel groups to finance wars against legitimate governments. 13 DAIM043 Star Radio Network Outcome Evaluation, 10 November 2007. x Day of the African Child Parade (DAIM062) Photo Courtesy of DAI reporting. This activity led to the first Youth Parliament by elected representatives in Liberia, providing a model that could be replicated in other counties. The process enabled children to discuss national issues and to participate in parliamentary activities. These activities and their outcomes may not have occurred without the funding and support from OTI. The above examples indicate a direct impact of OTI’s interventions and activities through the BRDG program, and the likely potential for lasting changes and benefits to individuals and institutions in Liberia. The evaluators undertook an appraisal of grants and STTAs with the above factors for success in mind and concluded that the following activities were OTI’s most effective interventions: TOP 5 STTA TOP 5 GRANTS 1 ‘Diamonds: Make Jobs, Not War!’ – MLME, Paul Temple (DAIM054) & 2 grants House of Freedom (DAIM069) & MLME (DAIM075) ‘Legislative Spotlight: The report Card, Is your Legislator Making the Grade?’ – JPC (DAIM037, DAIM073) & Star Radio (DAIM071) 2 ‘Across the Board: Standardizing Pay and Benefits for Civil Servants’ – CSA, John Fromm (DAIM017) & ‘Senior Executive Service Development’ – CSA, Carolyn Barnes (DAIM019) and Elenda Butler (DAIM020) ‘Employment Horizons for Mano River Union Youth’ – Ministry of Agriculture (DAIM001) 3 ‘Trans-Boundary Peace Park Consultancy’ – FDA, Jennifer Graham (DAIM072) ‘Replicating Hope: Amputee Footballers Take the Lead’ – NCDDRR (DAIM039) 4 ‘RIA Operations Manual’ – RIA, Laszlo Csiky (DAIM064), Sam Koduah (DAIM065) ‘Liberian Identity’ – YMCA (DAIM032, DAIM062 & DAIM078) 5 ‘Container Terminal Contract Bidding Process’ – Freeport of Monrovia, Audu Uba Mohammed (DAIM004) ‘GEMAP: One Year Later’ Interactive Community Theater & Community Radio – House of Freedom (DAIM030) & Star Radio (DAIM031) The above grants and STTAs are representative activities contributing to the success of the BRDG-Liberia program. Other contributing factors that led to the overall success of the program included the collaborative and participatory implementation, management and monitoring of BRDG-Liberia. OTI worked in close cooperation with its Implementing Partner to develop grants and STTAs, to monitor their progress, and to undertake continuous efforts towards improved implementation. OTI awarded a realistic number of strategically considered and collaboratively designed STTAs and small grants rather than a high number of random interventions. In addition, the newly-elected President of Liberia was pro-active to public sector reform and open to dialogue, enabling OTI greater access to the country’s central authority. Hence, OTI capitalized on the critical ‘window of opportunity’ to provide a program that responded to the country’s evolving political situation, stabilization needs and institutional reform agenda. xi Using a Time and Materials contract did not inhibit the implementation of BRDG-Liberia. However, using Cost Plus Fixed Fee (the SWIFT mechanism) is more flexible and advantageous to OTI. First, OTI is more familiar with this type of contract and second, consultants are easier to hire through a CPFF contract. Using a CPFF mechanism would provide more flexibility in selecting consultants for the STTA from all levels and countries. However, overall, the evaluation team viewed the BRDG mechanism as an effective approach to institutional support in the newly fledged government environment in Liberia. It possibly enabled OTI to provide short-term, quick impact human resource management and systems capacity enhancement to targeted institutions while simultaneously providing short-term grants to local partners. It also continued the focus of youth support from OTI’s precursor program, the Liberian Transition Initiative (LTI). Main Findings: • OTI’s decision to continue its support in Liberia was appropriate and essential given the inauguration of the new government and the President’s immediate impetus in developing a strategy for public sector reform. • The ‘blurring of the boundaries’ between BRDG and its precursor program, the Liberian Transition Initiative, by commencing a short-term technical assistance (STTA) activity at the end of LTI was a successful strategy. It provided a continuum of activities that effectively merged and evolved into the BRDG-Liberia program and led to improvements in subsequent STTAs. • OTI was strategic in its country program by consolidating its niche areas and supporting a few focused objectives that were reviewed, and modified, as Liberia’s political and security situation evolved. • BRDG-Liberia’s institutional support model (i.e. the use of STTAs) was clearly articulated and documented in the design phase, and implemented effectively, significantly contributing to the program’s success. • Instigating change within government institutions by hiring STTA consultants or change agents to provide assistance and advice was appropriate and effective for the activities implemented by OTI and well-matched with the objectives of the overall program. • OTI was receptive and responsive to the evolving political situation and to the nation’s rehabilitation and recovery strategy. • The BRDG-Liberia’s focus remained fixed on key transition points of the rehabilitation and recovery of Liberia, bringing appropriate resources to bear on these events/institutions. • The ability to phase and sequence new types of interventions depended upon the readiness and ability of institutions to implement quality interventions with responsibility and ownership. xii • BRDG-Liberia’s use of a participatory process for designing the Scope of Work, identifying consultants, monitoring the progress of STTAs and designing the program and budget for the grants was effective. • OTI’s capacity building and public sector change matched the short-term nature of the program and was appropriate and directly linked to program objectives, its goal and a window of opportunity to which OTI, through BRDG-Liberia, was responding. • Community radio networking was greatly effective and highly appropriate for OTI, enabling the public to receive relevant, ‘first-hand’ information. • Of OTI beneficiaries responding to an evaluation survey, 86% indicated that their expectations were met and that the intervention was successful. • Due to the BRDG interventions, 74% of respondents indicated that they had changed their work practices by becoming more active in meetings, realizing the need for timely record-keeping, and realizing the need for transparency and accountability. • Synergies occurred between grants and STTA activities and with other U.S. and donor programs. • While the Impact & Implementation (also known as Monitoring and Evaluation) STTA for the BRDG-Liberia program was time-consuming and expensive, the process enabled OTI to develop a monitoring and evaluation system specific to the needs of the program and could be considered a success. • Four management issues arose during the implementation of the program: (1) understanding a T&M contract versus a CPFF contract, (2) how ideas for the grants and STTAs were generated, (3) consultant recruitment, and (4) working from a shared vision; none of which adversely affected programmatic implementation. • OTI staff had complementary country knowledge and management skills with an effective division of labor. Recommendations: • If OTI supports a nascent government in another country, it should ideally implement the program over an 18-24 month timeframe, enabling time to implement and monitor short￾term technical assistance for institutional improvements. • Using a CPFF contract mechanism may be more flexible and advantageous to OTI than a T&M contract mechanism as it enables more flexible selection of consultants for the STTA from all levels and countries. xiii • OTI can replicate the two-phased institutional support model of short-term technical assistance and small grants in other countries. • OTI should replicate BRDG-Liberia’s use of participatory processes for designing a Scope of Work (SOW), identifying consultants, and monitoring the progress of STTAs as this process amplifies the capacity building impact of the activities. • If the overall program is short-term in nature, the STTA SOW is more effective when it has discrete deliverables that target key problems or issues within an institution or agency and within a realistic timeframe, rather than a ‘catch-all’ for technical assistance. • The STTA consultant should be paired with a ministry/agency counterpart for transferability of skills and as a means to continue to adopt and adapt lessons learned. • Fewer interventions with synergies and follow-on activities are preferable to a multitude of randomly scattered interventions. • Community radio networking should be considered an ideal intervention for implementing strategic communication activities and amplifying high-profile messages. • Future experiments on Implementation and Impact (Monitoring and Evaluation) should be encouraged and tailored depending on the length of the overall program. For example, the case study methodology could be replicated for ‘sustained’ institutional short-term technical assistance and partnerships over a 12 to 24 month program. • Donor coordination and cooperation could be improved by disseminating information on OTI activities as a foundation for potential handover of activities. 1 A. PROGRAM BACKGROUND AND DESIGN USAID’s Office of Transition Initiatives (OTI) entered Liberia in February 2004 after the August 2003 Accra Comprehensive Peace Agreement (ACPA) was signed, and during the October 2003 formation of the National Transitional Government of Liberia (NTGL). The widespread and intermittent 14-year civil war had severely decimated the country’s institutional human resource capacity and government systems. Therefore, NTGL’s mandate was the implementation of Liberia’s two-year (2004-2006) recovery plan, the Results Focused Transition Framework (RFTF). The ACPA outlined a distribution of power within the NTGL and associated agencies amongst former armed factions in the conflict.14 It also included commitments to democracy, good governance and sound macro-economic management. OTI commenced its support in Liberia during this period of transitional government with limited capacity and a long legacy of corruption and volatility. From February 2004 to August 2006, OTI implemented a two-and-a-half year Liberia Transition Initiative (LTI) through the SWIFT II contract mechanism. LTI’s aim was to support the NTGL towards consolidated peace after protracted civil war, predominantly through youth and education programs. Following elections in October 2005 and the inauguration of the new government in January 2006, the RFTF activities ended in March 2006. The inauguration of Ellen Johnson Sirleaf in January 2006 as President of Liberia, and the first democratically elected female President in Africa, was a major turning point in the country’s unstable political history. Immediately, the Sirleaf Administration documented a 150-day Action Plan of activities for reform to June 2006, as the first phase towards a long￾term government strategy for reconstruction and development based upon four pillars: 1. Expanding peace and security; 2. Revitalizing economic activity; 3. Rebuilding infrastructure and providing basic services, and 4. Strengthening governance and rule of law.15 At the conclusion of the 150-day Action Plan in June 2006, the GOL introduced an Interim Poverty Reduction Strategy (IPRS) from July 2006 to December 2007 with priority interventions to achieve Millennium Development Goals (MDG) aiming towards a Poverty Reduction Strategy in 2008. In the IPRS, the GOL identified four key ministries to mirror their mandate to address conflicts around land, chieftaincy, youth and inter-ethnic rivalries16: (1) Ministry of Labor, (2) Ministry of Land, Mines and Energy, (3) Ministry of Youth and Sports, and (4) Ministry of Internal Affairs. Hence, the Sirleaf Administration established clear plans for taking the country forward towards anti-corruption, good governance, and peace building. With LTI concluding in August 2006, OTI debated at length its decision to extend its presence and programming in Liberia. It commenced with an in-country rapid re-targeting exercise in January 2006,17 followed by four months of internal informal political analysis, 14 The NTGL comprised factions from the previous GOL and included two previous rebel forces, the Liberians United for Reconciliation & Democracy (LURD) and Movement for Democracy in Liberia (MODEL). Under the ACPA, key ministries and agencies were “distributed” between the various groups to share the power base. These political imperatives led to a negotiated settlement, ending the violence and regional security. 15 GOL, 150 Day Action Plan: A Working Document for a New Liberia, January 2006 (Executive Summary). 16 BRDG-Liberia Program Rationale – Update, 22 November 2006. 17 Eleanor Bedford (Senior Field Advisor), USAID/DCHA/OTI-Liberia, New Strategic Opportunities and Strategy Options Retargeting Report, February 2006. 2 including a two-year strategic framework for future program activities documented the following May.18 As a result of the strategic discussions, OTI launched the Building Recovery and Reform through Democratic Governance (BRDG) Initiative in Liberia in September 2006.19 The evaluation team considered OTI’s decision to continue its support in Liberia highly appropriate and essential, given the inauguration of the new government and the President’s immediate impetus in developing a strategy for public sector reform. Initially a $5.9m program, the budget for BRDG-Liberia was increased to $6.4m and the duration was extended from 12 to 15 months, ending in December 2007.20 The BRDG-Liberia program provided strategically targeted support for activities to assist the Government of Liberia (GOL) gain traction and consolidate legitimacy as it confronted the formidable challenges of national transformation. The three specific objectives21 of the BRDG initiative included: 1. Assist the Government of Liberia and other key actors to improve capacity in such areas as effective planning, budgeting, communication, and coordination with relevant counterparts; 2. Assist the Government of Liberia and other key actors to mount effective political responses to high-visibility issues; and 3. Support regional activities aimed at strengthening the Mano River Union. BRDG-Liberia supported these objectives through two mechanisms: (1) short-term technical assistance (STTA) to key GOL ministries and civil society organizations, and (2) small grants to ministries and local partners. The design and timing of BRDG-Liberia ensured that the support of selected key institutions coincided with the general institutional reconstruction and recovery of the country in an effort to move institutions towards functionality and good governance. The program’s rationale was based on (1) preparing institutions for the new government elected in October 2005 and inaugurated in 2006; and (2) the view that peace would not be sustainable without critical changes in the management of the public sector. The lack of democracy, good governance and financial transparency were considered among the root causes of the previous conflict. Overall, the evaluation team thought the BRDG program was an effective approach to institutional support in a nascent government environment. It enabled OTI to provide short￾term, quick-impact human resource management and systems capacity enhancement to targeted institutions while simultaneously providing small grants to local partners, and continuing the focus on youth supported during its precursor program, the Liberian Transition Initiative. This report forms the final evaluation of OTI’s BRDG-Liberia initiative, focusing on program strategy and implementation. 18 USAID/DCHA/OTI-Liberia, Liberia Transformational Universe, May 2006 (a two-year strategy). 19 Building Recovery and Reform through Democratic Governance (BRDG) is a USAID indefinite quantity contract (IQC) mechanism, however this report refers to BRDG as a program. 20 BRDG-Liberia extended from 30 August 2007 to 15 December 2007 (Modification No. 2, 25 May 2007) for 240 days for additional STTA senior level TCN labor. 21 The objectives changed throughout the program as discussed in Section C: Program Strategy and Implementation Findings. 3 B. EVALUATION APPROACH AND METHODOLOGY B.1 Purpose of Evaluation OTI-Liberia programs concluded on December 15, 2007. The purpose of the independent final evaluation is to conduct a systematic and objective assessment of the BRDG-Liberia program taking into consideration its design, management, and implementation strategies, activities, and results. The aim is to determine the program’s relevance and fulfillment of objectives, transitional effectiveness, and impact. B.2 Scope of Work The evaluation team consisted of a team leader and a mid-level evaluator, with a locally￾based Liberian supporting the team in-country with logistics and survey dissemination and collection. The team conducted the final evaluation in October and November 2007, with a week of desk review, a week of Washington DC-based interviews, and two weeks in country from 5-18 November. The evaluation culminated in a presentation to USAID/OTI on 28 November, with the draft report submitted on 3 December, and a final formal debrief on 13 December. It concluded on 24 December when the final report was submitted to OTI. Annex 3 details the team’s schedule. The evaluation report covers the BRDG-Liberia design and implementation period to the end of October 2007 and the Activity Database to 26 September 2007. Although the scope of work focused solely on BRDG-Liberia, the team also referred to the evaluation of its precursor, the LTI. An After Exit Review (AER) Contracts Analysis was conducted for OTI to analyze the efficiency and cost-effectiveness of using the BRDG Indefinite Quantity Contract mechanism22 and hence it is not covered in this final evaluation. Refer to Annex 2 for the full evaluation Scope of Work (SOW). B.3 Approach and Methodology The evaluation team assessed the design, management, strategies, performance, results, and impact of the BRDG-Liberia initiative through the following approaches: (i) Desk (Literature) Review The evaluators conducted a literature review of program documentation, made available by OTI. The documentation included program performance reviews, country strategy documents, Implementing Partner proposal and quarterly reports to OTI, relevant Foreign Assistance (F) indicators and structure for BRDG-Liberia, the task order, statements of work, success stories, case studies and other relevant materials. OTI also provided training for the evaluation team to access the BRDG-Liberia Activity Database. (ii) Interviews The evaluation team held interviews in Washington DC before deploying to Liberia. Interviews were with OTI staff involved in the design, development and implementation of BRDG-Liberia, staff from the Implementing Partner DAI, staff from the Department of State, and other USAID colleagues. Interviews were held face-to-face and by telephone, depending on the availability of the interviewee. In Liberia, interviews were held with DAI’s local partners, beneficiaries, ministerial and institutional representatives, short-term technical assistance consultants, and country staff from OTI, the USAID Mission, and the US Embassy. Overall, 48 interviews were conducted in DC and Liberia (detailed in Annex 4). 22 The AER Contracts Analysis was conducted by Robert Bonardi, Social Impact, December 2007. 4 (iii) Field Review During the two-week in-country evaluation from 5 to 18 November, the evaluators interviewed BRDG stakeholders to assess the program and to further understand the complexity of the Liberian political situation. This also included placing the evaluation of BRDG in context with the precursor LTI program. Within the Scope of Work (SOW), OTI recommended questions to address throughout the final evaluation (Annex 2) that assisted the evaluation team to hone stakeholder interviews and the field review to the specific requirements of the evaluation. (a) Evaluation Surveys The evaluation team devised two evaluation surveys that were conducted in Liberia: (1) STTA/Grant Clients and Beneficiaries (Annex 7) The locally based Liberian logistics assistant disseminated and collected the surveys, designed to capture qualitative and quantitative data to provide the evaluation team with additional information related to capacity building and program implementation. The logistics assistant disseminated 39 surveys (according to a list of beneficiaries provided by DAI) with 15 returned from a representative sample (a 39% response rate, considered to be valid for gaining insight).23 The evaluation team analyzed the survey information in the field in preparation for a debriefing to OTI in Monrovia on 16 November and in Washington on 28 November 2007. Annex 6 documents the survey results. (2) Other Governance Program Implementers in Liberia (Annex 8) The logistics assistant distributed four survey questionnaires to implementers of other USAID-funded governance programs in Liberia. One was returned (25%) with limited information as the respondent was not aware of BRDG-Liberia. However, this is important to note in terms of outreach regarding program information and its goals in the Liberian context. (b) In-country Site Visits As the BRDG-Liberia program was a national-level program, the majority of the field review was conducted in and around Monrovia, the capital of Liberia. The evaluation team traveled to Tubmanburg for a day-trip with the purpose of visiting a community radio station as part of the Star Radio network. C. PROGRAM STRATEGY AND IMPLEMENTATION FINDINGS C.1 Program Strategy Did BRDG-Liberia respond in the most strategic way to shifts in the transition process and evolving political situation in Liberia? Was OTI appropriately flexible and responsive to the ‘key windows of opportunity’ in each of its phases? ‘Strategy’ in OTI’s context applies to short-term ingenuity in addressing targets of opportunity and focusing attention on achieving medium-term strategic objectives rather than long-term strategic vision.24 OTI’s mandate is therefore to identify shifts in the transition process and to respond rapidly through subsequent and corresponding programmatic changes. 23 Refer to Annex 6. 24 Scope of Work, Annex 2. 5 In order for OTI to identify shifts, they have a system of ‘rolling’ (continuous) assessments and regular strategy meetings. OTI commenced strategizing for the BRDG-Liberia program during its decision-making process in which it decided to continue operations in country beyond the LTI. This culminated in the BRDG Task Order that outlined the program design and strategy. There was a ‘blurring of the boundaries’ between LTI and BRDG, by commencing short-term technical assistance in the Ministry of State for Presidential Affairs from September to November 2006 with the development of a Performance Improvement Plan (PIP)25 and strategic communication training so that the Sirleaf Administration could improve its public outreach. The ‘blurring’ was a successful strategy as it provided a continuum of activities that effectively merged and evolved into the BRDG-Liberia program. This also served to test the STTA provided to government institutions that led to improvements in the implementation of STTAs for the BRDG-Liberia program. Table 1 below outlines OTI’s changing objectives and their rationale over its 15 months implementation phase: Table 1: The Changing Objectives of the BRDG-Liberia Program THREE OBJECTIVES OF THE BRDG-LIBERIA INITIATIVE Task Order: 30 August 2006 Implementation: Oct 2006 12-month Stage: 26 June 2007 Assist the GOL in the near term to manage public expectations by increasing public access to information about government policies, priorities, & progress Assist the GOL & other key actors to improve capacity in such areas as effective planning, budgeting, communication, & coordination with relevant counterparts Assist the GOL & other key actors to improve capacity with an emphasis on communication systems Assist the GOL to take initial steps to translate specific objectives & policies into tangible, visible action Assist the GOL & other key actors to mount effective political responses to high-visibility issues No change to objective To provide targeted support to Liberia’s ability to contribute to & benefit from regional stabilization Support regional activities aimed at strengthening the Mano River Union No change to objective Rationale: The original 3 program objectives were amended shortly after the program began. They did not change the spirit of the program, rather broadened the strategy to allow for flexibility.26 Rationale: Other key donors were providing developmental assistance for capacity building in planning, budgeting and coordination. BRDG strategic advantage no longer present in these areas.27 Imperatives listed in the Task Order shifted soon after the program commenced to provide more flexibility. Objective 3 was narrowed from ‘sub-regional programming’ and ‘regional stabilization’ to the Mano River Union (MRU). The rationale was to focus contributions towards attainable peace in the Mano River Basin, involving collaborative links and liaison between Liberia, Sierra Leone, and the republic of Guinea. Halfway through the program, in June 2007, OTI re-focused the first objective to improve institutional capacity in planning, budgeting, communication, and coordination, to emphasize strategic communications. Thus during the BRDG-Liberia program, OTI was clearly receptive and responsive to the evolving political situation and to the nation’s recovery process. 25 OTI’s Liberian Transition Initiative funded the first STTA for this Ministry. One of the three consultants followed up in January 2007 under BRDG-Liberia (OTIM067). 26 DAI, USAID/OTI-Liberia First Quarterly Report, September to December 2006. 27 Musu Clemens, Memorandum to File, OTI/Liberia Modified Objectives, 26 June 2007. 6 The Changing Strategy of Objective 1: Assist the GOL to Improve Capacity OTI expected that Objective 1 would be the major programming focus.28 Hence, the mechanism to provide short-term technical assistance (STTA) in key ministries and institutions was central to the program from its commencement. Graph 1 below outlines the distribution of program activity, grants and STTAs, throughout the program, by objectives, showing the predominant focus on Objective 1. Graph 1: STTA and Grants Distribution by Objectives STTA and Grants Distribution by Objectives 0 2 4 6 8 10 12 14 16 Q1: Sept to Nov 2006 Q2: Dec 2006 to Feb 2007 Q3: March to May 2007 Q4: June to Aug 2007 Q5: Sept to Nov 2007 Objective 1 Objective 2 Objective 3 Source: OTI Activity Database for BRDG-Liberia to 26 September 2007 The BRDG-Liberia Program Rationale – Update, a major strategic document in November 2006, focused on the rationale and justification for programming decisions regarding OTI’s first objective. STTA consultants would work in tandem with relevant counterparts to improve capacity in such areas as effective planning, budgeting, communication and coordination within targeted ministries and agencies. It was recognized by OTI that its comparative advantage was in flexible, responsive support that could help catalyze more sustained reforms or improvements in the governance process. Thus OTI focused initially on the following ministries:29 • Sustained support for senior executive and coordinating offices such as the Ministry of State for Presidential Affairs and the Ministry of Planning and Economic Affairs. • Sustained support for line ministries such as Youth and Sports; Land, Mines and Energy; Internal Affairs; and Justice; in addition, various other ministries benefited from short-term, punctual assistance that sought to fill critical gaps and/or allowed other assistance programs to maximize their efforts. The most effective justifications in the November Program Rationale are those of the Ministry of Youth and Sports (MYS) and the Ministry of Land, Mines and Energy (MLME). The rationale for working with the MYS focused on the inter-relationships with other key institutions, such as the Agricultural and Industrial Training Board (AITB) and the Federation of Liberian Youth (FLY), to mitigate potential youth-related conflicts that continued the cross-cutting theme of youth developed during the LTI. The rationale for MLME also strives for harmonization over difficult land-related issues that could spiral into conflict if not 28 BRDG-Liberia Program Rationale – Update, 31 March 2007. 29 BRDG-Liberia Program Rationale – Update, 22 November 2006. 7 addressed. Overall, the strategy document soundly and effectively prepared the BRDG￾Liberia program to move forward with its goal and objectives. The documented rationale did not always transfer into implementation. After the program’s 22-23 March 2007 Implementation Review, Objective 1 was revisited and documented in the second Program Rationale in March 2007. This was predominantly due to expected sustained partnerships with some ministries and agencies that did not eventuate, such as the Executive Mansion and MYS. Instead, OTI noted that other ministries that initially were expected to have limited involvement gained a higher profile, such as the Ministry of Justice, MLME and the Civil Service Agency. Lessons learned over the first four months of STTA implementation included (1) the results are more effective when clients are more involved in the design of the SOW, and (2) the final out-brief should be conducted before departure. In June 2007, in their Mini Strategy Review, OTI amended Objective 1 to focus specifically on strategic communications, and documented the modification in a memorandum.30 Strategic Communication - Did OTI get the right ‘high-profile’ messages out at the right time in the right way? Prior to the change of strategy in June 2007 to Objective 1, BRDG had already awarded 20 activities with the theme of strategic communication, commencing another 13 activities in the second half of the program to 26 September 2007.31 Therefore, the theme was emphasized at the outset of the program. Rather than being a dramatic shift in strategy, it was, in effect, a de-emphasis on the areas of planning and budgeting at the government level. Strategic communication included ‘high-profile’ messages including the Kimberley Process Certification Scheme to mitigate the smuggling of diamonds, the promulgation of GEMAP32 activities, the results of the Legislative Report Card, the National Budget Process, and the Birth Certificate Process.33 Such messages, widely received by the public, raised their awareness and engaged them on critical national issues at important junctures within the public sector reform process. Hence, the information to the nation was both effective and timely, and disseminated through the widely used medium of community radio, as well as through other media such as newspapers, television, and community drama. Objective 2: Assist the GOL with Effective Political Responses to High-visibility Issues Objective 2 had peaks and troughs in its distribution throughout the program (Graph 1) with small grants focusing on the following themes: (1) public information and advocacy campaigns, (2) civil society, (3) media, (4) strategic communication, and (5) youth issues. Other themes included anti-corruption, peace building, and social or cultural activities. There are also grants pertaining to Liberia’s national identity. Generally, small grants under Objective 2 appear to be appropriate and relevant to Liberia’s political and security situation due to their focus on issues that mitigate conflict and dispute. Objective 3: Strengthening the Mano River Union The intention and rationale of Objective 3 was to support activities within the Mano River Union for regional stabilization: Liberia, Sierra Leone and Guinea. Although it was a lower priority for OTI, and programmatic activity was limited due to factors such as costs, logistics, and timing, OTI attempted to address this concern through a Country Representative visit to 30 Musu Clemens, Memorandum to File, OTI/Liberia Modified Objectives, 26 June 2007. 31 OTI BRDG-Liberia Activity Database to 26 September 2007 close-off date for the Final Evaluation. 32 The Governance & Economic Management Assistance Program is an anti-corruption program to secure Liberia’s revenues. 33 DAIM069, DAIM030, DAIM071, and DAIM088 respectively. 8 Sierra Leone for consultations with USAID-Sierra Leone and Guinea.34 However, the number of activities still did not increase due to logistical limitations and the lack of opportunities for suitable, relevant and timely interventions. Did OTI establish any lasting ties from the MRU? Despite the limited number of activities in Objective 3,35 the program contributed to the stabilization of the region. The BRDG program conducted youth-related activities critical to the region, including its first grant, a study visit to the Songhai agricultural scheme in Benin and work in the border area with Guinea. A reason for the success of activities such as ‘Employment Horizons for Mano River Youth’ (DAIM001) included the focus on youth, employment, food production and self-sufficiency that addressed socio-economic problems in the region. Another reason for its success included the establishment of links between USAID, the UNDP and FAO.36 The Songhai model for generating and sustaining growth in rural areas is one that can be replicated in similar situations as its underlying principles of tolerance and understanding can be integrated into other OTI and donor programs. The US State Department and USAID are currently in the process of rationalizing the provision of longer-term assistance. Stakeholders expect the increase in interactions and discussions at all levels (community to ministerial), especially with regard to food production, agriculture and trade within the region, with a legacy of lasting ties. Summary of Program Strategy Overall, the program objectives evolved to respond to the political situation in Liberia, as well as in response to other donor activities. It was appropriate for OTI to change objectives, demonstrating the program’s responsive and flexible design. OTI documented and rationalized extensively Objective 1 as it evolved, particularly during the first nine months. However, the selection of targeted institutions changed during implementation to reflect the evolving needs and issues within the public sector and local partners; sometimes resulting as lessons learned but mostly to the program’s benefit. C.2 Short-Term Technical Assistance (STTA) Were the STTAs effective, considering the timeframe of the BRDG? Did they lead to any positive and lasting changes within the ministry? Were recommendations implemented? Instigating change within government institutions by hiring STTA consultants or change agents to provide assistance and advice was appropriate and effective for the activities implemented by OTI and well-matched with the objectives of the overall program despite the short timeframe of BRDG. Through the Implementing Partner, DAI, OTI awarded a total of $1,523,57837 to government ministries or agencies for 29 STTAs with an average of $52,537 per STTA to 26 September 2007.38 This surpassed the target of 20 STTAs. Two initial approaches emerged for the selection of ministries that were to receive STTA: (1) a targeted number of ministries that were identified as sustained partners, and (2) ministries that required immediate support for a critical need through activities that provided a ‘spark 34 The visit occurred in late March 2007, BRDG-Liberia Program Rationale – Update, 31 March 2007. 35 OTI awarded 9 grants against Objective 3, to 26 September 2007. 36 Report of the Liberian Observer Delegation to Center Songhai, Benin, 10-18 October 2006. 37 This amount does not cover the full cost of consultants, such as airfares and other operational costs. 38 26 September 2007 was the cut-off date for accessing information from the OTI BRDG-Liberia Activity Database, as determined by the evaluation team. By the end of the program (December 2007), OTI had awarded 36 STTAs. 9 and bridge.’ If the BRDG mechanism was to be conducted over a longer period of time, such as 12 to 24 months,39 the evaluators prefer the first approach – selecting a number of key institutions for ‘sustained’ partnership (through a series of short-term STTAs), complemented by opportunistic ministerial support and small grants. This would enable the targeted ministries to receive more than one separate, but related, activities monitored and documented through case studies. The more successful STTAs, or those that had the most lasting impact, had discrete objectives and results, such as the development of a strategic plan, development of a budget, or a specific training for ministry staff complemented by beneficiaries who were keen to receive the assistance and make the suggested changes. One example was the STTA for the Civil Service Agency (CSA).40 DAIM017, ‘Across the Board - Standardizing Pay and Benefits for Civil Service’ conducted by John Fromm harmonized the benefits package for ministry employees at the director level,41 which was requested by the CSA. Fromm’s report is currently being used to inform the broader civil service reform goals that relate to pay and benefits beyond the director level; CSA has since adopted the recommendations. Following this STTA, the CSA realized there were insufficient resources to operationalize the Senior Executive Service (SES), thus requesting assistance to initiate this process. For one month, Carolyn Barnes (DAIM019) and Elena Butler (DAIM020) worked with the CSA to identify and write, through a participatory process with key CSA actors, job titles, descriptions and requirements for 100 SES members.42 The CSA considered the scope of work to be ambitious and therefore the consultants focused on 21 priority CSA SES positions, thus initiating a process that the CSA could continue. STTA, ‘Making Your Own Way: Kemah Town Women Standing Strong’ (DAIM056) is another example of a brief STTA with discrete activities in which recommendations were implemented. The STTA supported the Kemah Town market women through training and guidance on leadership structures, organizational policies, and sound market practices. Due to the technical assistance, the women have implemented new systems and procedures (such as regular monthly reports) and are seeking to register as an independent organization.43 There was some contention about the duration of some STTAs as several were as short as three weeks. Given that some of these were ambitious, such as the Senior Executive Service activity44 for the CSA, the timeframe is critical in determining the likelihood of success. 39 If OTI is not already in country before the BRDG program commences, it can be conducted as a longer program, considering that the average duration of OTI being in country is three years. 40 DAIM017, DAIM019, and DAIM020. 41USAID Office of Transition Initiatives: BRDG-Liberia Second Quarterly Report, 1 December 2006 to 28 February 2007. 42 OTI BRDG-Liberia Activity Database. 43 DAI, Non-key Actor Impact Interview, 27 July 2007. 44 DAIM019 and DAIM020. Kaymah Market (DAIM056) Photo Courtesy of DAI 10 However, the BRDG program learned from earlier STTAs and, in most cases, was able in consultation with the recipient ministry, to formulate discrete activities with more appropriate timeframes. As articulated in the OTI Liberia Program Performance Review Board memo of 22 October 2007, OTI learned that ‘STTAs were more successful if the SOW focused on a discrete activity, such as completing a strategic plan, developing a pay/benefits scale, or providing specific training, for example.’ Reasons for their success included the fact that OTI managed to keep the goals and objectives realistic, which also helped manage the expectations of STTA recipients. STTAs were also made effective by directly involving ministerial recipients in developing the scope of work and recommending possible consultants. BRDG-Liberia’s use of a participatory process for designing SOWs, identifying consultants, monitoring the progress of STTAs and designing the program and budget for the grants was effective. This process amplified the capacity building effect of activities. When ministries did not adopt recommendations, it was predominantly due to the lack of momentum or through the lack of follow-up observations and monitoring. For example, in the Ministry of Justice, where management teams were formed during the consultancy, the teams gradually met less frequently and lost sight of their role, until they eventually became non-functioning. Where STTA recipients did not adopt recommendations, generally due to the level of readiness or willingness of the institution, OTI determined that sustained assistance was not appropriate for these recipients. Early in the program, BRDG made some miscalculations in determining whether an institution had a verifiable interest in implementing the recommendations provided by the STTA consultant. One example was the technical assistance to the Ministry of State for Presidential Affairs (OTIM067) provided through the Liberian Transition Initiative. OTI did not carry this STTA forward under the BRDG program because the leadership did not demonstrate sufficient interest in necessitating changes to implement the Performance Improvement Plan. Can any conclusions be drawn about providing short-term technical assistance to ministries in transitional environments? The STTA mechanism, while it was experimental for OTI, was an ideal approach for implementation within a fledgling government. It was also timely in terms of Liberia’s public sector reforms. STTA activities were based on two factors: (1) an identified weakness within a government institution, and (2) an institution’s demonstrated interest to make improvements. Generally, weaknesses within ministries and agencies were defined in collaboration with the ministry, OTI and the Implementing Partner. A scope of work defined the weakness and the consultancy required to address the concern. This was reasonably well conducted due to the extensive collaboration and knowledge of the country’s rehabilitation strategy and needs. However, the variable factor was the institution’s interest, capability and willingness to implement recommendations for improvement. The following conclusions can be drawn from the provision of short-term technical assistance to ministries in transitional environments: • The Scope of Work should be specific with discrete deliverables to target key problems or issues within an institution or agency, within a realistic timeframe, rather than a ‘catch-all’ for technical assistance; • The consultant should ‘ground-truth’ the Scope of Work in the field and in collaboration with the ministry/agency and OTI; 11 • The STTA consultant should be paired with a ministry/agency counterpart for transferability of skills and as a means to continue to adopt and adapt lessons learned; • When more than one consultant is on assignment, it is preferable to have one designated team leader; • A consultative and participatory approach should be adopted; • Where practicable, the consultant should be accommodated within the ministry/agency for continuous interaction and consultation; • The approach used by the consultant should be practical and relevant to the country and ministry, rather than a ‘text-book’ approach; • The STTA should engage the GOL leadership; and • The STTA should be the impetus to drive forward procedural, process or systems development/reform within the institution at a specifically defined level. C.3 Programmatic Focus Were areas of programmatic focus appropriate and effective for OTI? OTI awarded a total of $1,523,57845 to government ministries for 29 STTAs with an average of $52,537 per STTA to 26 September 2007.46 In the same period, OTI awarded a total of $1,809,128 to local partners for 60 grants, with an average of $30,152 per grant.47 The OTI BRDG-Liberia Activity Database provides a general breakdown of themes allocated against grants and STTAs. Table 2 demonstrates that the capacity building was the major theme of the program under which the most grants were awarded. As requested in the evaluation SOW, the evaluators provide comments on the following programmatic areas: capacity building and public sector reform, the Liberian identity, anti-corruption and transparency, and radio networking. Table 2: Top 10 Themes of the BRDG-Liberia Program, by frequency, and funding THEME NUMBER OF ACTIVITIES % $ TOP 10 FUNDED THEMES 1 Capacity Building 55 12% $2,417,077 Capacity Building 2 Government of Liberia 49 10% $2,055,927 Technical Assistance 3 High Visibility Political Issue 43 9% $1,243,345 Government of Liberia 4 Technical Assistance 41 9% $2,057,816 Strategic Communication 5 Strategic Communication 34 7% $1,355,121 High Visibility Issue 6 Campaign – Public Info 33 7% $752,399 Administrative Support 7 Civil Society 30 6% $929,403 Civil Society 8 Administrative Support 25 5% $1,042,243 Training ($839,089) 9 Youth 24 5% $557,155 Anti-corruption ($776,441) 10 Media - networking 23 5% $732,667 Campaign – Public Info TOTAL 357 75% 357 Source: OTI BRDG-Liberia Activity Database to 26 September 2007 Note: The database assigns more than one theme to an activity C.3.1 Capacity Building and Public Sector Reform Capacity building and public sector reform – what does capacity building mean for OTI? 45 This amount does not cover the full cost of consultants, such as airfares and other operational costs. 46 26 September 2007 was the cut-off date for accessing information from the OTI BRDG-Liberia Activity Database, as determined by the evaluation team. By the end of the program (December 2007), OTI had awarded 36 STTAs. 47 OTI did not set targets for BRDG-Liberia grants due to the addition of the experimental STTA activities. However, the number indicates a reasonable number of specifically targeted grants. 12 All three program objectives included activities that contributed to the capacity building and public sector reform of Liberia. However, it is important to clarify what capacity building means to OTI since in traditional development programs capacity building is a long-term effort and OTI programs rely on being fast and flexible to garner immediate impact. OTI’s approach to capacity building matched the short-term nature of the program as well as each activity to achieve the objectives and program goal. For OTI, capacity building became more of a tool for empowerment via short-term activities, rather than the provision of skills that necessitate constant follow-up and monitoring, which could take place over a longer program. Hence, the capacity building activities of BRDG-Liberia were geared toward short￾term goals and discrete activities such as the provision of equipment, an improved budget, quality radio programming, or the development of job descriptions for a specific government institution. However, some BRDG-Liberia short-term activities will likely have a lasting impact as they provided a catalyst for change and learning. Was the focus on capacity building and public sector reform appropriate and effective? Given the goal of the program, to help the new government of Liberia ‘gain traction and consolidate legitimacy’,48 it was appropriate for OTI to contribute towards capacity building and public sector reform. OTI’s program design stemmed from the new government’s demonstrated reform agenda and willingness to respond to the Liberian people’s needs. Capacity Building Despite the short-term timeline for the program, nearly all activities BRDG-Liberia implemented had an element of capacity building by providing targeted information to empower beneficiaries and launch initiatives that responded to the program objectives and program goal. The nature of capacity building under the BRDG-Liberia constituted activities where ministerial or agency staff, and grant recipients, would, within a short period, build upon specific existing skills, processes, or procedures or learn new methods and strategies to improve individual or institutional capacity for increased productivity. For example, capacity building included the following activities: time management skills, organizational scheduling, streamlining functions, defining departmental or staff roles and responsibilities, writing press releases, financial management and budgeting, procurement procedures, service delivery, documenting action or strategic plans, team-building, leadership skills, marketing, and revenue raising. The activities focused upon how people work, how they interact with each other and other departments, and increasing their ability to transition from plans to the implementation of effective and efficient work practices. Other capacity building activities involved building the capabilities of organizations such as the grants to community radio stations through training and workshops on programming, radio production, editing, software training, and journalism, as well as the provision of equipment. Several stakeholders commented on their improved capacity and ability to offer services to their target audience and the public through the provision of materials, furniture, and equipment. The indirect benefit of office equipment was that it enabled organizations to conduct more internal and external meetings as well as improve their service delivery. To this end, OTI’s short-term capacity building focus was both appropriate and effective towards 48 Final Evaluation Scope of Work, page 1, Background section. 13 increased individual and institutional productivity, particularly in terms of awareness raising and introduction to streamlined or alternative procedures. Public Sector OTI public sector activities were appropriate to the program and to OTI’s implementation strategy but they were not necessarily geared towards reform. The BRDG-Liberia public sector activities were geared to short-term goals that had immediate impact on grant or STTA beneficiaries. Instead of identifying these activities as Liberian public sector reform, it would be more appropriate to say that they existed to jumpstart change in the public sector. The STTA for the Ministry of Lands, Mines and Energy (MLME) involved Paul Temple (DAIM054) to establish the Kimberley Process Certification Scheme within MLME and its regional offices over three months consultancy from May 2007. Temple’s main tasks were to serve as an advisor, conduct training, and provide support for the KPCS working group.49 The US Embassy requested this assistance to fill a gap left by a consultant who worked on behalf of the European Commission that was assisting the KPCS. Temple kept the momentum going by training MLME staff, developing procedures and assisting with the establishment of 10 MLME regional offices, and developing procedures for tracking diamonds from the mines to brokers and dealers through a Mineral Voucher number. He also helped raise awareness in mining communities informing them of the legal parameters of mining diamonds. The Manager of GOL Diamond Office commented that while the consultant’s contribution was too short, Temple was able to create lasting impact by establishing office procedures as well as by finding supplemental grants for equipment and community awareness programs on behalf of the MLME. The public information campaigns that complemented this STTA, implemented through the House of Freedom (HoF) grant called ‘Let It Shine: Kimberley Process Community Theatre’ (DAIM069), was successful at raising awareness and initiating change in the diamond mining business. Because the regional offices of the MLME were tasked with implementing the KPCS, it was understood that if change in the public sector was to be lasting, citizens of the communities and miners needed to understand and support the process. HoF sent teams to the west and central corridors to conduct a total of 100 KPCS performances and answer questions from miners and community members. The MLME regional officers accompanied HoF to answer questions.50 BRDG-Liberia provided important assistance to ensure that the Kimberley Process continued to move forward in making reforms and raising awareness. The work with the public sector was appropriate and directly linked to program objectives, its goal and a window of opportunity to which OTI, through BRDG-Liberia, was responding. 49 OTI BRDG-Liberia Activity Database. 50 OTI BRDG-Liberia Activity Database, DAIM069. Paul Temple Training the Ministry of Land, Mines and Energy (DAIM054) Photo Courtesy of DAI 14 C.3.2 Liberian Identity Were Liberian identity interventions appropriate and effective for OTI? BRDG activities that focused on Liberian identity were appropriate and effective for OTI as they contributed to the program objectives and the reform goals of the GOL. The YMCA conducted activities that raised the issue of the Liberian identity amongst children and youth (DAIM032, DAIM062 and DAIM078). All activities focused on children’s views about their identity using techniques such as art, song, and drama. Although the activities were intended to benefit hundreds of children throughout Liberia, the YMCA realized that there were also thousands of indirect beneficiaries – relatives, parents, friends, and community members who were learning from the children. Additionally, the YMCA representatives at first were not sure what the outcome of these activities would be, but after the first grant, they realized its importance. They are currently researching opportunities to continue the momentum. One idea is to include issues of identity in the national education curriculum. C.3.3 Anti-corruption and Transparency Were anti-corruption/transparency interventions appropriate and effective for OTI? The BRDG-Liberia program contributed, through a niche role, to enhanced issues of anti￾corruption and transparency, particularly at the government level with a ripple effect to communities. Other donors had a long-term involvement in anti-corruption activities, predominantly through the provision of Internationally Recruited Financial Controllers to aid the directive of the Governance and Economic Management Assistance Program (GEMAP). OTI’s role was specific, building upon existing work or focusing upon means by which the community would participate more and become more knowledgeable of government processes. The Copenhagen Conference from 9-11 May 2005 galvanized the donor community into a cohesive, unified force for development assistance to Liberia and spawned GEMAP in September 2005 with an emphasis on anti￾corruption measures to securing the country’s future revenues. The BRDG-program’s role in supporting GEMAP included a public awareness campaign, conducted through a grant awarded to the House of Freedom (DAIM030) ‘GEMAP: One Year Later’. Another effort was through the STTA to Freeport of Monrovia (DAIM004) to establish a transparent, best practice process for competitive bidding and procurement evaluation for improved port operations. Liberian Identity in Nimba County (DAIM032) Photo Courtesy of DAI House of Freedom Production on GEMAP (DAIM030) Photo Courtesy of DAI 15 Additionally, Star Radio worked with the Catholic Justice and Peace Commission to broadcast the results of the Legislative Report Card for the Senate and House of Representatives (DAIM071 and DAIM073),51 which were converted to CD-rom and distributed to each county and aired over community radio stations. This effort had a great impact in empowering citizens to question the actions of their government officials. Because of this program, citizens were more informed about the actions of their legislators and called the station during the broadcast of the report card to voice their opinions and their perspective from the grassroots level. C.3.4 Media – the Star Radio Network Were media interventions appropriate and effective for OTI? Community radio networking was highly effective and appropriate for OTI. It expanded upon the work undertaken through USAID’s Community Peace Building and Development program (2002-2007) to expand radio programming throughout the country by empowering Star Radio through workshops and the provision of equipment. Commencing with a pilot of three stations and expanding to a network of six, community radio stations were able to form a two-way avenue for program sharing. Programming, such as the public campaigns for GEMAP and the Legislative Report Card, was now able to reach the majority of the public, thus providing them with timely, relevant, ‘first-hand’ information. It was exceptionally appropriate for OTI to support the Star Radio network. Star Radio received four grants that helped broadcast information for the government or another grantee to achieve BRDG-Liberia objectives. Three grants were to: (1) air information regarding rural perceptions of the GOL’s first year in office, (2) examine GEMAP one year later, and (3) raise awareness about the budget.52 The fourth grant (DAIM071) was formulated in conjunction with the Catholic Justice and Peace Commission’s grant that developed the Legislative Report Card (DAIM037). The grant enabled Star Radio to air information gathered by the JPC on the Legislative Report Cards via its radio network and provided an avenue for citizens to comment. These media grants catalyzed people to understand the importance of specific issues pertinent to Liberia and build their capacity to be informed and respond to, react to, and advocate for the newly elected national government. The Program Editor from Star Radio remarked that the programs ‘opened the eyes of the citizens’ especially regarding the GEMAP process through a talk show organized with the Minister of Finance. Star Radio translated the highlights into 16 local languages and aired on 19 community radio stations.53 51 According to the OTI Activity Database, the JPC began attending legislative sessions in April 2007 and compiled a monthly Report Card noting the number of times a Legislator attended sessions, introduced bills (and what the bill addressed), voted on bills, and traveled to their constituency. The JPC discussed the report card on a monthly radio program simultaneously broadcast on 4 radio stations (DAIM037). 52 DAIM024, DAIM031 and DAIM48 respectively. 53 Based on information provided in the Outcome Evaluation, 30 April 2007, Roseline Hunter (DAI/BRDG￾Liberia Information and Impact Specialist). Radio Bomi (DAIM002) Photo Courtesy of DAI 16 Star Radio grants also provided material assistance to increase the capacity of community radio stations. The BRDG initiative (DAIM002) worked through Star Radio to establish a network of radio stations for program sharing, starting with several existing community stations as a pilot, by providing them with equipment and training. When some communities saw the model network created by Star Radio, they initiated the formation of their own stations with their own resources. For example, staff from Star Radio explained that in Gbarnga, the community worked in collaboration with the local carpentry skills training center to provide building materials, labor and food for workers to build the station while BRDG provided equipment. C.4 Perception of STTA and Grant Clients How did STTA and grant ‘clients’ perceive and react to the assistance? As part of the evaluation, the evaluation team disseminated a survey questionnaire to 39 STTA and grant clients and beneficiaries in Liberia from 5 to 16 November 2007 to determine their perception of the outcome of BRDG-Liberia interventions. Fifteen surveys (39%) were returned for analysis.54 The majority of respondents (86%) agreed or strongly agreed that their expectations had been met (67% of STTA recipients and 91% of grant recipients). Only 7% of respondents felt that their expectations had not been met. Of grant or STTA respondents, 86% also agreed or strongly agreed that the intervention had been successful. Only 7% of respondents were neutral about the success of the intervention. Respondents indicated the following self￾generated outcomes from a grant or STTA intervention: • Regular meetings and improved interaction with each other & community (20%); • Improved ability of people, especially youth, to discuss their opinions (13%); • Public outreach program (13%); • Office procedures and systems developed (13%); • Increased productivity in the organization; • Improved radio programming content; • Improved unity; • Increased level of civic consciousness; • Shared visions; • Youth participation; • A strengthened parliament; • A completed census for Liberia; • Received office equipment; and • Improved travel to communities and clients. Due to the BRDG interventions, 74% of respondents indicated that they had changed their work practices. Individual respondents had become more active in meetings, realized the need for timely record-keeping, and realized the need for transparency and accountability. Did OTI adequately consider the absorptive capacity of BRDG-Liberia clients? The evaluation team determined that BRDG-Liberia staff did due diligence in determining the absorptive capacity of grantees prior to awarding a grant. BRDG staff and clients addressed issues during the grant design and budget development phases, thereby viewing the 54 A full survey analysis appears in Annex 6. 17 process as transparent. The consideration of the absorptive capacity is evidenced by minimal grant cancellations. Cancellations were generally due to the lack of input by a key stakeholder in the grant activity or a disagreement with regards to the product of the grant and slowness in distributing the product. Specifically, only three grants were cancelled: (1) DAIM029 with the Ministry of Youth and Sports, (2) DAIM052 with House of Freedom, and (3) DAIM055 with Green Advocates.55 Was assistance adequately calibrated? OTI developed budgets in conjunction with grant clients leading to the evaluation team’s observation that they were adequately calibrated. All respondents were open to additional funding for the activities to expand their reach. In one instance, because of unanticipated numbers of participants for the YMCA activities, there could have been additional funding for the grant. YMCA had a ceiling for the number of participants that could attend events. Nevertheless, additional participants arrived. As a result, YMCA staff had to determine alternative ways to provide additional food and materials for newcomers. Additionally, part of the grant included the purchase of digital cameras. YMCA representatives would have preferred funds to include additional participants since they already owned digital cameras. Despite this minor discrepancy, the evaluators did not see any evidence that assistance was not adequately calibrated. What was the clients’ motivation for seeking or accepting OTI assistance? The evaluation team identified four types of client motivations for seeking OTI assistance: 1. To provide direct benefits to the organization, such as funds to implement an activity or procure equipment to bolster the organization’s infrastructure; 2. To quickly provide funding for a specific activity in the absence of other funding; 3. To assist OTI/BRDG-Liberia achieve its objectives by disseminating a message or information for the government or agency; and 4. To expand activities already taking place. Direct Benefits to the Organization Some grant recipients accepted BRDG assistance because it would provide materials or funding to bolster the organization’s infrastructure or prospects for sustainability. An example is the grant for Green Advocates and the Extractive Industries Transparency Initiative Publish What You Pay Coalition (EITI-PWYP) Coalition to develop a resource center (DAIM027). According to Coalition representatives, BRDG-Liberia was the only organization offering support. BRDG also supported two Coalition members to travel to Cameroon for an EITI regional workshop. Star Radio also received grants that helped support its existence and sustainability. Specifically, the ‘Star Radio Relocation’ grant (DAIM044) was intended to strengthen the radio station’s financial status by moving it to cheaper and more strategically placed facilities in central Monrovia. The grant covered the cost of relocation and paid the building lease for two years.56 55 OTI Activity Database to 26 September 2007. 56 OTI BRDG-Liberia Activity Database, DAIM044. 18 Funding for a Specific Activity in the Absence of other Funding In some instances, grant clients were motivated to seek assistance from OTI to fund a specific activity in the absence of other funding. For example, the Ministry of Youth and Sports approached OTI/BRDG to support the ‘National Youth Day’ (DAIM005) in October 2006, celebrated annually since 1969 when it was enacted into legislation.57 There were delays in passing the government budget at the end of the fiscal year and the government could not allocate funds to the event in time. Funding was critical, without it, the MYS would have had to delay the National Youth Day celebrations. Disseminating OTI/BRDG-Liberia Messages In some cases, the motivation of the grantee was to help BRDG-Liberia achieve its program objectives. BRDG-Liberia explored ways in which it could maximize technical assistance to government institutions or the impact of a grant activity. Thus, OTI provided funding to a client to disseminate information or a message created by another actor. For example, HoF was asked by BRDG to implement four grants that helped disseminate information and raise awareness regarding key government issues such as GEMAP, the Kimberley Process and the birth certificate process (DAIM030, DAIM069 and DAIM088). HoF also had a grant that provided a mechanism for the government to communicate other high visibility issues (DAIM010). Expanding Activities Already Taking Place Another motivation for receiving OTI assistance was to expand activities already being implemented. For example, the Liberian Young Men’s Christian Association (YMCA) was established in 1881 focusing on the growth and development of youth through its non￾political leadership and sporting program.58 The YMCA was already conducting activities through its Saturday youth programs. BRDG approached the organization to incorporate the theme of Liberian identity into their existing activities. BRDG provided three grants to the YMCA to expand their Saturday activities to enable children to express their thoughts about what it means to be Liberian (DAIM032, DAIM062 and DAIM078). Was OTI’s assessment of the political situation and strategic importance accurate or were there other contributing political or non-political motivations driving the partners’ decision to work with BRDG-Liberia? The answer to this question is twofold. OTI’s assessment of the political situation and strategic importance was accurate and, given the destructive impact of many years of civil conflict, it was natural for grant clients to seek assistance to support their livelihoods. Clients were motivated by a combination of wanting to bolster the newly elected government and contribute to democratic reform, peace and stability, as well as needing funding to live. However, during the interviews, their responses were predominantly that motivations were for their country and the betterment of the Liberian people and not individual agendas. This is 57 OTI BRDG-Liberia Activity Database, DAIM005. 58 OTI BRDG-Liberia Activity Database, DAIM032. National Youth Day (DAIM005) Photo Courtesy of DAI 19 evidenced in the survey to beneficiaries conducted during the field review of this evaluation. Table 3 outlines respondents’ motivation for seeking OTI assistance. Table 3: Motivation of Grant Clients in Seeking or Accepting OTI Assistance GENERAL MOTIVATIONS SPECIFIC MOTIVATIONS • To help restore Liberia to normalcy after the war; • To continue work with children and youth; • To improve systems; • To broaden partnerships and build synergy; and • To strengthen links between political leaders, communities, and partisans • To equip an office, provide equipment, and enhance capacity (21%); • To promote & sustain peace in Mano River Basin; • To improve radio programming content; • To fund production of communication material; and • To conduct the census project C.5 STTA and Grant Synergies Did OTI and the Implementing Partner effectively tie together the STTA and small grants? Combining STTA and grants in the BRDG-Liberia program was an innovative and effective strategy for OTI. As a result of adopting this approach, OTI activities had both deeper and broader outcomes. Small grants complement the STTA mechanism by reinforcing the training and good governance messages resulting in a multiplier effect. Examples of synergies occurred between STTAs and grants. For example, the Ministry of Land, Mines and Energy STTA (DAIM054) generated two follow-up grants (DAIM069 and DAIM075) to amplify the messages of diamond mining. The HoF series of grants effectively tied together STTAs and grants through public information campaigns, such as the GEMAP process and the Kimberley Process Certification Scheme, as did the Star Radio campaigns. Synergies also occurred amongst grants, and with other donor programs. GEMAP provided an opportunity for OTI to link together the work of other donors with STTAs and grants that contributed towards anti-corruption and transparency activities. An example of GEMAP links were the grants with the Bureau of Budget (DAIM081) and Star Radio (DAIM031). The STTA with the Forestry Development Authority, ‘Trans-boundary Peace Park’ (DAIM072) linked with a larger USG program initiative. Adam Smith International is currently implementing a Department For International Development-funded program aimed to support the CSA and LIPA that follows on from the BRDG-Liberia program with the CSA. The Ministry of Agriculture grant (DAIM001), ‘Employment Horizons for Mano River Youth’ included the establishment of links between USAID, the UNDP and FAO. BRDG-Liberia’s Most Effective Grants and STTAs BRDG-Liberia clearly contributed towards a foundation for institutional change in Liberia. Its interventions can be characterized as ‘leverage for change’ – catalyzing positive change towards (1) anti-corruption, transparency and accountability of government institutions, (2) work practice changes and improved productivity within government ministries, departments or agencies, and (3) public information campaigns that generated understanding, debate, and comment from the community regarding democratic and governance issues. Hence, factors that characterized highly effective OTI activities were as follows: • Synergistic and coordinated activities that linked grants and STTAs, leading to follow-on activities; • Public campaigns that amplified the messages of a specific grant or STTA; • Interventions that led to changes within a ministry, agency or government department, such as positive work practices and updated processes, procedures, or systems; and 20 • Activities that embedded a lasting memory or impact within the public consciousness. Five important successes of BRDG-Liberia underline its aims and objectives: 6. The Legislative Report Card (LRC); 7. The Kimberley Process Certification Scheme (KPCS) technical assistance; 8. Public information campaigns for GEMAP59, LRC, KPCS and the birth certificate; 9. The establishment of the Star Radio network of community radio stations; and 10. Youth-related activities. Beneficiaries and the public have already observed the impact of the above activities indicated by documentation, interviewees and responses to a survey conducted by the evaluation team.60 For example, OTI funded the Catholic Justice and Peace Commission (JPC) to produce and publish a monthly Legislative Report Card (DAIM037) recording the legislators’ attendance and performance at sessions over a period of seven months in 2007. Some legislators called JPC admitting that the process improved and strengthened their actions in the National Legislature, pledging to work harder for the betterment of Liberia.61 OTI awarded follow-on grants to amplify the message through radio and television (DAIM071 and DAIM073). OTI’s STTA to the Ministry of Lands, Mines and Energy (DAIM054), and follow-on grants (DAIM069 and DAIM075), to improve its capacity to implement the Kimberley Process Certification Scheme and to provide legal support to miners in regional camps and communities through drama performances was an innovative way to prevent the flow of conflict diamonds.62 By establishing processes and educating miners, it led to a system that enabled Liberia to mine and trade in diamonds legitimately. The grants for STAR Radio (seven grants totaling $331,100) to develop six new affiliate stations and conduct training enabled the immediate dissemination of reliable and credible information to rural communities, increasing its audience from 100,000 to 180,000 and improving radio programming. The senior senator of Maryland County wrote in a formal communication that ‘the impact of the grant would help foster reconciliation and peace, transfer knowledge, and improve the lives and livelihood of individuals’.63 OTI’s grant to the Liberian Institute of Statistics and Geo-Information Services (LISGIS), ‘One by One - Counting Liberia's Population’, for field communication equipment and GIS materials for mapping population census areas (DAIM050) boosted insufficient funds from the GOL and the United Nations Population Fund (UNFPA). It was also OTI’s highest funded single grant ($205,651). This enabled LISGIS to complete the first national census since 1984. One example of a youth-related activity is the grant awarded to the Liberian Children Parliament (DAIM089) to establish children assemblies in electoral districts within Montserrado County and to train representatives in advocacy, parliamentary processes, 59 GEMAP = Governance & Economic Management Assistance Program – an anti-corruption program to secure Liberia’s revenues. 60 Refer to Annex 6. 61 DAIM037, 071 & 073 Combined Outcome Evaluation, 12 November 2007. 62 Conflict diamonds are rough diamonds used by rebel groups to finance wars against legitimate governments. 63 DAIM043 Star Radio Network Outcome Evaluation, 10 November 2007. 21 financial management, planning and reporting. This activity led to the first Children’s Parliament by elected representatives in Liberia, providing a model that could be replicated in other counties. The process enabled children to discuss national issues and to participate in parliamentary activities. These activities and their outcomes may not have occurred without the funding and support from OTI. The above examples indicate a direct impact of OTI’s interventions and activities through the BRDG program, and the likely potential for lasting changes and benefits to individuals and institutions in Liberia. The evaluators undertook an appraisal of grants and STTAs with the above factors for success in mind and concluded that the following activities were OTI’s most effective interventions (Table 4): Table 4: Evaluators’ Top 5 STTA and Grants TOP 5 STTA TOP 5 GRANTS 1 ‘Diamonds: Make Jobs, Not War!’ – MLME, Paul Temple (DAIM054) & 2 grants House of Freedom (DAIM069) & MLME (DAIM075) ‘Legislative Spotlight: The report Card, Is your Legislator Making the Grade?’ – JPC (DAIM037, DAIM073) & Star Radio (DAIM071) 2 ‘Across the Board: Standardizing Pay and Benefits for Civil Servants’ – CSA, John Fromm (DAIM017) & ‘Senior Executive Service Development’ – CSA, Carolyn Barnes (DAIM019) and Elenda Butler (DAIM020) ‘Employment Horizons for Mano River Union Youth’ – Ministry of Agriculture (DAIM001) 3 ‘Trans-Boundary Peace Park Consultancy’ – FDA, Jennifer Graham (DAIM072) ‘Replicating Hope: Amputee Footballers Take the Lead’ – NCDDRR (DAIM039) 4 ‘RIA Operations Manual’ – RIA, Laszlo Csiky (DAIM064), Sam Koduah (DAIM065) ‘Liberian Identity’ – YMCA (DAIM032, DAIM062 & DAIM078) 5 ‘Container Terminal Contract Bidding Process’ – Freeport of Monrovia, Audu Uba Mohammed (DAIM004) ‘GEMAP: One Year Later’ Interactive Community Theater & Community Radio – House of Freedom (DAIM030) & Star Radio (DAIM031) The following pages highlight a model STTA – ‘Diamond: Make Jobs, Not War!’ and a model grant – ‘Legislative Spotlight: The Report Card, Is your Legislator Making the Grade?’ 22 Diamonds: Make Jobs, Not War! OTI BRDG-Liberia Model STTA Ministry of Lands, Mines & Energy Model STTA Defined: Short-term Technical Assistance (STTA) that has embraced the concept of BRDG￾Liberia by providing strategically targeted support to assist the GOL gain traction and consolidate legitimacy. The STTA Activities: Diamonds have played a pivotal role in the wars of Liberia and Sierra Leone by providing funds for weapons. In April 2007, the UN Security Council lifted its diamond embargo on Liberia. It was admitted to the Kimberley Process Certification Scheme (KPCS) allowing it to export its rough diamonds legally to other Kimberley Process Members. As the UN continues to sanction nearby Côte d’Ivoire’s diamonds, it is necessary for Liberia to establish and reinforce a chain of custody system to demonstrate that the diamonds were mined in Liberia. The GOL is making efforts to be compliant with the KPCS. In response, BRDG-Liberia provided an international diamond expert to the Ministry of Lands, Mines and Energy (MLME) for 3 months from May 2007. The aim was to improve the capacity of the MLME to implement a credible KPCS system so that Liberians can benefit from the natural wealth of their country. The STTA Consultant conducted training and also served as an advisor to the Mano River Union regional efforts to harmonize export taxes and fiscal regimes in the mining sector. The main goal was to lessen incentives for smuggling, a vital intervention towards Liberia's peace and security. Justification for Selection of Model STTA Support to US Foreign Policy: The STTA supports the US Foreign Policy objective by helping the ministry advance peace and democracy. The Ministry: The MLME has 2 ongoing programs with other donors – (1) USAID/GEMAP/LIBAMM, (2) the UNDP D4D program. The objective of the GEMAP/LIBAMM is to improve mining management through resident technical assistance (TA), while D4D aims to improve diamond related legal and policy harmonization within the sub-region. Before the STTA: Before BRDG-Liberia support, there was limited procedures for export and import of diamonds; limited awareness by the public of MLME activities and regulations for mining; and no tracking of exported diamonds. After the STTA: The STTA Consultant provided training to MLME staff, developed procedures and assisted with the establishment of 10 MLME regional offices (ROs). The STTA Consultant also developed procedures for tracking diamonds from the mines to brokers and dealers through a Mineral Voucher Number. Follow-on Activities: The STTA Consultant generated 2 grants: (1) ‘Let It Shine: Government and Miners come together for KPCS’, and (2) ‘Communication and Transparency Initiatives in the Diamond Sector’. The first grant enabled the GOL to establish a Government Diamond Office. The GDO conducted a 2-day seminar to miners and RO staff on their responsibilities in following the certification scheme. House of Freedom presented community theater in mining communities to amplify the message on mining regulations. The second grant facilitated the networking of valuation computers with the MLME database, and webpage development to ensure the transparent flow of sensitive information. Lessons Learned: 1. The STTA was demand driven by the Ministry of Land, Mines and Energy 2. The STTA Consultant’s Scope of Work was specific 3. It was a strategic approach combined with opportunism 4. The STTA and follow-up grants were synergistic 5. Drama and theater amplified key messages to the targeted communities Photo: www.dbddiamonds.com 23 Legislative Spotlight: The Report Card OTI BRDG-Liberia Model Grant Catholic Justice & Peace Commission Model Grant Defined: A grant that has embraced the concept of BRDG-Liberia by providing strategically targeted support to assist the GOL gain traction and consolidate legitimacy. The Grant Activities: The GOL has 3 separate branches, the Legislature, the Executive and the Judiciary. To encourage a more appropriate assessment of the GOL’s performance, the Catholic Justice and Peace Commission (JPC) designed and published a Legislative Report Card covering the legislature's performance over 7 months from March to September 2007. The JPC monitored the legislators each session each month. Unidentified monitors graded individual legislators independently and non-judgmentally on set benchmarks. These included: number of sessions a month; attendance rate; number of late attendances; what legislation the member introduced; what the proposed bill was; how each member voted on critical legislation; and the number of times the legislator visited his/her home district and met with constituents. At the end of the month, the JPC printed and distributed a consolidated Legislative Report Card. The grant covered labor costs and stipends for 4 legislative monitors, 15 county monitors, a data processor, materials and media outreach for the JPC to discuss the Report Card on a monthly radio program simultaneously broadcasted on 4 radio stations. Justification for Selection of Model Grant Support to US Foreign Policy: The grant supports the US Foreign Policy objective by helping local partners advance peace and democracy. The Local Partner: The Catholic JPC has been involved in advocacy for human rights, good governance and democracy in Liberia for over a decade and has presence in all of the 15 counties. Before the Grant: The grant is a completely new concept in Liberia. Legislators had a ‘don’t-care’ attitude to their work, often missed legislative sessions, and rarely visited their constituencies. The public did not know what the legislators did and rarely saw them. After the Grant: People began to understand and comment on their legislator’s attendance and performance. Legislators are now accountable and transparent. People at all levels of the community congratulated the JPC on their initiative and for raising their consciousness. Follow-on Activities: The initial grant led to 2 further grants. Also implemented by JPC was ‘Televising the Legislative Report Card’. Due to swelling public interest, and to amplify the coverage of the Report Card, JPC worked with 4 TV stations to televise the daily proceedings of the House of Representatives and Senate for 8 weeks for an hour in the evening. The second grant, ‘Is Your Legislator Making the Grade?’ implemented by Star Radio, simulcast the Report Card summary each month to rural communities through community radio stations, including through Star Radio’s network of affiliated stations. They also distributed 25 sets of the Report Card to each of the 15 counties. Lessons Learned: 6. There were clear benchmarks to determine legislators’ performance 7. Monitors were independent and non-judgmental – they reported what they observed 8. JPC adopted one clearly defined issue, of public interest, and it received maximum media outreach through follow-on activities. After BRDG-Liberia: The momentum of public interest may diminish and legislators will revert to their previous ways. The JPC is seeking additional funding from donors to continue in 2008. Photo: Martina Nicolls 24 C.6 Follow-on Activities and Projects What examples of follow-on activities have resulted from OTI’s intervention? Given that no formal handover of the mechanism is planned, what has OTI contributed to the overall USG assistance strategy? The focus of the BRDG-Liberia program, supporting the newly elected Liberian government, was also the focus of other international partners’ programs. Without being duplicative, the BRDG-Liberia program offered gap-filling assistance to the longer-term governance programs of other donors. This provided an opportunity for OTI to engage partners beyond the USAID Mission to explore handing over elements of the program. It appears, however, that OTI did not take full advantage of this prospect. Engaging Other Partners There are several ways OTI could have engaged other partners working on governance issues such as the National Democratic Institute, IFES, and the Carter Center. One of the ways would have been to invite them to attend a street drama staged by HoF. Another way would have been to disseminate the Legislative Report Card results to organizations and donors complemented by a letter informing them of radio broadcasts: the station and time. The BRDG-Liberia team could have also brought together one or several Liberian government institutions, such as the Ministry of Lands, Mines and Energy with some implementing partners, to discuss pertinent capacity building issues in a roundtable environment. A grantee recommended that local partners receive a copy of the activity evaluation so that they can learn from any weaknesses that arose during implementation, and so they can demonstrate to other donors their strengths and successes. Engaging other implementing partners in these ways could have encouraged them to adopt OTI initiatives to amplify its program results as was done with Adam Smith International.64 C.7 Monitoring Implementation and Evaluating Impact65 How did OTI’s processes to monitor implementation and evaluate impact contribute to the overall program performance? OTI intended to monitor the implementation and impact of BRDG-Liberia through a series of three STTA consultancies66 conducted by two consultants; one to develop program-level and activity-level indicators and another to document case studies. The purpose of the consultancies was to ‘present an honest account of the achievements and shortcomings of the BRDG program as it unfolds over the course of the year-long program’. The tasks of the I&I consultants were to: (1) develop an operational framework to guide program decision-making based on the three program objectives, (2) design approaches to assessing activity-level impacts, and (3) take periodic ‘snapshots’ of program components and beneficiaries.67 The result of the STTA (DAIM013 – I&I Phase I, DAIM034 and DAIM035 – I&I Phase II) was to ‘contribute to OTI/BRDG-Liberia’s capacity to prioritize specific initiatives, identify 64 Adam Smith International is implementing a Department For International Development-funded program aimed to support the CSA and LIPA. 65 Due to the experimental nature of the I&I process used during the BRDG-Liberia program, a detailed assessment is provided in Annex 5. 66 Phase I (DAIM013); Phase II (DAIM034 and DAIM035); Phase III was not conducted. 67 I&I Scope of Work, 27 October 2006. 25 critical partners, and assess impact of activities'.’68 Overall, the I&I process achieved this result because it provided a mechanism for OTI to assess whether each activity was meeting the program objectives and responding to the program goal. Specifically, it helped OTI learn from each activity and identify other activities to fill gaps or provide additional assistance. The two main expected products were five case studies and a document that outlined the program-level and activity-level indicators. While the case studies were not as helpful as intended in ‘telling the story’ of the program through the lens of selected actors, the indicator work provided a foundation of knowledge that the BRDG team used to develop an I&I system that best suited the program. This work also provided a professional development opportunity for the BRDG staff that were in charge of or participated in the I&I process. The I&I process designed by BRDG-Liberia consisted of three elements. The first element was a participatory STTA or grant design. The second element monitored the STTA or grant progress during the implementation. The third was to conduct follow-up interviews with STTA and grant recipients to determine how the technical assistance was internalized, whether objectives were achieved, and what effect the intervention had on the institution and its staff. The information was documented using one of the following formats: (1) Non-Key Actor Interviews with the STTA beneficiary, (2) an outcome evaluation report based on interviews with grant recipients, or (3) notes in the database that were written based on either interviews or observations saved in the evaluation section of the activity on the database or in the custom field. The process was helpful for the BRDG team to monitor progress and identify the impact of STTA and grant activities. This system also helped them document what type of assistance to provide an institution or grantee. For example, the Non-Key Actor Interview with the Minister of Justice (DAIM006)69 determined that the ministry had not been proactive in following recommendations provided by the consultant. Hence, BRDG-Liberia concentrated on STTAs that would provide assistance for discrete activities resulting in greater impact. The subsequent STTA with the Ministry of Justice (DAIM018) focused specifically on the provision of legal advice on their financial systems. The Outcome Evaluation report awarded the highest possible rating – ‘high positive’ – to the grant conducted by House of Freedom (HoF), ‘Tell it Like it is: Strategic Communication of High Visibility Issues’ (DAIM010) because HoF achieved their activity objectives. Subsequently, HoF received four additional BRDG small grants over the course of the 15-month program.70 Case Studies OTI envisioned that a consultant would produce five case studies that would document the progress and impact of STTA upon government ministries receiving continuous, or ‘sustained’71 intervention over the course of the program. Each case study would include three stages commencing with a baseline assessment (before the BRDG intervention), then a mid-term review (during), and a final assessment of the impact. However, the following two key factors caused OTI to change the scope of the cases studies: (1) ministries did not receive sustained technical assistance, which OTI regarded as receiving ‘limited’72 assistance, and (2) there was a misunderstanding on the part of the consultant about the case study framework. 68 I&I SOW, 27 October 2006 69 Non-Key Actor Interview conducted on 17 May 2007 with the Minister of Justice 70 Grant DAIM052 was cancelled because the GOL was supposed to take a lead role in coordinating the performance, but after two months and little action on the part of the government, the grant activities became less relevant than at the time they were conceptualized (Source: OTI Activity Database) 71 BRDG-Liberia Program Rationale – Update, 22 November 2006. 72 BRDG-Liberia Program Rationale – Update, 31 March 2007. 26 Possible institutions for sustained assistance included the Ministry of State for Presidential Affairs, the Ministry of Planning and Economic Affairs, the Ministry of Youth and Sports, the Ministry of Land, Mines and Energy, the Ministry of Internal Affairs, and the Ministry of Justice. However, for a variety of reasons, such as ministerial staff turnover or the lack of readiness/willingness to receive support, sustained assistance did not materialize. Additionally, despite extensive discussions, no framework or template for the case studies was finalized. The combination of a shift in ministerial partners and a lack of a case study template led to the reduction of the number of case studies, from five to two: one on public sector reform focusing on the Civil Service Agency (CSA) and the Liberian Institute of Public Administration (LIPA) and the other on Star Radio73. Indicator Development The assignment for the other I&I consultant involved three steps as outlined by the I&I SOW: (1) developing activity-level and program-level indicators accompanied by methods on how to collect the data; (2) conducting an assessment of whether the collection of data for the indicators was informing program decisions, determining if the process should be revised, identifying any training or technical assistance needed to improve/keep the process going, and reviewing the key actor interviews and the OTI Activity Database; and (3) writing a final report tying all of the major findings together. The evaluation team concludes that the consultancy that developed program and activity￾level indicators focused on a conceptual, traditional and developmental perspective of monitoring and evaluation when OTI preferred innovative, transitional, practical and applicable information on the indicators. As a result, OTI developed the indicators and the I&I system for the BRDG-Liberia program by adapting, testing and revamping the I&I consultant’s outputs in consultation with the Implementing Partner. Summary Overall, the evaluation team regarded I&I for the BRDG-Liberia program as partially successful, although specific STTA activities did not reach fruition as planned. Specifically, instead of five case studies that provided a before, during, and after perspective of targeted institutions, only two case studies were finalized. Instead of detailing assistance provided to these beneficiaries, the case studies provided a short description of the actors, activities, and outcomes of the assistance. Additionally, the identification of BRDG-Liberia program and activity level indicators from which the activities could be monitored was not fully conducted. As a result, OTI instead of the consultant developed the indicators. During Phase II of the I&I consultancy, OTI realized that the scope of work objectives were not being achieved. Subsequently, OTI cancelled Phase III. While the I&I experiment was expensive, time-consuming and disappointing for OTI, the process enabled a self-generated, ‘lessons learned’ monitoring and evaluation system specific to the needs of BRDG-Liberia. Future applications and experiments should not be discouraged as the process, particularly the case studies, could be replicated for sustained ministerial or institutional technical assistance and partnership over a 12-24 month program using the BRDG mechanism. Activity Database The OTI BRDG-Liberia Activity Database was a centerpiece for the BRDG-Liberia program and was crucial for the final program evaluation. Overall, BRDG-Liberia database users can 73 Jean Poe, Case Studies: Public Sector Reform and Star Radio Network, 6 November 2007. 27 be confident that the information is current, accurate and complete. This is a testament to the efforts of the OTI Country Representative and Deputy Country Representative combined with support from DAI’s Chief of Party (CoP) and Senior Grants Manager. At the beginning of the program, OTI conveyed to DAI that the database was to be a management and reporting tool. Further, at the request of OTI, the database was to be updated every Tuesday in time for the weekly grant meeting on Wednesday mornings. DAI complied with these requests and, as a result, implementing staff were serious and disciplined when entering grant and STTA activity information. The database was not a passive tool, but one that was continually improved. If the DAI team did not understand a particular field or found a field to be redundant, they would ask for changes and OTI-Washington responded promptly with explanations or changes. D. MANAGEMENT AND OPERATIONS FINDINGS D.1 Management Issues What management issues arose in implementing the program? How were these handled? Overall, this evaluation finds that there were minimal management and operations issues that arose during the course of the program. For those issues that did arise, they were not detrimental to the implementation of the program. It is important to note that the working relationship between OTI and the Implementing Agency, DAI, was positive, collaborative, and open. Likewise, the OTI-Liberia Country Representative and Deputy Country Representative had complementary experience with extensive Liberian and management experience and an effective division of labor. OTI-Liberia was able to contribute significantly to the program by ensuring that resources (both human and material) were available to implement successful and strategic activities. Four issues arose during the implementation of the program: (1) understanding a Time and Materials (T&M) contract (BRDG) (2) how ideas for the grants and STTAs were generated, (3) consultant recruitment, and (4) working from a shared vision. T&M (BRDG) While there is a CPFF contract mechanism designed specifically for OTI, OTI chose to use the BRDG T&M contract mechanism. Because OTI had never used a T&M contract prior to BRDG-Liberia, there was a significant learning curve for OTI. Specifically, T&M deals with financial and program implementation issues differently. For example, the T&M contract has fixed labor categories and rates, and no fees are applied to the operational and grant. Subsequently, DAI provided formal and informal training to OTI about T&M contracts. The training proved useful because OTI internalized the information relatively quickly and program implementation was not interrupted. Generating Ideas for Grant and STTA Activities An issue identified by stakeholders that seemed to have caused some frustration was how ideas were generated for grant and STTA activities. For this program, OTI expected that the Implementing Partner would generate many of the ideas. However, OTI provided the majority of ideas and conveyed disappointment that more ideas did not derive from DAI. However, from the outset of the program, it was DAI’s understanding that USAID, OTI and the US Embassy would provide ideas, as discussed during the strategy meeting in November 2006. At this meeting, the Implementing Partner agreed that a USG entity would initiate an activity idea because BRDG-Liberia responded to the needs of high-level officials and 28 relationships were political. Nevertheless, this issue did not detract from, nor stall, program implementation. Consultant Recruitment Consultant recruitment became a challenge for BRDG-Liberia because it was a T&M contract as opposed to a CPFF contract. This challenge manifested in three ways. First, the BRDG T&M contract was designed to facilitate hiring Americans rather than Third Country Nationals (TCNs) or Cooperating Country Nationals (CCNs). This became an issue when OTI requested that many consultants be TCNs, particularly from Africa. One of the reasons OTI selected DAI as the Implementing Partner was because its proposal indicated that it could easily field TCNs using the resources of its subsidiaries in the United Kingdom, South Africa and Palestine.74 However, in reality, with a T&M contract, acquiring the services of a TCN and CCN was complex and took longer than desired, in part, because USAID’s Office of Acquisition and Assistance (OAA) had to be involved in the approval process contrary to OTI’s understanding at the outset. This was disappointing to OTI because they were anticipating tapping into DAI’s vast roster of TCNs from Africa as a quick way to recruit for and to implement STTAs. From DAI’s perspective, they worked diligently to identify and deploy TCNs and CCNs for BRDG-Liberia, however many of these consultants were too expensive for the fixed labor rates or they were not the right fit for the scope of work. Although the recruitment process and deployment of TCNs and CCNs was slow in comparison to recruiting Americans, it did not impede the mobilization of STTA consultants and no STTA had to be cancelled. Another challenge was with the professional levels of the consultants. A T&M contract, and the BRDG IQC, has fixed labor categories. OTI chose not to include junior level consultants in the BRDG-Liberia task order. An identified junior consultant was not eligible under BRDG to implement an STTA. However, from the Implementing Partner’s perspective, this did not have a negative outcome because most of the recipients of the STTA requested and or required senior level consultants. This issue did not result in cancelling an activity because the proposed consultant could not be identified but it was a learning experience for the BRDG-Liberia team. The third challenge of consultant recruitment for BRDG-Liberia resulted because of the short-term nature of the STTAs – some of them as short as 20 days. Three or four cases arose when a consultant withdrew at the last minute, prior to signing their consulting agreement, because of a long-term offer. The Implementing Agency devised systems to avoid these situations, such as receiving a commitment from each candidate to be available with short notice and for short-term. However, consultants were free to withdraw as long as they were not under contractual agreements. Working from a Shared Vision From the Implementing Partner’s perspective, there were two different strategies for BRDG￾Liberia implementation. One was to work with five key ministries as sustained partners throughout the entire year in order to build ministerial capacity. The second strategy was to implement activities to fill immediate needs or gaps for capacity building. After several months, the Implementing Partner realized that the second strategy predominated. At the outset, this did not delay implementation of activities but it did cause confusion for the Implementing Partner when prioritizing STTAs. It also impacted the case study activity for 74 DAI, Technical Proposal in Response to Request for Task Order Proposal under BRDG IQC No. DFD-I-00- 005-00220-00, 7 August 2007, page 19. 29 the Implementation and Impact consultancy. However, after the initial uncertainty, OTI and the Implementing Partner worked together with one vision for the program. D.2 OTI, USAID Mission and US Embassy Views Were OTI’s views of the issues and the way they were handled the same as the USAID Mission’s and broader US Embassy’s views? OTI collaborated with the US Embassy and USAID to ensure that the strategic direction of the program corresponded with the USAID Mission strategy and USG foreign policy goals. USAID Mission representatives noted the positive collaboration initiated by OTI. Additionally, OTI welcomed input from USAID and the Embassy, particularly with regard to the selection of beneficiaries for STTAs and grants. The Mission and the Embassy remarked that the BRDG program provided the US effort in Liberia with a flexible mechanism to address capacity building needs that arose after the transitional government. However, they were less certain about the program’s sustained impact given that the technical assistance, especially in capacity building, was short-term. Overall, it was agreed that the BRDG program implemented activities that complemented the work undertaken by the Mission, such as the Legislative Report Card. Additionally, the Mission appreciated OTI’s work with GEMAP and the Kimberley Process. D.3 Lessons Learned on Management and Operations What lessons learned can be drawn from the management of these issues? Although there were no serious management issues, OTI could apply lessons learned to future OTI programs. Though new for OTI, using a T&M contract did not inhibit the implementation of BRDG￾Liberia. While some processes took more time than desired, OTI adhered to its flexibility and rapid response approach during the program. Many STTA beneficiaries stated that BRDG was able to identify and mobilize STTA consultants promptly and to their satisfaction. Using a CPFF mechanism would provide more flexibility in selecting consultants for the STTA from all levels and countries. However, using CPFF is more flexible and advantageous to OTI. First, OTI is more familiar with this type of contract and second, TCNs and CCNs are easier to hire through a CPFF contract and, at the same time, it is not complicated to hire Americans. Although OTI did a thorough job of collaborating with other USG and non-USG agencies, it could have better educated USAID, the Embassy, and other donors about its program in Liberia and how OTI’s programs were linked to their activities and country strategies. This may have led to a ‘handover’ of some BRDG-Liberia activities, such as the Legislative Report Card that has been embraced by the Mission. Improved donor coordination and cooperation could form a foundation for future programs. Donor coordination constitutes a development resource; improved linkages and relationships may yield positive results. Key partners in institutional support in Liberia are the World Bank, the EC, DFID, and UNDP through its Trust Fund. As the rehabilitation and democratization of Liberia, in accordance with the principles of rule of law, good governance and sound macroeconomic management, is a common aim of the international community, donor coordination is a viable means for a unified country-wide effort. 30 E. LESSONS LEARNED The BRDG program was effective and pertinent to the Liberian context because the new Sirleaf Administration was receptive to public sector reforms at a time when government administrative capacity and systems had been severely decimated after years of civil war. Targeting ministries for short-term technical assistance was a key factor in the success of the BRDG program because the reforms contributed to the country’s strategic objectives. Messages were amplified in areas such as diamond mining, birth certification and passports, the National Budget Process, and the Legislative Report Card. Due to the one-year timeframe, OTI had to make quick decisions regarding which ministries and agencies would be targeted for intervention, and while OTI initially expected that some ministries would receive ‘sustained’ assistance, this did not eventuate. Interventions that provided discrete short-term capacity building to ministries or government agencies, as well as small grants to target the program objectives were the most effective throughout the BRDG program. The most effective activities had the following characteristics: • Activities that were synergistic and coordinated, providing follow-on or linked grants or STTAs; • Public campaigns that amplified the messages of a specific grant or STTA; and • Activities that embedded a lasting memory or impact within the public consciousness. Other possible factors contributing to the program’s success include: • OTI worked collaboratively in close cooperation with its sole Implementing Partner; • The short-term program had three clearly-defined strategic program objectives monitored through a process of rolling assessments; • The program awarded a realistic number of STTAs and small grants that were well￾thought out and well designed, rather than a high number of random interventions; • BRDG-Liberia was a national-level program and therefore most activities were focused in and around the capital; and • The President of Liberia was pro-active to public sector reform and open to dialogue, enabling OTI greater access to the country’s central authority. Lessons Learned: • OTI’s decision to continue its support in Liberia was appropriate and essential, given the inauguration of the new government and the President’s immediate impetus in developing a strategy for public sector reform. • The ‘blurring of the boundaries’ between BRDG and its precursor program, the Liberian Transition Initiative, by commencing a short-term technical assistance (STTA) activity at the end of LTI was a successful strategy. It provided a continuum of activities that effectively merged and evolved into the BRDG-Liberia program and led to improvements in subsequent STTAs. • OTI was strategic in its country program by consolidating its niche areas and supporting a few focused objectives that were reviewed, and modified, as Liberia’s political and security situation evolved. 31 • BRDG-Liberia’s institutional support model (i.e. the use of STTA) was clearly articulated, identified and documented in the design phase, and implemented effectively, significantly contributing to the program’s success. • Instigating change within government institutions by hiring STTA consultants or change agents to provide assistance and advice was appropriate and effective for the activities implemented by OTI and well-matched with the objectives of the overall program. • OTI was clearly receptive and responsive to the evolving political situation and to the nation’s rehabilitation and recovery strategy. • The BRDG-Liberia’s focus remained fixed on key transition points of the rehabilitation and recovery of Liberia, bringing appropriate resources to bear on these events/institutions. • The ability to phase and sequence new types of interventions depended upon the readiness and ability of institutions to implement quality interventions with responsibility and ownership. • BRDG-Liberia’s use of a participatory process for designing the Scope of Work, identifying consultants, monitoring the progress of STTAs and designing the program and budget for the grants was effective. • OTI’s capacity building and public sector change matched the short-term nature of the program and was appropriate and directly linked to program objectives, its goal and a window of opportunity to which OTI, through BRDG-Liberia, was responding. • Community radio networking was greatly effective and highly appropriate for OTI, enabling the public to receive relevant, ‘first-hand’ information. • Of OTI beneficiaries responding to an evaluation survey, 86% indicated that their expectations were met and that the intervention was successful. • Due to the BRDG interventions, 74% of respondents indicated that they had changed their work practices by becoming more active in meetings, realizing the need for timely record-keeping, and realizing the need for transparency and accountability. • Synergies occurred between grants and STTA activities and with other U.S. and donor programs. • While the Impact & Implementation (also known as Monitoring and Evaluation) STTA for the BRDG-Liberia program was time-consuming and expensive, the process enabled OTI to develop a monitoring and evaluation system specific to the needs of the program and could be considered a success. • There was a misunderstanding on the part of the I &I consultant about the case study framework. 32 • OTI and the BRDG I&I team developed a relevant and effective system that made sense for the program in Liberia and created tools that complemented the system, such as the Non-Key Actor Interview and the Outcome Evaluation Report. • One of the reasons for the successful I&I system developed by OTI and the BRDG team was a result of the OTI Activity Database. From the beginning of the program, OTI raised the importance of a well-kept database with DAI. As a result, DAI took the database seriously and worked closely with OTI in Liberia and in Washington to ensure that it was functional. • Four management issues arose during the implementation of the program: (1) understanding a Time and Materials (T&M) contract (BRDG), (2) how ideas for the grants and STTAs were generated, (3) consultant recruitment, and (4) working from a shared vision; none of which adversely affected programmatic implementation. • OTI staff had complementary country knowledge and management skills with an effective division of labor. F. RECOMMENDATIONS Recommendations: • If OTI is commencing in another country with a nascent government, it should ideally implement the BRDG program over an 18-24 month timeframe, enabling time to implement and monitor short-term technical assistance for institutional improvements. • Using a CPFF contract mechanism may be more flexible and advantageous to OTI than a T&M contract mechanism as it enables greater selection of consultants for the STTA from all levels and countries. • OTI can replicate the two-phased institutional support model of short-term technical assistance and small grants in other countries. • OTI should replicate BRDG-Liberia’s use of participatory processes for designing a Scope of Work, identifying consultants, and monitoring the progress of STTAs as this process amplifies the capacity building effect of activities. • If the overall program is short-term in nature, the SOW is more effective when it has discrete deliverables to target key problems or issues within an institution or agency, within a realistic timeframe, rather than a ‘catch-all’ for technical assistance. • The STTA consultant should be paired with a ministry/agency counterpart for transferability of skills and as a means to continue to adopt and adapt lessons learned. • Fewer interventions with synergies and follow-on activities are preferable to a multitude of randomly scattered interventions. • Community radio networking should be considered an ideal intervention for implementing strategic communication activities and amplifying high-profile messages. 33 • Grant clients found it difficult to determine the impact of the activity they implemented because there was no ability to follow-up and no internal assessment. Hence, local partners should receive a copy of the activity evaluation so that they could learn from any weaknesses that arose during implementation, or so they could share and demonstrate to other donors their strengths and successes. The sharing of the final activity evaluation could occur at the close-out meeting (if the grantees are involved and if the grantees are not involved then invite the grantees for part of the meeting that discusses the evaluation of the activity). • Future experiments on Implementation and Impact (Monitoring and Evaluation) should be encouraged and tailored depending on the length of the overall program. For example, the case study methodology could be replicated for ‘sustained’ institutional short-term technical assistance and partnerships over a 12 to 24 month program using the BRDG mechanism. • Donor coordination and cooperation could be improved by disseminating information on BRDG activities as a foundation for potential handover of activities. • The I&I system developed for BRDG-Liberia by OTI and the Implementing Partner can be replicated for other OTI programs as the model is not complex and its application can be adapted provided that there is buy-in from both parties. • I&I is an ongoing process that is more effective with a dedicated consultant or staff person that can provide regular input rather than periodic interactions during the program. Ideally, this person would begin system development following the start-up conference. If a consultant is hired, he or she could be placed on a ‘retainer’ and monitor the system quarterly through data quality assessments and making necessary adjustments. If it is a staff person, he or she will be able to monitor the system daily. • Clear instructions need to be given to I&I consultants so they understand their role and deliverables. • OTI conducted capacity building activities in a short-timeframe – some as short as 20 days and some as long as three months. This generated a question from the US Ambassador regarding whether it was possible to have any impact as a result of such short activities. To avoid this question, it would have been advantageous for OTI to provide him with copies of the Non-Key Actor Interviews so he could see the results and impact of the STTAs. 34 ANNEX 1: ANSWERS - OTI’S MISSION STATEMENT & BRDG PROGRAM 1. Did the program support US foreign policy objectives? Did it accomplish this by helping local partners advance peace and democracy? According to the USAID/Liberia Strategic Statement for 2006-2009,75 the United States foreign policy response to the human suffering in Liberia was ‘through the effective and efficient use of U.S. economic development assistance.’ As a result, USAID developed a program that would ‘establish the conditions necessary for transformational development to take place in Liberia through two goals: (1) avert and resolve conflict and (2) manage crises and promote stability, recovery, and democratic reform.’ USAID devised four strategic objectives (SO) to achieve these goals: 1. Reinforce African conflict mitigation capacity; 2. Increase access to essential services; 3. Advance inclusive governance; and 4. Restore and maintain basic economic activity and livelihoods. The cross-cutting themes for inclusion in each of these strategic objectives were gender, HIV/AIDS, youth and urbanization. Additionally, given Liberia’s history of being simultaneously a victim and instigator of regional conflict, USAID and other US-funded programs were to be designed to support Liberia as a bastion for peace rather than conflict. The BRDG-Liberia program supported US foreign policy objectives for the country and the region. The goal and objectives of the BRDG-Liberia program echo each of the four strategic objectives outlined in the USAID Strategy document. Specifically, OTI designed the BRDG program to assist the new government of Liberia to gain traction and consolidate legitimacy by:76 1. Assisting the Government of Liberia and other key actors to improve capacity in such areas as effective planning, budgeting, communication and coordination with relevant counterparts;77 (SO3) 2. Assisting the Government of Liberia and other key actors to mount effective political responses to high visibility issues; (SO3) 3. Supporting regional activities aimed at strengthening the Mano River Union. (SO1) Further, the BRDG-Liberia program succeeded in following US foreign policy objectives because the program design and the formulation of each grant or STTA activity aimed to help local partners advance peace and democracy. Many grant and STTA recipients were organizations or institutions that demonstrated their commitment to peace, stability and democratic reform within Liberia and within the region, especially in the Mano River Basin. For example, House of Freedom Production Studio and Star Radio received grants to help promulgate messages of democratic reform and peace initiated by different government institutions or non-governmental organizations. Additionally, OTI awarded two grants78 to the Center for Transparency and Accountability to raise public awareness about corruption and urge the government to establish systems to minimize the potential for and pervasiveness 75 USAID/Liberia Strategic Statement: 2006-2009, December 2005. 76 Final Evaluation Scope of Work, August 2007. 77 Objective 1 was adjusted in June 2007 to focus more on strategic communication for the last months of the program. 78 OTI BRDG-Liberia Activity Database, DAIM021 and DAIM045. 35 of corruption.79 Finally, because the countries that comprise the Mano River Basin have a legacy of instability, BRDG-Liberia concentrated several grants to increase the prospects for peace by working with the Mano River Union and organizations such as the Mano River Youth Parliament; the Forestry Development Authority (FDA); the Organization of Liberian Muslim Youth; and the Ministry of Agriculture. The STTA recipients included government institutions and agencies that shared the common goal of democratic reform.80 For example, BRDG-Liberia provided consultants to the Ministry of Justice to assess civilian policing in Liberia as well as to provide recommendations to improve the internal infrastructure and organization of the ministry.81 The Civil Service Agency received two STTAs to help standardize pay and benefits at the director level as well as provide advice on establishing the Senior Executive Service. A final example is the assistance provided to the Ministry of Lands, Mines and Energy to conduct trainings and provide advice and support for the Kimberley Process Certification Scheme (KPCS) working group. OTI assigned a theme to all BRDG-Liberia activities to ensure that they supported peace and democracy,82 demonstrating OTI’s focus on designing activities (grants and STTAs) by which local partners would advance peace and democracy. For example, some activity themes included Civil Society Organization Advocacy, Public Information Campaigns, Capacity Building, Civil Society, Community Based Organizations, Extractive Industries, Gender, High Visibility Political Issues, Inter-Ministerial Coordination, Justice/Human Rights, Land Administration Reform, Media, Mano River Union Strengthening, Social and Cultural Activities, Strategic Communication, and Youth. Additionally, it is evident that the BRDG-Liberia activities also covered cross-cutting areas noted in the USAID/Liberia Strategy for 2006-2009, namely gender and youth. 2. Did OTI work on the ground to provide fast, flexible, short-term assistance targeted at key political transformation and stabilization needs? OTI, through the BRDG-Liberia program, provided responsive, innovative, short-term assistance that supported the country’s strategy of rehabilitation and recovery. Particular activities that contributed towards Liberia’s political transformation and stabilization included youth-related activities, such as the STTA with the Federation of Liberian Youth, ‘Student Debate: Promoting Constructive Dialogue in Liberia’ and the grant with the National Youth Movement for Transparency Elections. Coupled with these effective activities was the first grant with the Ministry of Agriculture, ‘Employment Horizons for Mano River Youth’ that supported the critical border regions in regional stabilization at the Mano River Basin. This involved bringing together youth from Liberia, Sierra Leone and Guinea for employment skills related to food production, marketing and self-sufficiency that addressed socio-economic problems. Addressing youth-related issues through employment and skills training is critical to stabilization. The BRDG-Liberia program also addressed key political transformations within the newly formed democratically elected government. Activities included public campaigns on the Legislative Report Card that documents Legislators’ performance and broadcasts of the information on community radios across the country. Legislators were assessed by 79 OTI BRDG-Liberia Activity Database. 80 OTI BRDG-Liberia Activity Database. 81 OTI BRDG-Liberia Activity Database, DAIM003 and DAIM006. 82 OTI BRDG-Liberia Activity Database. 36 independent reviewers, from April to September 2007, on their legislative session attendances, whether they arrived late, how often they meet with their constituents and whether they introduced a bill. The results were collated and broadcast on a monthly basis. The public were invited by the radio stations to comment on their Legislator’s performance. The activity enhanced the political transparency and accountability of the government. 3. Did the program adapt, as necessary, to ongoing political developments/milestones relevant to the premises of the OTI program in Liberia? Put another way, did OTI’s program seize critical windows of opportunity? How was this accomplished? OTI recognized that its comparative advantage was in flexible, responsive support that could help catalyze more sustained reforms or improvements in the governance process. OTI did this through a combination of STTA to ministries and key government agencies and a small grants mechanism. The two mechanisms were conducted simultaneously and in a coordinated manner such that they leveraged the benefits between them. For example, STTA consultants worked in tandem with relevant counterparts to improve capacity in such areas as effective planning, budgeting, communication and coordination within targeted ministries and agencies. These STTAs linked with grants to further amplify messages, such as the Kimberley Process Certification Scheme to mitigate diamond smuggling. The initial STTA developed systems and procedures, while the two grants established public awareness campaigns through radio programs and a television broadcast. The work commenced in the STTA was followed through for effective dissemination of the processes and information to the public. This was a critical window of opportunity for public sector reform. Other key messages that reflected a window of opportunity that matched the ongoing political and government developments included ‘high-profile’ messages on the promulgation of GEMAP anti-corruption activities, the results of the Legislative Report Card, and the Birth Certificate Process and passport requirements. The information to the nation was both effective and timely, and disseminated through the widely used medium of community radio, as well as through other media such as newspapers, television, and community drama. Overall, the program objectives evolved to respond to the political situation in Liberia, as well as in response to other donor activities. OTI was highly appropriate in changing objectives, showing a responsive and flexible approach to the design and methodology of the program. Sometimes partnerships with targeted institutions changed during implementation to reflect the evolving needs and issues within the public sector and local partners resulting in lessons learned. This informed future decisions and strategies. 4. Did the program, within 18 months of start-up, have a clear strategy and plan for OTI’s exit, and if applicable, a handover plan for continuation of activities/actors relevant to the specific objectives of the OTI program in Liberia? The BRDG-Liberia program was originally a 12-month initiative, launched in September 2006. Initially a $5.9m program, the budget for BRDG-Liberia increased to $6.4m and the duration extended from 12 to 15 months, ending in December 2007.83 For the short-term program, an exit strategy and plan was not formulated, particularly as the perception of the BRDG mechanism was seen to be expensive – for the recruitment of short￾term consultants. Unlike the Liberian Transition Initiative that had clearly defined programs, such as the Youth Education for Life Skills program and the Accelerated Learning Program, 83 BRDG-Liberia extended from 30 August 2007 to 15 December 2007 (Modification No. 2, 25 May 2007) for 240 days for additional STTA senior level TCN labor. 37 there was no discrete BRDG-Liberia program to handover. Although it is too early to assess, some donors and the GOL have commenced discussions to rationalize the carry forward of specific BRDG activities, such as the Legislative Report Card. The focus of the BRDG-Liberia program, supporting the newly elected Liberian government, was also the focus of other international partners’ programs. Without being duplicative, the BRDG-Liberia program offered gap-filling assistance to the longer-term governance programs of other donors. This provided an opportunity for OTI to engage partners beyond the USAID Mission to explore handing over elements of the program. It appears, however, that OTI did not take full advantage of this prospect. However, in a short-term program that requires immediate, responsive, quick impact implementation of activities, OTI was fully engaged in strategizing the selection of targeted assistance to ministerial departments and agencies, as well as in generating small grants that were synergistic with the STTAs and with other grant activities. 5. Does the evaluation find significant impact in strengthening democratic processes and/or increasing momentum for peaceful resolution of conflict? Based upon interviews, observations, I&I information, and the OTI Activity Database, the evaluation finds that the BRDG-Liberia program contributed significantly to strengthening democratic processes and increasing a momentum for peaceful resolution of conflict. OTI’s program complemented other USG and donor program activities with the same goal and similar objectives. BRDG-Liberia grant activities that contributed to peace were the YMCA and CHF activities, which brought together children throughout Liberia to discuss what it means to be Liberian. An indirect impact of these activities was that parents, relatives, and friends also became involved in these discussions. STTA activities that were factors in strengthening democratic processes in the country were the consultancies to support the Civil Service Agency, which had been plagued by nepotism. In addition, it received special incentives during the previous decade resulting in incompatible pay scales and inconsistent allowances for similar positions.84 The objectives of the STTA were to harmonize the pay and benefits package throughout the government, strengthen the CSA’s ability to provide an equitable pay and benefits mechanism for all government employees, and strengthen the GOL’s ability to provide consistency throughout government agencies. The overall goal was to bring more transparency and accountability to the government thereby contributing to strengthening democratic processes. 84 OTI Activity Database, DAIM017. 38 ANNEX 2: SCOPE OF WORK Final Evaluation Building Recovery and Reform through Democratic Governance (BRDG) - Liberia USAID/DCHA/OTI I. Introduction USAID's Office of Transition Initiatives’ (OTI) mission is to support U.S. foreign policy objectives by helping local partners advance peace and democracy in priority countries in crises. Seizing critical windows of opportunity, OTI works on the ground to provide fast, flexible, short-term assistance targeted at key political transition and stabilization needs. II. Background In September 2006, USAID’s Office of Transition Initiatives (OTI) launched the Building Recovery and Reform through the Democratic Governance (BRDG) Initiative in Liberia. This new initiative follows the two-and-a-half-year Liberia Transition Initiative through SWIFT II that began in 2003 as part of the U.S. Government's effort to support the Liberian political transition and helped consolidate peace after 14 years of civil war. The final evaluation of the Liberia Transition Initiative program will be provided and should be considered in this final evaluation, although the scope of this evaluation is limited to BRDG-Liberia. BRDG-Liberia provides strategically targeted support for activities that help the new Government of Liberia (GOL) gain traction and consolidate legitimacy as it undertakes the formidable challenges of national transformation. OTI works to further three specific objectives: 1. Assist the Government of Liberia and other key actors to improve capacity in such areas as effective planning, budgeting, communication, and coordination with relevant counterparts; 2. Assist the Government of Liberia and other key actors to mount effective political responses to high-visibility issues; and 3. Support regional activities aimed at strengthening the Mano River Union. BRDG-Liberia supports these objectives through short-term technical assistance (STTA) to key ministries in the Government of Liberia and civil society organizations and small grants to local partners. III. Objective OTI will be closing out of Liberia by December 15, 2007. The purpose of this task order is to conduct an independent final evaluation of the Liberia program between August 2006 and December 2007. Final Evaluation The nature of OTI’s mandate, including its short-term objectives, shall be a driving factor in the final evaluation. The evaluation shall document accomplishments and lessons learned for the use of both USAID staff and the general public. Specifically, the final evaluation report shall assist and inform the USAID Mission and its partners in Liberia, guide other current and future OTI country programs, and inform the public on the impact of OTI’s work in Liberia. OTI would like specific examples of impact, both in quantitative and qualitative form. The evaluation should address the following questions. The questions are structured around the evaluation first addressing the overall strategy and then detailing the impact of specific activities and series of interventions. The approximate weighting of each section is indicated below. Note that for all questions, “strategy” in this context is more concerned with short-term ingenuity in addressing targets of opportunity and focused attention on achieving medium-term strategic objectives rather than with long-term strategic vision. 39 I. Program Strategy and Implementation (80% of the report) 1. Given OTI’s mandate, did BRDG-Liberia respond in the most strategic way to shifts in the transition process and the evolving political situation in Liberia? OTI prides itself in its ability to identify shifts through rolling assessments and to change the program as necessary to respond to these shifts. Was OTI appropriately flexible and responsive to the “key windows of opportunity” in each of its phases, taking into consideration the broader context including other donor and Government of Liberia activities, political sensitivities, and typical obstacles that exist in transition environments? 2. A unique aspect of this program, when compared to other recent OTI programs, is the STTA. Were the STTAs to assist GOL ministry internal management effective, considering the timeframe of BRDG? Did they lead to any positive and lasting changes within the ministry? Were recommendations implemented? Why or why not? These should be compared and contrasted to ministry STTAs that produced more discrete outputs. Can any conclusions be drawn about providing short-term technical assistance to ministries in transitional environments? Results of STTAs to civil society organizations should also be looked at. How did the STTAs contribute to OTI’s program strategy? Did OTI and the Implementing Partner effectively tie together the STTA and small grants and what kinds of synergies were created? 3. How did STTA and grant “clients” (ministries, civil society organizations, etc.) perceive and react to the assistance? How well did OTI manage client expectations? Did OTI adequately consider the absorptive capacity of BRDG-Liberia clients? Was assistance adequately calibrated? What was the clients’ motivation for seeking OTI assistance or accepting OTI assistance in the cases where OTI set the agenda? Was OTI's assessment of the political situation and strategic importance accurate or were there other contributing political or non-political motivations driving the partners’ decision to work with BRDG-Liberia? 4. BRDG-Liberia is meant to be a catalyst; OTI seeks to fill gaps, initiate processes, and provide alternatives. What examples of follow-on activities or projects have resulted from OTI’s intervention? Given that no formal handover of the mechanism is planned, what has OTI contributed to the overall USG assistance strategy? 5. Were areas of programmatic focus appropriate and effective for OTI? In particular, the following clusters of BRDG-Liberia activities should be addressed, although there may be others identified. ƒ Liberian identity ƒ Anti-corruption/transparency ƒ Youth ƒ Capacity building or public sector reform – What does “capacity building” in a transitional environment mean for OTI? ƒ Strategic communication – Did OTI get the right “high-profile” messages out at the right time in the right way? ƒ Media, particularly the Star radio network ƒ Mano River Union – Did we establish any lasting ties? 6. How did OTI’s processes to monitor implementation and evaluate impact throughout the program contribute to the overall program performance? II. Management and Operations (10% of report) 1. What management issues arose in implementing the program? How were these handled? 2. Were OTI’s views of the issues and the way they were handled the same as the USAID Mission’s and broader US Embassy’s views? 3. What lessons learned can be drawn from the management of these issues? 40 Annex: OTI’s Mission Statement and the BRDG-Liberia Program (10% of the report) In a brief annex, address the following high-level questions concerning OTI’s Mission Statement, criteria for engagement, and BRDG-Liberia. This annex of the final report should not repeat facts, observations, or findings from the first section. Rather it should specifically answer the questions and justify each answer with a summary, referring back to Sections 1 and 2 of the report if necessary. These responses will be utilized in OTI’s Annual Report and for other reporting needs and therefore should be designed for public consumption. 1. Did the program support U.S. foreign policy objectives? If no, please explain. If yes, did it accomplish this by helping local partners advance peace and democracy? 2. Did OTI work on the ground to provide fast, flexible, short-term assistance targeted at key political transition and stabilization needs? If not, please explain. 3. Did the program adapt, as necessary, to ongoing political developments/ milestones relevant to the premises of the OTI program in Liberia? Put another way, did OTI’s program seize critical windows of opportunity? If not, please explain. If yes, how was this accomplished? 4. Did the program, within 18 months of startup, have a clear strategy and plan for OTI’s exit, and if applicable, a handover plan for continuation of activities/actors relevant to the specific objectives of the OTI program in Liberia? If no, please explain. 5. Does the evaluation find significant impact in strengthening democratic processes and/or increasing momentum for peaceful resolution of conflict? If no, please explain. IV. Team Composition (Key Personnel) Final Evaluation 1. One senior-level evaluator with extensive experience (at least 10 years) designing and conducting evaluations of political transition/post-conflict programs. The senior level evaluator will serve as team leader and be responsible for the field review, interviews, the draft and final evaluation reports, and the debriefs in the field and in Washington, DC. 2. One mid-level evaluator with experience (at least 6 years) conducting evaluations of programs in fragile states. Note that regional (West Africa) experience, knowledge of USAID, experience with U.S. Foreign Policy, and experience of political transition/post-conflict (as opposed to long-term traditional development or humanitarian assistance) programming are all necessary qualifications of the evaluation team. 3. One locally-based Liberian assistant who can provide support in advance of and during the teams visit as well as conduct pre-evaluation surveys and/or follow-on research should it be needed. V. Tasks Final Evaluation 1. Recruit and hire three-person final evaluation team (to include one local staff) 2. Participate in a one-day OTI orientation and OTI database training. The evaluators are expected to utilize OTI’s Activity Database as one of instrument in the evaluation of the overall program, particularly in ascertaining relevant quantitative data. 3. Develop, in coordination with OTI, an appropriate methodology for the evaluation. 4. Evaluate the performance and impact of BRDG-Liberia through: a. A literature review of documentation on the Liberia program. All necessary documentation and database access will be provided by OTI, the majority electronically. b. Interviews in Washington DC with current OTI staff, staff from Development Alternatives, Inc., Department of State, and other USAID colleagues. c. Field surveys if possible. 41 d. A field review, including interviews with OTI staff; USAID/Liberia Mission; US Embassy and other USG personnel as needed; Government of Liberia representatives; implementing partners; and program beneficiaries at project sites where feasible. The majority of the field review will be spent in and around Monrovia. 5. Provide an out-briefing to OTI staff in-country before departure. 6. Provide a briefing to OTI in Washington upon completion of the field visit, but prior to finishing the draft report. 7. Document, in a draft evaluation report for OTI comment, findings, conclusions, and lessons learned from the program, as well as recommendations for the future (details on the report are presented below); 8. Provide a final evaluation report for public distribution, incorporating feedback from OTI on the draft, to include a Power Point presentation; and 9. Provide an official public presentation in Washington on the final report. VI. Deliverables Final Evaluation The contractor shall provide the following deliverables: 1. Brief outline of methodological approach for evaluation. 2. A proposed itinerary, schedule for interviews, and list of all logistical support needs for the field visit based on desk review of documents and grants database, interview lists, and initial conversation with OTI and implementing partner staff regarding the OTI Liberia program. This deliverable shall be submitted to the CTO prior to departure to Liberia; however, it can be adjusted within the first few days in Liberia. 3. Draft of the evaluation report, not to exceed 30 pages single-sided with Times New Roman 12 point (or equivalent) font, plus additional annexes (report and annexes to be submitted electronically). The format of the report is flexible, but should be discussed with the country team. The report shall include photographs (to be taken by the evaluators and/or to be selected from OTI and DAI’s photograph collections). The following sections are recommended for the final report: Table of Contents, Acronyms, Executive Summary, Background (OTI’s mission and general approach to programming, country context, evaluation objectives and methodology, overall observations, findings (answers to questions in scope), conclusions, recommendations. 4. Final evaluation report in English, deliverable no later than two weeks after receipt of all comments from OTI on first draft. A total of 20 bound copies should be delivered to OTI in addition to an electronic copy in Microsoft Word format. 5. Final PowerPoint presentation summarizing conclusions and recommendations to be submitted electronically at the same time as the final report. The PowerPoint presentation and Final Evaluation Report should adhere to the USAID graphic standards identified at http://www.usaid.gov/branding/acquisition.html. VII. Proposed Timeframe and Location The Task Order period will begin immediately after signing and extend until January 31, 2008. OTI expects the field visit portion of the evaluation to take place in early November 2007. Work under the Final Evaluation portion of the Task Order will take place in Washington, DC and Liberia. VIII. Government Furnished Documents and Assistance 1. Suggested Interviewee Lists (Washington and Liberia) with contact information. There will be separate lists for the final evaluation and AER with some overlap. 2. OTI will provide database training on OTI’s Activity Database. 3. Final evaluation of the Liberia Transition Initiative program (precursor to BRDG) 42 4. Liberia Activity database, Monitoring and Evaluation documentation, Program Performance Review (PPR) documents, list of project sites, country strategy documents, relevant F indicators and structure for BRDG-Liberia, previous assessments, partners’ statements of work, contractor monthly reports, etc. 5. Monthly reports, success stories, and other documents are available on the OTI website. 6. OTI will work with the contractor’s locally-based Liberia assistant to help arrange in-country logistics, including transport, for field visits. However, the evaluation contractor shall be responsible for all expenses including travel (such as international and domestic flights), per diem, supplies, and other direct costs. IX. Implementation and Management Plan The Contractor shall provide contract management necessary to fulfill all the requirements of this task order. This includes cost and quality control under this contract. X. Performance Monitoring Plan The contractor’s performance shall be evaluated based on the completion of specific tasks as outlined in the Task Order, adherence to the work plan, and reports submitted to the Task Order Cognizant Technical Officer (TOCTO). 43 ANNEX 3: EVALUATION SCHEDULE Evaluation Team The evaluation team consists of the following personnel: Dr Martina Nicolls (Team Leader) is a senior-level evaluator with extensive experience designing and conducting evaluations of political transition/post-conflict programs. She has been involved in Liberian programs since 2004, including the USAID/OTI LTI Accelerated Learning Program; the Final Evaluation of USAID’s Community Peace Building and Development program (2002-2007); and the Final Evaluation of the European Commission’s Institutional Support program (2004-2007). Susan Kupperstein is currently Social Impact’s Democracy and Governance Practice Leader and has experience conducting evaluations of democracy and governance programs. Additionally, she has extensive experience providing technical leadership on program design, monitoring and evaluation and results based management and implementation of political party strengthening, legislative development, and civil society capacity building. Axel Addy is a locally based Liberian supporting the evaluation team with logistics and the dissemination and collection of surveys in preparation for, and during, the in-country review. EVALUATION PHASES Martina Nicolls (Team Leader) Susan Kupperstein (Evaluator) TOTAL DAYS Phase 1: Desk Review 7 7 14 days Phase 2: DC Interviews & OTI Activity Database training 6 2 8 days Phase 3: In-country review 14 14 28 days Phase 4: Post-country draft report writing & presentation 6 1 7 days Phase 5: Finalization of report 5 5 10 days Travel 4 2 6 days TOTAL DAYS 42 31 73 Evaluation Timeframe Generally, two weeks of field review would be inadequate time to conduct a final evaluation of an OTI program. A more in-depth evaluation of specific areas would require more time, particularly in country. However, the extended desk review time, and OTI’s prompt, timely and comprehensive access to electronic copies of documentation relating to the BRDG-Liberia and Activity Database enabled the evaluation team to receive extensive preparation time before commencing interviews in DC. OTI also provided responsive answers to queries submitted by the evaluators at all times, particularly during the desk review. In addition, OTI’s Scope of Work was specific, thoroughly documenting their aims, objectives, a structure and weighting for evaluation questions, and a clear guide to deliverables. The field review did not require community visits with excessive travel (although one visit was recommended) which afforded the evaluation team time to mentally process information gained at interviews, and to monitor the survey questionnaire and prepare an analysis of results for the OTI debrief before departing the country. 44 ANNEX 4: PEOPLE INTERVIEWED Contacts Interviewed Position Organization USG Mr. Patrick Fn’Piere OTI Field Advisor OTI Mr. Adam Reisman Program Manager, M&E OTI Ms. Stacia George Africa Team OTI Mr. Konrad Huber Team Leader, Africa OTI Mr. John Gattorn Program Manager, Africa OTI Ms. Donna Kerner Deputy Team Leader, Africa OTI Mr. John Langlois Media Advisor USAID/OTI Mr. Pete Davis Liberia Desk Officer USG, State Department Mr. Wilbur Thomas Former USAID/Liberia Mission Director USAID (by telephone) Ms. Sharon Pauling Program Officer USAID Ms. Louise Fahnbulleh Administrative Assistant OTI Ms. Musu Clemens Country Representative OTI Ms. Elizabeth Callender Deputy Country Representative, Liberia OTI Mr. Tom Moore Political Officer OTI Ambassador Booth Ambassador US Embassy Ms. Brooks Anne Robinson Deputy Chief of Mission US Embassy Ms. Meg E. Riggs Public Affairs Officer US Embassy Mr. Dan Whyner Lands and Community Forestry Advisor USDA Forest Service Mr. Bob Simpson US Forest Service USG, State Department DAI Ms. Barb Lauer Chief of Party DAI Ms. Aimee Teplinsky Finance Manager DAI Ms. Gathoni Mungai Senior Grants Manager DAI Ms. Roseline Hunter Information & Impact/M&E Specialist DAI Mr. Peter Wilson Program Development Officer DAI Ms. Gwen Appel Former Senior Business Manager DAI Ms. Amanda Brondy Project Coordinator DAI Ms. Heather McHugh Project Manager DAI Mr. Patrick Tarr Program Development Officer DAI Mr. Rhett Gurian Former Senior Grants Manager DAI (by telephone) STTA CONSULTANTS Mr. Richard Blue STTA Consultant, Information & Impact System Assessment DAI (by telephone) Mr. Michael Chelius STTA Consultant, Ministry of State DAI Mr. Sam Howard Journalist/Consultant DAI LOCAL PARTNERS Mr. Paul Seidi Director House of Freedom Mr. Martin Kerkula Chair EITI PWYP Coalition EITI PWYP Coalition 15 Representatives EITI PWYP Coalition Mr. Julius K. Fele Secretary General YMCA Ms. Tilay A. Kollie Program Officer YMCA Cllr. J. Augustine Toe National Director Justice and Peace Commission Ms. Mabel Kear Program Officer CHF Mr. J. Babi Weedor Accountant CHF 45 Contacts Interviewed Position Organization GOL INSTITUTIONS & AGENCIES Hon. Dr. William Allen Director General Civil Service Agency Prof. Harold Monger Director General Liberia Institute of Public Administration Mr. Richard Panton Deputy Director General for Training Liberia Institute of Public Administration Mr. Jerome Wotorson Manager GOL Diamond Ministry of Land, Mines and Energy Mm. Etmonia T Davis Minister of Youth & Sports Ministry of Youth & Sports Hon. Sam E Hare, Jr. Deputy Minister, Youth Development Ministry of Youth & Sports OTHER DONORS Mr. Nellie Mayshak Team Leader, Liberian Civil Service Capacity Building Project (CISCAB) DFID Mr. Emmanuel Altit TRC Specialist European Commission 46 ANNEX 5: ASSESSMENT OF BRDG-LIBERIA’S IMPLEMENTATION & IMPACT Introduction Based on the original scope of work (SOW) for the Information and Impact (I&I) STTA85 (27 October 2006), the goal was to be able ‘to tell the story of the effort at the level of significant program components and the program as a whole’. Further, the purpose of the three-phased consultancies was to ‘present an honest account of the achievements and shortcomings of the BRDG program as it unfolds over the course of the year-long program’. To achieve the goal and adhere to the purpose, the I&I consultants were to: (1) develop an operational framework to guide program decision-making based on the three program objectives, (2) design approaches to assessing activity-level impacts, and (3) take periodic ‘snapshots’ of program components and beneficiaries. The I&I STTA was originally envisioned to be conducted in three phases with specific tasks and deliverables assigned to each phase (see table below): I&I Tasks and Deliverables 5-page summary of findings from desk audit of existing assessments [USAID, UN, World Bank, GOL, etc.] on political context to understand opportunities and constraints facing the new administration in Liberia Short document outlining activity-level and program-level indicators with suggested methods for data collection Phase 1 Up to 5 ‘before’ case studies documenting how key government ministries or other actors interact internally and externally Assessing progress in gathering objective-level and program-level indicators and feeding impact data into programmatic decision-making processes, and suggesting revisions to impact data collection process o formal assessment of tools and processes that evolved since the first STTA o identify training and TA needed o review key actor interview o review OTI Activity Database Phase 2 Up to 5 ‘during’ case studies as to how actors are improving their processes Phase 3 Up to 5 ‘after’ case studies documenting how actors improved their processes Final report tying together all major findings gathered on activity-level and program-level impact Findings The result of the STTA (DAIM013 – I&I Phase I, DAIM034 and DAIM035 – I&I Phase II) was to ‘contribute to OTI/BRDG-Liberia’s capacity to prioritize specific initiatives, identify critical partners, and assess impact of activities.’86 Overall, the I&I process achieved this result because it provided a mechanism for OTI to assess whether each activity was meeting the program objectives and responding to the program goal. Specifically, it helped OTI learn from each activity and identify other activities to fill gaps or provide additional assistance. The two main expected products were five case studies and a document that outlined the program- and activity-level indicators. While the case studies were not as helpful as intended in ‘telling the story’ of the program through the lens of selected actors, the indicator work provided a foundation of knowledge that the BRDG team used to develop an I&I system that best suited the program. This work also provided a professional development opportunity for the BRDG staff that were in charge of or participated in the I&I process. The I&I process designed by BRDG-Liberia consisted of three elements. The first element was a participatory design of the STTA or grant. The second element was monitoring progress during the implementation of the STTA or grant. The third element was to follow-up interviews with the STTA and grant recipients to determine how the technical assistance was used/internalized by the stakeholder institution or whether the grantee was able to achieve the objectives of the activity and the resulting impact of the activity. This system was developed as a result of consultations conducted by 85 There were two STTA phases: (1) from November 2006 to January 2007, and (2) February to July 2007. 86 I&I SOW, 27 October 2006. 47 one of the I&I consultants, but was adapted, tested and revamped by the BRDG-Liberia team led by OTI’s Deputy Country Representative, DAI’s Senior Grants Manager, and DAI’s I&I Specialist. The first step of the I&I process, the design of the STTA or grant, was conducted with the participation of OTI, DAI and representatives from the STTA or grant beneficiary. Together they developed the SOW and budgets for the activity. From the perspective of most of the STTA and grant recipients, this was seen as a positive step in the process because they felt that they had a stake in the activities even though the assistance was in-kind as opposed to cash. As a second step, a BRDG team member monitored the activity or consultancy that was being implemented so that both parties were assured that what was expected to take place was in fact occurring. Finally, after the consultancy or grant activity, the I&I specialist and usually another BRDG team member, met with the STTA stakeholder or grantee approximately six weeks after the close of the activity to interview and determine the outcome and impact of the assistance. The information gathered was documented using one of the following formats: (1) Non-Key Actor Interviews with the STTA beneficiary, (2) an outcome evaluation report based on interviews with grant recipients, or (3) notes in the database that were written based on either interviews or observations saved in the evaluation section of the activity on the database or in the custom field. It is clear that this process was helpful for the BRDG team to monitor progress, and identify the impact of STTA and grant activities. This system also helped them document what type of assistance to provide an institution or grantee. For example, the Non-Key Actor Interview with the Minister of Justice (DAIM006)87 determined that the ministry had not been proactive in following the recommendations provided by the consultant. As a result, BRDG-Liberia concentrated on STTAs for discrete activities that resulted in greater impact. The subsequent STTA with the Ministry of Justice (DAIM018) focused specifically on the provision of legal advice on the Ministry’s financial systems. With regards to the grantees, in the Outcome Evaluation report for the first activity House of Freedom (HoF) conducted under BRDG-Liberia, ‘Tell it Like it is: Strategic Communication of High Visibility Issues’ (DAIM010), it was given the highest possible rating – ‘high positive’ – because HoF achieved their objectives for the activity. Subsequently, HoF received four other grants from BRDG over the course of the 15-month program.88 While the overall I&I system was successful for the BRDG-Liberia program, the components of the I&I activities did not come to fruition as planned. Specifically, instead of five case studies that provided a before-, during- and after- perspective of specific key institutions, two case studies were finalized – one on public sector reform focusing on the CSA and LIPA and the other on Star Radio. Instead of detailing assistance provided to these beneficiaries, the case studies provided a short description of the actors, activities, and outcomes of the assistance. Additionally, the identification of BRDG-Liberia program- and activity-level indicators from which the activities could be monitored was not fully conducted. The document that was developed was more of a guide on how to create indicators appropriate to OTI’s programming. As a result, OTI had to develop the indicators. Because OTI realized that the objectives of the I&I consultancies were not being achieved as of Phase II, Phase III was cancelled. Case Studies As noted above, the intention was to have one of the I&I consultants focus on writing five case studies of key ministries or other key actors as they received BRDG-Liberia technical assistance beginning with a baseline assessment, then a mid-term review, and a final assessment of the impact. However, the following two key factors caused OTI to change the scope of the cases studies: (1) not having any sustained partners to write about, and (2) a misunderstanding on the part of the consultant about the framework for a case study. 87 Non-Key Actor Interview conducted on 17 May 2007 with the Minister of Justice. 88 Grant DAIM052 was cancelled because the GOL was supposed to take a lead role in coordinating the performance, but after two months and little action on the part of the government, the grant activities became less relevant than at the time they were conceptualized (Source: OTI Activity Database). 48 OTI had envisioned that the five case studies would be based on government entities that would receive ‘sustained’ assistance.89 The possible institutions were Ministry of State for Presidential Affairs, the Ministry of Planning and Economic Affairs, the Ministry of Youth and Sports, the Ministry of Land, Mines and Energy, the Ministry of Internal Affairs and the Ministry of Justice. However, for a variety of reasons, the sustained assistance did not materialize. With the Ministry of State for Presidential Affairs, because much of the Performance Improvement Plan (PIP) proposed by STTA consultants went unimplemented, BRDG-Liberia chose to discontinue assistance until further progress was made. With the Ministry of Youth and Sports, BRDG determined that ample assistance had been provided under the previous OTI program in Liberia – the Liberian Transition Initiative – combined with other donor assistance, so the Ministry became a ‘limited’ partner.90 Additionally, despite extensive discussions, no framework or template for the case studies was finalized. This impasse regarding the format may be attributed in part to a lack of comprehension by the consultant regarding what OTI wanted the case studies to look like. Hence, the combination of a shift in the ‘sustained’ partners and a lack of a case study template led to the reduction of the number of case studies and a feeling that there was a disconnect on the part of the consultant in terms of understanding and delivering exactly what OTI requested. Indicator Development The assignment for the other I&I consultant involved three steps as outlined by the I&I SOW: (1) developing activity- and program-level indicators accompanied by methods on how to collect the data, (2) conducting an assessment of whether the collection of data for the indicators was informing program decisions, determining if the process should be revised, identifying any training or technical assistance needed to improve/keep the process going; and reviewing the key actor interviews and the OTI Activity Database, and (3) writing a final report tying all of the major findings gathered on activity- and program-level impact. The product of the first consultancy trip set the tone for the subsequent work on indicator development. The report, entitled ‘Information and Impact System’ (December 2006), was more of an information guide on how to develop indicators for OTI and the BRDG program as opposed to the development of actual indicators. The consultant felt that before the design of indicators could take place, there needed to be a document that helped OTI understand the meaning and process of establishing indicators. An excerpt taken from the background section of the report demonstrates this intention: This report provides OTI-DAI BRDG a proposed comprehensive rationale, framework, processes and tools for managing an integrated Information and Impact data gathering and evaluation system. In doing so, it addresses questions at the forefront of the OTI programming process such as strategic planning, decision making, setting grant (STTA and IT) objectives, as well as providing processes and tools for I and I. It also addresses resources needed to implement the system, most of which are already found in the BRDG staff. The overall consensus of this I&I intervention was that the consultant focused on a conceptual, traditional and developmental perspective of monitoring and evaluation when OTI wanted innovative, transitional, practical and applicable information on the indicators. For example, the consultant thought that OTI would find it helpful to have recommendations for how to make the OTI Activity Database more I&I friendly by using different approaches and changing the way the terminology or concepts were used to guide information gathering and entry into the database.91 The consultant’s trip report, 12 April 2007, stated, ‘While considerable work had been done, the results fell rather short of the introduction and implementation of a systematic and integrated approach to I&I as part of the 89 BRDG-Liberia Program Rationale – Update, 22 November 2006. 90 BRDG-Liberia Program Rationale – Update, 31 March 2007. 91 Interview with I&I consultant, 25 October 2007. 49 program cycle. Still valuable lessons had been learned, and these formed the basis for the modifications, additions and workshop based training which took place.’ As a result, OTI had to develop the indicators and systems for the BRDG-Liberia program and were not completely satisfied with the consultant’s intervention. Analysis of Findings • There was a misunderstanding between the consultants and OTI regarding exactly what OTI wanted. Although the SOWs were clear, during implementation the intentions of OTI and the work of the consultants diverged. For example, with the case studies, the consultant did not grasp what the documents should entail. With the indicator development, the consultant drifted away from creating a system for BRDG-Liberia and instead wrote a reference guide for OTI I&I systems in general. • The OTI and the BRDG I&I team did excellent work in developing a system that made sense for the program and creating tools that complemented the system, such as the Non-Key Actor Interview, the Outcome Evaluation Report, etc. • One of the reasons for the successful I&I system was a result of the OTI Activity Database. From the beginning of the program, OTI raised the importance of a well-kept database with DAI. As a result, DAI took the database seriously and worked closely with OTI in Liberia and in Washington to ensure that it was functional. Recommendations • The I&I system developed for BRDG-Liberia by the OTI Deputy Country Representative, the DAI Senior Grants Manager and the BRDG I&I Specialist can be replicated for other OTI programs. The model is not complex and its application can be adapted provided that there is buy￾in from both OTI and the implementing partner. • I&I is an ongoing process that is more effective when there is a dedicated consultant or staff person that can provide regular input rather than having periodic interventions during the program. Ideally, this person should be part of the implementation design that occurs immediately post-award and would begin the system development following the start-up conference. If a consultant is hired, he or she could be placed on a ‘retainer’ and monitor the system quarterly by doing data quality assessments and making any necessary adjustments. If it is a staff person, he or she will be able to monitor the system daily and make necessary improvements as they arise. • Clear instructions need to be given to the consultants so they understand their role and deliverables. One consultant’s deliverable was more like a teaching guide when OTI wanted an operational template for the Outcome Evaluation reports. This demonstrates that there was a misunderstanding, which should have been dealt with right after the first product was submitted. 50 ANNEX 6: STTA/GRANT CLIENTS & BENEFICIARIES SURVEY RESULTS BRDG-Liberia: Client Perceptions of STTA and Grant Interventions 16 November 2007 Introduction The OTI-funded Building Recovery and Reform through Democratic Governance (BRDG-Liberia) initiative, from September 2006 to December 2007, provided strategically targeted support for activities to assist the Government of Liberia (GOL) gain traction and consolidate legitimacy as it commenced the formidable challenges of national transformation. The three specific objectives of the BRDG initiative included: 1. Assist the Government of Liberia and other key actors to improve capacity in such areas as effective planning, budgeting, communication, and coordination with relevant counterparts; 2. Assist the Government of Liberia and other key actors to mount effective political responses to high-visibility issues; and 3. Support regional activities aimed at strengthening the Mano River Union. BRDG-Liberia supported these objectives through two mechanisms: (1) short-term technical assistance (STTA) to key GOL ministries and civil society organizations, and (2) small grants to ministries and local partners. OTI conducted an external, independent final evaluation of BRDG-Liberia in October and November 2007. As part of the evaluation, the evaluation team disseminated a survey questionnaire to 39 STTA and grant clients and beneficiaries in Liberia from 5-16 November 2007 to determine their perception of the outcome of BRDG-Liberia interventions. The locally-based Liberian logistics assistant distributed and collected the survey questionnaires in the Monrovia area. A challenge for the logistics assistant was the collection of completed surveys, primarily due to the oral traditions of Liberians and their unfamiliarity with or aversion to writing their opinions and views. Fifteen (15) surveys (39%) were returned for analysis, which was considered a good response rate for the purpose of gaining insight.92 Profile of Respondents Of the 15 respondents, six (40%) were from the Government of Liberia and nine (60%) were representatives of the program’s local partners. Seventy-three per cent were recipients of grants, 20% were recipients of a STTA, and 7% were recipients of both forms of intervention. Ninety-three per cent of respondents indicated that they were involved in the decision-making process to receive the intervention from the BRDG program (100% for STTA respondents and 91% for grant respondents). The average duration of the interventions received by the respondents was about 2 months, but ranged from 1 week to 5 months. Motivation for seeking the grant or technical assistance Most responses focused on specific reasons relating to their organization’s missions. Respondents indicated the following motivations for seeking assistance: • To equip an office, provide equipment, and enhance capacity (21%); • To help restore Liberia to normalcy after the war; 92 Educational Benchmarking, Inc. (http://kb.webebi.com) indicates that although higher response rates are valued because they elicit a higher likelihood of obtaining balanced results and lower sampling errors, there is no single standard or ideal response rate. The important factor is the representation of the entire sample. Without a representative sample, no result can be considered valid regardless of the response rate or sample size. EBI indicates that a high response rate is less important for gaining insight than for measuring the effects, or making generalizations, about a larger population. 51 • To promote and sustain peace in the Mano River Basin; • To improve systems; • To improve radio programming content; • To continue work with children and young people; • To broaden partnerships and build synergy; • To strengthen links between political leaders, communities, and partisans; • To strengthen parliament at the local level and to provide a model for neighboring countries; • To fund the production of communication material; and • To conduct the census project. The situation before the grant/STTA intervention Most responses on the situation before recipients received the grant or STTA assistance related to their limited ability to improve their productivity and provide services, as well as the fragility of peace and stability. The responses received included: • Inadequate equipment (20%); • A lack of hope and serious despair (13%); • A state of confusion and political crisis (13%); • A lack of funds to inform the public of what we do (13%); • Growing tension and mistrust; • A lack of shared views about our Liberian identity; • Poor perceptions of the radio station and poor programming quality; • An inactive parliament that did not reflect youth participation; • No key processes and procedures; and • Low capacity of staff. Expectations from the grant/STTA intervention Respondents indicated the following expectations in receiving a grant or STTA intervention: • Improved capacity, performance and productivity of the organization (27%); • Improved systems and delivery of services (13%); • Sustained partnership with the funding provider (13%); • Sensitized and mobilized youth along the border; • Improved delivery of their services; • Youth articulating what the Liberian identity means to them; • Strengthened parliament; • Series of communications materials; and • A completed census. Fifty-three per cent of respondents strongly agreed that their expectations had been met, and 33% agreed that their expectations had been met. Only 7% of respondents felt that their expectations had not been met. Hence, 86% of grant or STTA recipients were satisfied with the BRDG-Liberia intervention. Of the STTA respondents, 67% strongly agreed that their expectations were met (with one no response), while 91% of grant respondents agreed or strongly agreed that their expectations were met. 52 Intervention Expectations Met (STTA & Grant Respondents) Strongly agree Agree 53% 33% No response 7% Disagree 7% Advantages and Disadvantages of the grant/STTA intervention The following chart outlines the respondents’ comments regarding their perception of the advantages and disadvantages of receiving an intervention from the BRDG program: ADVANTAGES DISADVANTAGES • Enhanced systems, capacity of staff and the organization – financial, material & technical (46%) • Helped us empower local communities through more interaction with them (27%) • Enhanced office (13%) • Enhanced coordination of work • Added value • Activity based • Assisted us to institutionalize and strengthen parliament • Too short (27%) • None (20%) • No response (13%) • Poor quality equipment and no vendor information for maintenance/repair (13%) • No evaluation of the intervention • Youth needs were listed but not addressed • Uncertainty about grant continuation • No added value • Funding inadequate Outcomes of the grant/STTA intervention Respondents indicated the following outcomes from a grant or STTA intervention: • Regular meetings and improved interaction with each other & community (20%); • Improved ability of people, especially youth, to discuss their opinions and express themselves (13%); • Public outreach program (13%); • Office procedures and systems developed (13%); • Increased productivity in the organization; • Improved radio programming content; • Improved unity; • Increased level of civic consciousness; • Shared visions; • Youth participation; • A strengthened parliament; • A completed census for Liberia; • Received office equipment; and • Improved travel to communities and clients. 53 Respondents’ behavioral or work change due to the intervention Due to the BRDG intervention, 74% of respondents indicated that they had changed their work practices. Respondents indicated the following ways in which their behavior or style of work changed after receiving the grant or STTA intervention: • Work practices and productivity improved (40%); • More active participation in meetings; • Hired local workforce; • Realized the need for timely record-keeping; • Realized importance of volunteerism; • Realized the need for transparency and accountability; and • Learnt about grant support. There was a significant difference in responses between STTA and grant respondents with regard to behavioral change. Only one of the three STTA survey respondents replied to the question, stating that there was no change. However, 64% of grant respondents indicated that they had changed their behavior to their work practices (27% did not respond). Behavioral Change to Work (STTA & Grant Respondents) No response 13% No 13% Yes 74% Overall perception of the grant/STTA intervention Sixty-six per cent of respondents strongly agreed that the intervention had been successful, and 20% agreed that it was a success. Seven per cent of respondents were neutral about the success of the intervention. Hence, 86% of grant or STTA recipients thought that the intervention with the BRDG￾Liberia intervention was successful. Of the STTA respondents, 67% strongly agreed that the intervention was successful, while 91% of grant respondents strongly agreed or agreed that the grant was successful (9% were neutral). Intervention Perceived as Successful (STTA & Grant Respondents) No response 7% Neutral 7% Agree 20% Strongly agree 66% 54 ANNEX 7: STTA/GRANT CLIENTS & BENEFICIARIES SURVEY SURVEY: GRANT CLIENTS AND BENEFICIARIES (Ministries and Civil Society Organizations) (CONFIDENTIAL: Please note that your information & responses will be kept confidential) Name: (Optional) ____________________________________________________________ Ministry or Organization: ______________________________________________________ Type of grant or short-term technical assistance (STTA) received through the BRDG: ___________________________________________________________________________ ___________________________________________________________________________ Approximate duration of technical assistance (if known): _____________________________ Were you involved in the decision-making to receive the grant or STTA from the BRDG: _ Yes: _______ No: _______ If yes, what was your motivation for seeking the grant or technical assistance: ____________ ___________________________________________________________________________ ___________________________________________________________________________ 1. What was the situation before the grant/STTA: ___________________________________________________________________________ ___________________________________________________________________________ 2. What were your expectations from the grant/STTA: _______________________________ ___________________________________________________________________________ 3. Our ministry or organization’s expectations were met: a. Strongly agree: ψ b. Agree: ψ c. Neutral: ψ d. Disagree: ψ e. Strongly Disagree: ψ Comment: ______________________________________________________ 4. What were the specific advantages of the grant/STTA to your organization’s needs? ___________________________________________________________________________ ___________________________________________________________________________ 5. What were the specific disadvantages of the grant/STTA?___________________________ ___________________________________________________________________________ 55 6. What outcomes resulted from the grant/STTA (what changes took place as results)?____________________________________________________________________ ___________________________________________________________________________ ___________________________________________________________________________ ___________________________________________________________________________ 7. How did you personally react to the grant/STTA: _________________________________ ___________________________________________________________________________ 8. Did your behavior/actions of working change because of the grant/STTA? If so, in what way? ___________________________________________________________________________ ___________________________________________________________________________ ___________________________________________________________________________ 9. Overall, in my opinion, the grant/STTA was successful: a. Strongly agree: ψ b. Agree: ψ c. Neutral: ψ d. Disagree: ψ e. Strongly Disagree: ψ Comment: ______________________________________________________ ____________________________________________________________________________ ____________________________________________________________________________ ____________________________________________________________________________ - - - END - - - 56 ANNEX 8: GOVERNANCE PROGRAM IMPLEMENTERS SURVEY SURVEY: IMPLEMENTERS OF DEMOCRACY AND GOVERNANCE PROGRAMS IN LIBERIA (Non-BRDG staff) (CONFIDENTIAL: Please note that your information & responses will be kept confidential) Name: (Optional) _______________________________________________________________ Organization: __________________________________________________________________ Governance Program in Liberia: ___________________________________________________ 1. I am aware of the USAID-funded Building Recovery and Reform through Democratic Governance (BRDG) program in Liberia: a. Strongly aware: ψ b. Aware: ψ c. Not aware: ψ 2. If so, please list the BRDG activities you are aware of: ________________________________ ______________________________________________________________________________ Please indicate your level of agreement or disagreement with the following statements: 3. BRDG has assisted the GOL and other key actors in improving their capacity to do the following activities: Planning a. Strongly agree: ψ b. Agree: ψ c. Neutral: ψ d. Disagree: ψ e. Strongly Disagree: ψ Budgeting, a. Strongly agree: ψ b. Agree: ψ c. Neutral: ψ d. Disagree: ψ e. Strongly Disagree: ψ Communication, a. Strongly agree: ψ b. Agree: ψ c. Neutral: ψ d. Disagree: ψ e. Strongly Disagree: ψ Coordination with relevant counterparts a. Strongly agree: ψ b. Agree: ψ c. Neutral: ψ d. Disagree: ψ e. Strongly Disagree: ψ Other: ________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ 4. BRDG has assisted the GOL and other key actors in mounting effective political responses to high￾visibility issues, such as ???? (suggest providing a general example or two).: a. Strongly agree: ψ b. Agree: ψ c. Neutral: ψ d. Disagree: ψ e. Strongly Disagree: ψ Comment: ________________________________________________________ ______________________________________________________________________________ 5. BRDG has supported regional activities aimed at strengthening the Mano River Union: 57 a. Strongly agree: ψ b. Agree: ψ c. Neutral: ψ d. Disagree: ψ e. Strongly Disagree: ψ Comment: ______________________________________________________ ____________________________________________________________________________ ____________________________________________________________________________ 5. BRDG has been flexible and responsive to the opportunities for democracy and governance in Liberia: a. Strongly agree: ψ b. Agree: ψ c. Neutral: ψ d. Disagree: ψ e. Strongly Disagree: ψ Comment: ______________________________________________________ ____________________________________________________________________________ ____________________________________________________________________________ 6. Are you involved in donor coordination meetings with BRDG staff: Yes: _______ No: ______ If yes, please comment on the usefulness (or otherwise) of these meetings: ________________ ____________________________________________________________________________ 7. Overall, in my opinion, the BRDG program was successful: a. Strongly agree: ψ b. Agree: ψ c. Neutral: ψ d. Disagree: ψ e. Strongly Disagree: ψ Comment: ______________________________________________________ ____________________________________________________________________________ ____________________________________________________________________________ - - - END - - -