USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT NOVEMBER 2011 This publication was produced for review by the United States Agency for International Development. It was prepared by Janneke Roos and Alemneh Tadele Lake through the Global Health Technical Assistance Project. USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT DISCLAIMER The views of the authors expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. This document (Report No. 11-01-597) is available in printed or online versions. Online documents can be located in the GH Tech website library at http://resources.ghtechproject.net. Documents are also made available through the Development Experience Clearinghouse (http://dec.usaid.gov). Additional information can be obtained from: The Global Health Technical Assistance Project 1250 Eye St., NW, Suite 1100 Washington, DC 20005 Tel: (202) 521-1900 Fax: (202) 521-1901 info@ghtechproject.com This document was submitted by The QED Group, LLC, with CAMRIS International and Social & Scientific Systems, Inc., to the United States Agency for International Development under USAID Contract No. GHS-I-00-05-00005-0. ACKNOWLEDGEMENTS The assessment team appreciates the strong support provided by USAID and its country team, and thanks all those who provided their valuable input in interviews. Our special appreciation goes out to all local partners and their beneficiaries who offered their time to share their experiences in capacity building. USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT i ii USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT CONTENTS ACKNOWLEDGEMENTS................................................................................... III ACRONYM LIST ................................................................................................... V EXECUTIVE SUMMARY....................................................................................VII Introduction............................................................................................................................... vii Summary of Key Recommendations.....................................................................................ix I. INTRODUCTION ...........................................................................................1 Background..................................................................................................................................1 Purpose.........................................................................................................................................1 Assessment Questions..............................................................................................................2 Methodology ...............................................................................................................................2 Limitations ...................................................................................................................................4 II. CAPACITY BUILDING CONCEPTUAL APPROACH ..............................5 III. FINDINGS ........................................................................................................7 Methods and Tools....................................................................................................................7 Common Organizational Capacity Strengths and Gaps ..................................................20 Conclusion.................................................................................................................................28 Key Recommendations ...........................................................................................................29 APPENDIXES APPENDIX A. SCOPE OF WORK.....................................................................33 APPENDIX B. PERSONS CONTACTED .........................................................41 APPENDIX C. LIST OF ATTENDEES ..............................................................47 APPENDIX D. KEY INFORMANT INTERVIEW GUIDES .............................49 APPENDIX E. ETHIOPIA IMPLEMENTING PARTNERS ORGANIZATION CAPACITY ASSESSMENT FOCUSED DISCUSSION GUIDE.................................................................55 APPENDIX F. CAPACITY BUILDING FRAMEWORK...................................57 APPENDIX G. DRAFT OF WORK PLAN ........................................................59 APPENDIX H. DRAFT OF EVALUATION METHODOLOGY .....................65 APPENDIX I. ITINERARY ..................................................................................71 APPENDIX J. REFERENCES...............................................................................79 USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT iii TABLES TABLE 1. ORGANIZATIONAL CAPACITY TOOLS BEING USED IN EACH IP...........................................................................................8 TABLE 2. CAPACITY-BUILDING TOOLS USED BY IPS AND THEIR LOCAL PARTNERS.....................................................9 TABLE 3. LIST OF DIFFERENT MATERIALS USED BY IPS AND LOCAL ORGANIZATIONS....................................................15 TABLE 4. APPROACHES/METHODS TO CAPACITY BUILDING..............16 TABLE 5. LEVEL OF EFFORT............................................................................38 iv USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT ACRONYM LIST AIDS Acquired immune deficiency syndrome APS Annual payment statement CBO Community-based organization CCRDA Consortium of Christian Relief and Development Associations COTOCA Community mobilization tool in the process of building community capacity CSC Child Support Committee CSO Civil society organization CSS Community systems strengthening ESSWA Ethiopian Society of Sociologists, and Social Workers, and Anthropologists FBO Faith-based organization FOG Fixed obligation grant GO Government organization GoE Government of Ethiopia GSM Grant Solicitation Management HAC HIV/AIDS Committee HAPCO HIV/AIDS Prevention and Control Office HIV Human immunodeficiency virus HR Human resource IFHP Integrated Family Health Plan IP Implementing partner HSS Health systems strengthening ISAPSO Integrated Services for AIDS Prevention and Support Organization MARPs Most-at-risk-populations M&E Monitoring and evaluation MOH Ministry of Health NGO Non-governmental organization NPI New Partners Initiative OCAT Organizational Capacity Assessment Tool OSSA Organization for Social Services for AIDS OVC Orphans and vulnerable children PAC Project Advisory Committee PEPFAR United States President’s Emergency Plan for AIDS Relief USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT v PIAI Participatory Institutional Analysis Instrument PLWHA People living with HIV/AIDS PMP Performance management plan PSI Population Service International SCRHA Strengthening Communities’ Responses to HIV/AIDS SOW Scope of work SWOT Strengths, weaknesses, opportunities, and threats TOCAT TransACTION Organizational Capacity Assessment Tool TOT Training of trainers USAID United States Agency for International Development WAC Woreda Advisory Committee vi USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT EXECUTIVE SUMMARY INTRODUCTION Building on the initial emergency response from 2004–2009, the second phase of the U.S. President’s Emergency Plan for AIDS Relief (PEPFAR), which spans from 2009–2013, emphasizes country ownership and sustainability. The United States Agency for International Development (USAID) in Ethiopia has, in collaboration with the Government of Ethiopia, provided support through PEPFAR to strengthen organizations providing human immunodeficiency virus (HIV) and acquired immune deficiency syndrome (AIDS) prevention, as well as non-clinical care and support services. In addition to subgranting to local organizations to provide these services, and improve their technical capacity to enhance the quality of service delivery, USAID, through its implementing partners (IPs), provides technical assistance to improve the organizational capacity of over 600 national and local civil society organizations (CSOs), non-governmental organizations (NGOs), faith-based organizations (FBOs) and community-based organizations (CBOs). Assessment Scope and Methodology An in-country assessment of the organizational capacity-building efforts of select PEPFAR/USAID partners was conducted September 13, 2011–October 6, 2011, by a team of two independent external consultants: A USAID/Washington capacity-building technical advisor and a pair of USAID/Ethiopia health systems strengthening specialists. USAID/Ethiopia developed the scope of work (SOW), provided input into the assessment methodology and the assessment instruments, and participated in the site visits. The purpose of this capacity-building assessment is three-fold:  Assess the appropriateness of the various capacity-building tools and approaches used by the partners  Identify the effectiveness of the local capacity-building interventions  Identify potential future directions for capacity-building interventions The assessment examined the progress being made to sustain three integrated and reinforcing components of capacity building: individual/workforce development: institutional strengthening and system development; and resource use, management, and external organizational environment. The data collection methods that were used included document review, in-depth key informant interviews, focused discussions with beneficiaries, site visits to six out of the nine regional states and two city administrations, online surveys, and direct observation. Six IPs that undertake local organizational capacity building were included in the assessment: 1) World Learning’s Grants Solicitation and Management (GSM) works on prevention of HIV/AIDS, orphans and vulnerable children (OVC) care and support, and local organizations' capacity development; 2) Geneva Global’s New Partnership Initiative (NPI) works on HIV/AIDS prevention, care and support, and capacity building; 3) Pact, which works in capacity building and OVC; 4) PATH’s Strengthening Communities’ Responses to HIV/AIDS (SCRHA) project; 5) Save the Children USA’s TransACTION works on most￾at-risk populations (MARPs) HIV/AIDS prevention programs; and 6) Pathfinder’s Integrated Family Health Program (IFHP) implements reproductive, maternal, neonatal, and child health-related activities. USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT vii Besides interviewing the six IPs in Addis Ababa, group discussions were used for getting information on current capacity-building practices and lessons learned from the staff of local organizations, as well as those who participated in capacity-building activities, including community committees and beneficiaries. Analysis of the interviews with USAID/Ethiopia Staff, the six IPs, and 18 local organizations, together with focused discussions held with beneficiaries and national stakeholders (four regional HIV/AIDS Prevention and Control Offices [HAPCOs], one regional health bureau, and the Ministry of Health [MOH]), was combined with data from documents and direct observation to develop findings and draw conclusions and recommendations. Limitations included time constraints to analyze results, absence of good baseline data, and outcome targets for capacity-building activities and overall performance. Findings Informants agreed that in several instances, capacity-building support by the IPs has contributed to local organizations maturing into more professional institutions, in some cases expanding their action radius to other districts and even to other zones. Informants cited that skills, knowledge, and abilities were often transferred to other program elements. Capacity assessments are conducted across the board, with tools that are more or less similar. More often the approaches are focusing on gap identification; there would have been better alignment with community accountability had these approaches been more needs-focused. In general, capacity building is embedded in the SOW and the work plans of the organizations (IPs and local organizations alike), and to a certain extent is included in the overall monitoring and evaluation (M&E) framework. In the absence of a capacity-building framework, there is no evidence that suggests that capacity-building assessments are used as a baseline, and that the assessment tools are being used to rigorously track capacity-building progress and outcomes against indicators and benchmarks toward maturity. Outcome indicators mostly relate to evidence of a completed successful activity for each result. Although most informants recognized the importance of quality data, in most instances the focus was mainly on the availability of data; data demand and use was widely viewed as an area in need of attention. Capacity building is mostly focused on enhancing individual knowledge and skills through training and supervision. A well-structured organizational development strategy is generally lacking. Training is not necessarily based on the needs of local organizations, but serves the purpose of complying with USAID regulations and work planning. In addition, in the absence of central training databases, the ability to track the number of persons trained is often not possible. Local organizations expressed varying appreciation for the quality of supervision by their headquarters, which at times may be controlling rather than problem solving. In contrast, supervision carried out by local organizations to the sub-sub partners was generally perceived as more supportive in rolling out capacity to the community. In many instances, the improvement of human resource (HR) management has been cited, and many organizations have HR policies, manuals, and benefit packages in place. Especially in FBOs, there is strong commitment due to the shared vision. In several instances, an improved working environment contributed to lower staff turn-over. However, several local organizations still struggle with leadership and management issues, affecting transparency and staff retention. Good collaboration between local organizations and government at the local levels, operating through coordinating bodies with multisectoral representation and interface with the community, was observed as a step toward increasing country ownership and sustainability. Such structures at the community and viii USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT district levels include Project Advisory Committees (PACs), Woreda Advisory Committees (WACs), HIV/AIDS Committee (HACs), Children Support Committees (CSCs), and government organization (GO) and NGO Partnership Forums that were established and institutionalized by the IPs. Government stakeholders cited the challenge of effective capacity-building coordination as a major concern in the HIV response, especially at the regional and district level, and expressed the need for a more strategic framework for capacity building. Such a framework should help harmonize capacity building to ensure alignment with government priorities and standards, while allowing for measuring progress, not only in regards to efficiency and output, but also in relation to outcome effectiveness and impact. Flexibility in budget allocations for capacity building to local organizations is limited. In most instances capacity-building budgets are managed directly by the IP. Local informants identified areas in need of improvement, such as slow turn-around times on budget requests, which at times create serious disruptions. They also expressed concerns that, in the absence of effective exit strategies, target beneficiaries may be left out before new funding comes in. In general, the proportion of the budget allocated for capacity-building activities compared to the overall program/project budget is low for all IPs. Capacity-building budgets for institutional capacity building through the provision of materials and equipment are itemized under operations/administrative lines, which take up approximately 30% of the budget, per the country’s NGO Operation Legislation. This compels most IPs to make limited contributions toward building long-term organizational development and viability with local organizations. Although partnerships are valued for their knowledge, best-practice sharing, and avoidance of duplication of effort, there appears to be room to manage knowledge more effectively by networking and strategic partnering toward common goals, both at the level of the IPs and among their local partners. Web sites could be effective tools in promoting experience sharing (such as communities of practice, Skype conversations, webinars), but they are underused; more traditional channels only offer critical “how to do” knowledge management. It was noted that the organizations could take greater leadership in mainstreaming gender issues in their organizational development. SUMMARY OF KEY RECOMMENDATIONS For USAID  To better evaluate capacity-building achievements, both the Government of Ethiopia and PEPFAR/USAID should identify indicators and targets for progressive levels of capacity built, as well as graduation at local organization levels. For new capacity-building activities, baseline information and outcome targets should be included in the design.  To build more sustainable capacity, USAID should consider outside technical assistance for organizational capacity building to help develop in-country talent for organizational development work. The in-country talent can work with local IPs, while being mentored by the outside technical assistance. The in-country talent could include a university (business school) or other in-country organizational development consultants.  USAID should assist government stakeholders in developing a strategic framework for capacity building at the national level that is aligned with national priorities for HIV/AIDS and the broader health and social sector, which would promote the development of more strategic, systematic, and measurable approaches to capacity building at all levels  USAID should consider direct funding through Fixed Obligation Grant (FOG) and Annual Program Statement (APS) mechanisms for more established organizations that demonstrate adequate financial USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT ix management systems. These systems enable them to further channel funds to CBOs through small grants that require relatively simple proposals and accounting.  USAID should consider direct funding to larger network or umbrella organizations to create a critical mass at the local level. This would lessen the administrative and management burden, resulting in greater cost-effectiveness and efficiency. For IPs and Local Organizations  IPs and local organizations should promote the value of capacity building by linking funding requirements to plans, actions, and goals. IPs should tie expectations for comprehensive capacity building plans to funding, as well as provide assistance in building the capacity of organizational and health system leaders so they understand how to develop such plans, how to implement and monitor specific capacity-building activities, and how to evaluate and report progress.  More emphasis should be placed on assisting organizations in understanding the synergistic relationships between the levels of capacity building and developing the primary functions of leadership and governance as part of organizational development. Leadership and governance are especially critical, since they bind and promote the effectiveness of all the other functions.  In closing the gap between technical capacity-building and organizational development, IPs should conduct needs based capacity-building assessments and shift from a project-based approach toward an organizational capacity development approach, thereby allowing more flexibility for local organizations to manage their own capacity-building budgets.  IPs should adopt more innovative approaches to capacity building and organizational development. Areas of innovation to be expanded are geographic information systems, knowledge management, and data quality assurance. For enhanced knowledge management, IPs should promote a broader definition, one that encourages field staff to reach out to others working on similar issues and share lessons on “how-to-do” through the traditional channels, as well as through widening access to the Internet, which would allow for using inexpensive social media tools such as Skype and webinars in addition to the current practices of experience-sharing through review meetings and regional cluster program review forums.  IPs and local partners should systematically include gender concerns in capacity-building plans.  IPs and their local partners should do more to foster strategic partnering (in larger networks or umbrella organizations) toward national priorities in the following ways: through heightened attention to the organizational development factors affecting health system and community system strengthening; through indicators of progress in health system strengthening (HSS) and community systems strengthening (CSS). They should step up advocacy with national stakeholders and the Mission and show how organizational development can improve sector-wide performance in HIV/AIDS. x USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT I. INTRODUCTION BACKGROUND Since 2004, USAID/Ethiopia has, in collaboration with the Government of Ethiopia (GoE), provided support through PEPFAR to strengthen organizations providing HIV/AIDS prevention, and non-clinical care and support services. These include support for orphans and vulnerable children (OVCs), prevention among MARPs, HIV counseling and testing, condom distribution, behavior change communication, prevention with positives, palliative care and support for persons living with HIV/AIDS (PLWHA) and support for income generating/economic strengthening activities. In addition to sub￾granting to local organizations to provide these services and improving their technical capacity to enhance the quality of service delivery, USAID, through its IPs, has also provided technical assistance to improve the organizational effectiveness of over 600 national and local civil society organizations (CSOs), non-governmental organizations (NGOs), faith-based organizations (FBOs), and community￾based organizations (CBOs). With a new five-year Health Sector Development Program IV (2010–2014), the country’s Growth and Transformation Plan (2011–2015) and the U.S. Government Global Health Initiative in place, USAID/Ethiopia will build upon the successes and lessons learned from its institutional strengthening interventions to-date. Capacity building of health facilities and local organizations to improve the non￾clinical services they provide is an integral part of the health system strengthening (HSS) approach under PEPFAR and the Global Health Initiative. It is critical to ensuring country ownership and sustainability of PEPFAR programs. To further build on successes and identify areas for future enhancement, an in-country assessment of the organizational capacity interventions of select PEPFAR/USAID partners was conducted from September 13, 2011, to October 6, 2011. The conclusion of this assessment will inform programmatic decisions regarding future institutional capacity-building approaches and activities. The scope of work (SOW)1 describes the purpose and the proposed approach/assessment methodology, timeline, and deliverables of the assessment. PURPOSE The primary goal of the assessment was to determine the strengths, weaknesses, and best practices of local institutional capacity-building approaches used by select USAID implementing partners (IPs), and to provide a set of practical recommendations to the Mission for its future investments in institutional capacity-building strategies and activities. The results of this work will ensure that USAID supports programs that have proven to be effective. Objectives of the assessment 1. Describe the various approaches, models, and tools used by partners to design, implement, monitor, and evaluate local capacity-building activities, and to assess whether these approaches, tools, etc., align with published or documented best-practices and recommendations on local capacity building. 2. Identify strengths, weaknesses, challenges, and best practices with respect to local capacity- building interventions. 1 See Appendix A for the SOW. USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT 1 3. Identify innovative capacity-building interventions that can be reinforced and/or replicated in future programs, and make recommendations on how USAID can standardize support, monitor, and evaluate systems of IPs working on capacity building of local organizations. ASSESSMENT QUESTIONS Specific evaluation questions to address in the assessment: 1. What methods and tools were used by the implementing partners to assess the needs of local organizations, design appropriate capacity-building interventions based on identified gaps, implement the intervention, and monitor and evaluate improvements in capacity? a. What key areas of capacity building/organizational development are being targeted by the IPs? b. What are the common features and main differences among partners’ approaches to capacity building? c. Are partners using tools and approaches that have been previously tested or are based on standardized/evidence-based practices? d. How do IPs measure, monitor, and evaluate improvements in capacity over the long term to ensure the sustainability of their interventions? Are there approaches to providing continuous quality improvement, addressing staff turnover, and ensuring the diffusion of skills transfer and learning? 2. What are the most common organizational capacity strengths and gaps identified in terms of human, institutional, material, financial, and technical and project management among targeted CBOs? a. What do beneficiaries, IPs, and other stakeholders perceive are the contributing factors for both strengths and gaps? b. What do beneficiaries and IPs perceive are the most effective methods/approaches for closing the performance gaps for each capacity improvement area? c. What are the main challenges and best practices for working with local organizations? 3. What approaches to capacity building and organizational development should USAID reinforce or scale up in the future? a. Are there interventions/approaches to capacity building that should be discontinued? If so, why? b. Should USAID consider having a separate entity that provides technical assistance on organizational capacity building to all IPs under the Health, AIDS, Population and Nutrition Office? If so, how should this be designed? c. Should USAID consider having capacity building integrated or mainstreamed among IPs? If so, what should USAID do to better standardize interventions and indicators/metrics to measure improvement among the IPs? What are specific recommendations for programming to include in future capacity-building plans? d. Is there a tested and/or recommended monitoring and evaluation (M&E) system for the organizational capacity-building interventions? If yes, what are they? If no, what do you recommend? METHODOLOGY The assessment team consisted of an international consultant with experience in organizational development and evaluation, a national capacity development specialist, and a logistics coordinator. The lead consultant focused on program-wide capacity-building issues affecting overall performance of the IPs and their sub-grantees, client satisfaction, and future directions. Together with the capacity 2 USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT development specialist, she conducted field-based interviews. The capacity-building development specialist focused on reviewing and analyzing all of the capacity-building and training aspects of the assessment, in addition to providing input and comments on other aspects of the assessment, and sharing responsibility for the preparation of the final report. Kenneth Sklaw, the USAID/Washington capacity building technical adviser, assisted in conducting the assessment and provided input in the final report. The USAID/Ethiopia team consulted with the assessment team on the development of the work plan and data collection methods, and participated in the field visits. The assessment used five primary data collection methods: review of key documents; in-depth key informant interviews, both at the federal level and in the regions visited by team members; focused group discussions; online searching; and direct observation. Documents included reports and publications of USAID and the IPs that describe tools, approaches, and progress.2 Team members reviewed the IPs Web sites, as well as other relevant e-sources of information. Key informant interviews with stakeholders were conducted in person.3 The interviews followed a semi￾structured format using an interview guide that allowed for relevant, unplanned discussions.4 In all, six international IPs, 18 national IPs, eight beneficiary groups, and six government organizations were interviewed, representing the projects, government sectors, and NGOs. Team members visited all regions with substantial U.S. Government investments in local NGOs as case studies of the technical and organizational areas of capacity building. In view of the considerable sample size and the limited period of time, there were two teams, one visiting the northern, northeastern, and northwestern parts of Ethiopia, and the other in the southern, southeastern, and southwestern parts of the country, over a period of 12 days. A total of 18 sub-partners in 17 towns in Ethiopia, including the capital city, were visited during the assessment. In addition to conducting key informant interviews, the assessment team members held group discussions with former project trainees, observed the premises, and took stock of the manuals, guidelines, and monitoring and evaluation information available at the project sites. In total, the assessment teams received input from country-based individuals representing GOs, IPs, USAID, NGOs, and trainees. The evaluation team issued a brief survey to obtain budget information for capacity building related activities from the IPs. Key informants interviewed included:  USAID Mission staff/focal persons, including those working on health systems strengthening and capacity development.  USAID IPs— Pact, Geneva Global, World Learning, Save the Children, Pathfinder, and PATH staff.  Staff of a select three local organizations supported by each of the six IPs.  Government of Ethiopia representatives including Regional HIV/AIDS Prevention and Control Organization (HAPCO) office staff, Ministry of Health (MOH) and Regional Health Bureaus. 2 See Appendix F. 3 See Appendix B for organizations and individuals interviewed. 4 See Appendix D for the questionnaires and survey instruments. USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT 3 Focused discussions were used for interviewing local organization staff—both management and administrative—and those who have participated in capacity-building activities (e.g., training and workshops) supported by the IPs. Local subgrantees of the IPs were specifically selected for the assessment, and included samples of those that had low, medium, and high performances after the capacity-building interventions by the respective IPs. Criteria for the selection of the CBOs/NGOs were: 1. NGOs/CBOs needed to be sufficiently significant to make an impact. 2. Capacity building needed to improve networking and institutional capacity for service delivery (high, medium, low). 3. Capacity building needed to improve networking and institutional capacity for sustainability (high, medium, low). 4. Capacity building needed to strengthen data demand and use (high, medium, low). 5. Capacity building needed to contribute to successfully to generate, organize, process, and disseminate useful knowledge in support of operations (high, medium, low). LIMITATIONS Organizations’ Sample Selection: The site-visit data collection was drawn as a purposive sampling based on the criteria of high, medium, and low performance of the local partners of each IP. The use of a purposive sampling rather than a randomly drawn sampling limits the potential for generalizing the assessment findings. In view of the considerable geographic distance, the sampling size and the nature of the assignment was limited, compared to the large number of local organizations supported under the program. There existed a considerable variance between the IPs’ perceptions of performance, in some instances what was considered a high performing organization by one IP would score at the lower end of another IP, which hampered their ability to create a good comparison between the IPs. Selection of Informants: Although the interviewers had some input into what organizations were interviewed, the IPs selected interviewees. In some instances there was a mix of head office staff, local NGO staff, and beneficiaries, which might have influenced the responses. Time Constraints: The time allocated for the in-country assessment limited data collection, analysis, and writing. Due to considerable travel distances, time for interviews was sometimes inadequate. Lack of Base Line Data for Capacity Building: The lack of baseline data and outcome data for capacity-building activities were limiting factors in this assessment. The study questions required the assessment team to make a determination about the levels of satisfaction on capacity building in terms of perceptions, which may not have provided sufficient insight into the actual situation. 4 USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT II. CAPACITY BUILDING CONCEPTUAL APPROACH Capacity is here defined as the ability or power of an organization to apply its skills, assets, and resources to achieve its goals.5 Just as capacity is not static, but instead requires continuous renewal, capacity building is also a continuous process of improvement within an individual, organization, or at the system level, rather than a one-time event. It is an internal process, but it may be enhanced or accelerated by outside assistance, e.g., by organizations providing support in the form of technical assistance, training, mentoring, and coaching. Capacity building emphasizes the need to build on what exists, to utilize and strengthen existing capabilities, rather than arbitrarily thinking of starting from scratch.6 An essential aspect of capacity building should be to build capacity to cope with change and to inculcate a more holistic and integrated approach to thinking about addressing problems at hand, rather than traditional, sector-oriented ways of thinking. In conclusion, capacity building is a broad concept, which overlaps with and includes HR development and various management issues and trends. For the purpose of this report the competency areas are defined as:  Organizational governance and leadership: These include policies and regulations; strategic and operational planning; and program and grant management.  Financial management: This includes budgeting and accounting; financial control and reporting; purchasing and material provision.  Human resource (HR) management: This includes skills, motivation, and the opportunity to make the best possible contribution to the NGO, as well as that which it requires.  Monitoring and evaluation: These include monitoring and evaluation framework; monitoring and evaluation planning; database development; supportive supervision; and checklists.  Service delivery: This refers to technical and sector expertise, community ownership, and the ability to measure the impact of a program.  Resource mobilization and infrastructure: These include adequate resources and cash flow, and a diverse resource base.  Networking and partnerships: These include collaborative and supportive relationships with communities, government agencies, advocacy for the NGO’s own interest and that of its members, and access to local resources to contribute to its overall sustainability.  Knowledge management.  Systems and documentation. Effective capacity development programs work in genuine partnerships, adopting a process approach with long-term perspective and commitment. They also monitor, as well as coach, and support the partners during the change process. Partnerships should develop and change over time, with outcomes that demonstrate increased leadership by local and national partners.7 5 Organizational Capacity Building Framework-AIDSTAR-Two, 2010. 6 UNESCO. 7 “Draft” PEPFAR Capacity Building Framework 2011. USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT 5 The relationships between the “parts-to-be-improved” and the “whole” within a country and international frameworks are often lost. Capacity development is an attempt to see that whole.8 The “draft” PEPFAR Capacity Building Framework 2011 reflects an integrated and reinforcing set of capacity building activities that address the individual/workforce level, organizational level, and systems levels of capacity to further host leadership in addressing HV/AIDS. The document encourages U.S. Government teams to examine their programs through the lens of the framework.9 8 Qualman and Bolger, 1996. 9 See Diagram 1 of the Capacity Building framework. 6 USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT III. FINDINGS METHODS AND TOOLS Key Areas of Capacity Building and Organizational Development For NGOs, the most important component of the value chain should be the process through which they develop, implement, fund, and measure programs. Crafting a successful process—one that increases social impact—draws on the full range of an organization’s skills. Across all organizations in this assessment, capacity building input is geared toward closing the skill gap by increasing competencies and efficiencies at the individual/workforce level, organizational level, and at the system/policy level. Individual and workforce level capacity building activities are typically within the context of organizational development. Most IPs are also strengthening community systems and local government bodies. Key areas of capacity building at the workforce level improve the performance of staff according to specific, defined competencies, such as accounting, database development, outreach activities, etc. Organizational capacity building is intended to facilitate and accelerate the development of sustainable institutions, and thus to strengthen the ability for organizations to finance, plan, manage, implement, and monitor programs, both in the immediate and longer term. They do this through the strengthening of internal organizational structures, administrative systems and processes, quality assurance systems, leadership and management, resource mobilization, and overall staff capacity.10 Organizational capacity building in the organizations included in this assessment is based on institutional assessments by the IPs. They used a variety of organizational capacity assessment tools, which are designed to understand the level of capacity development of an NGO, and the areas that need the most attention to build the capacity of the NGO and to plan for its future development. These tools are the Organizational Capacity Assessment Tool (OCAT), used by Pact, PATH, and Pathfinder, the community mobilization tool in the process of building community capacity (TOCAT), used by Save the Children; and the Participatory Institutional Analysis Instrument (PIAI), used by World Learning. Geneval Global uses the NPI Capacity Assessment tool. In addition, to determine strategies to build the organizational and technical capacity of the NGO, the assessment tools may also be used to organize scarce resources, and to develop systems, procedures, manuals, training plans, or organizational monitoring and evaluation plans. The different tools and the organizational development areas to be assessed, per IP, are presented in Table 1. 10 AIDSTAR-II Organizational Capacity Building Framework. USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT 7 Table 1. Organizational Capacity Tools Being Used in Each IP Name of IP Specific tool Key target areas Pact Ethiopia OCAT Governance: Legal status, board of directors, fiscal committee, the executive team, mission and goals, beneficiary group, leadership. HR: Staffing, HR development, internal work style, gender issues, supervision, salary and benefits. Financial Management: Budget, financial control and inventory management, financial reports. Program Management: Program development, sectoral expertise, beneficiary group involvement, program monitoring and evaluation, program reports. External Relation and Partnerships: Relationship with the beneficiary, relationship with NGOs, relationship with government, relationship with public and private donors, and media. Sustainability: Program sustainability, institutional sustainability, financial sustainability. PATH SCRHA OCAT Organizational Development/Institutional Development: Project Planning, coordination, monitoring and evaluation, support supervision, communication and advocacy, quality assurance, HR management, resource mobilization and management, HIV/AIDS programming. Organizational Capacity: Monitoring and evaluation, program design and development, governance and structure, HR management, financial management, resource mobilization and management, advocacy and communication, networking, linkages and collaboration. Capacity to provide technical assistance to other CSOs. Pathfinder OCAT Governance: Board, Mission, goals, legal status, stakeholders, International leadership. Ethiopia/IFHP Management Practice: Organizational structure and culture, planning, personnel, program development, administration procedures, risk management, administration procedures, information systems, program reporting. HR: HR development, HR management, work organization, diversity. Financial Resources: Accounting, budgeting, stock control, financial reporting, diversification of income base. Service Delivery: Sectoral expertise, stakeholders’ commitment/ownership, assessment, marketing and awareness building. External Relations: Stakeholder relations, inter-NGO collaboration, government and funder collaboration, public relation, local resources, media. Sustainability: Program/benefit sustainability, organizational sustainability, financial sustainability, resource sustainability. Save the Children US/ TransACTION Technical and Organizational Capacity Assessment Tool (TOCAT) General Management: Governance and leadership, strategic and operational planning, structure (including roles and responsibilities), staffing, and HR management, partnering and networking, adequacy of physical infrastructure. Finance: Financial planning and budgeting, cash, banking, 8 USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT Name of IP Specific tool Key target areas accounting, and record-keeping. Logistics and Information: Procurement, distribution, stock and inventory management, management information system. World Learning PIAI Financial Management: Financial control, accounting, budgeting, Ethiopia/GSM audit/external financial review, resource base. Sub-grant Administration, Monitoring and Evaluation: Monitoring and evaluation of staff, planning and implementation, database and information management systems, performance. Staff Roles: Task management, performance management, and staff development, salary administration, team development, and conflict resolution. Governance: Board/other governing body, Mission, legal status, and constituencies. Leadership, Management of Information: Administration, planning, communications, program development and implementation, sectoral expertise, community ownership, impact assessment. Public Relations: Government collaboration, NGO collaboration. Advocacy and Resource Mobilization Geneva NPI Capacity Governance and Management: Human and financial resource Global/NPI Assessment Tool management. Networks and Linkages, Program delivery, Monitoring & Evaluation, Physical Infrastructure The organizational assessment tools provide mechanisms that contribute to the sustainability of an NGO, provided all of its members are involved in the process and the findings are followed through by a comprehensive capacity-building plan that rigorously monitors and evaluates not only outputs, but also outcomes and impacts of the capacity-building interventions. In the key areas of financial and grant management, all IPs have applied financial and grant management tools such as financial and grant reporting formats and computerized applications. The different capacity building tools used by IPs and their partners are summarized in Table 2. Table 2. Capacity-building Tools Used by IPs and Their Local Partners Tools Application of Strengths Gaps the tool Capacity assessment tools OCAT, TOCAT, PIAI, and Strengths, Weaknesses, Opportunities, and Threats (SWOT) analysis Baseline, planning, and monitoring There are many different organizational capacity assessment tools designed to get information about an organization’s capacity to conduct its core business, such as OCAT, TOCAT and PIA, SWOT, which are used as baseline, planning and monitoring tools. These globally tested tools have been adapted to the Ethiopian context. Only a few of the local organizations found in the assessment are using specific capacity assessment tools. USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT 9 Tools Application of Strengths Gaps the tool It was found that generally IPs apply these tools to examine common areas of organizational capacity, such as leadership, management, administration, organizational resources, organizational structures and systems and institutional linkages. However, there appears to be a wide variety in the application of these tools at the level of the local organizations. Organizational development tools Organizational Management and All local organizations have board Although board and governance and leadership and general assembly, bylaws, have general assembly exists in leadership tools designed strategic plans, and have local organizations, the (including produced financial policies, and board is often not well bylaws, internal procedures manuals and guidelines. functioning. It was noted policies and Some IPs (PACT, Geneva Global that their involvement in procedures, NPI, and World Learning GSM) supporting the guidelines, helped their local counterparts in organizations’ overall board) revising their bylaws, policies, and strategic plans. Only in few instances did IPs provide leadership and management training. Local partners do undertake different governance and leadership related capacity building activities themselves, or with very limited support from IPs. growth was too limited. The contribution of IPs in governance- and leadership-related areas is limited. Financial and Financial and Both IPs and local partners have grant grant done a lot in this area. Reporting management management and formats in compliance with IPs and tools (including compliance USAID regulations are available. financial and All IPs have trained staff from local grant reporting organizations and are regularly doing formats, and follow-ups on the application of computerized financial and grant management applications) tools. Program managers have been included in different trainings on financial and grant management. All IPs and local organizations use computer applications (Excel, Peachtree, or QuickBooks) for financial and grant management activities. Some efforts have also been seen in supporting local organizations to revise their financial policies and procedures for a more transparent 10 USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT Tools Application of Strengths Gaps the tool system. Internal and external audit systems are built in all organizations. HR tools Recruitment and All organizations have their own HR Although HR policies are (including HR management of manual. in place, most manuals) HR Some local organizations, like the Ethiopian Gubaye Egiziabher Betekiristian, are very flexible in addressing staff needs, and have created room for negotiations. organizations do not revise these based on need, nor do they address staff turnover and dissatisfaction. There is a gap between local organizations and IPs in deciding on salary and benefits. Most organizations have lost staff since their respective IPs were not supporting salary and benefits revisions. Some organizations, like the Organization for Social Services for AIDS (OSSA), are very rigid in their own policies on HR, which has caused a considerable staff turn-over. Resource Fundraising and Many local organizations are actively The assessment team mobilization resource working in raising funds from abroad. found little evidence that (including mobilization Pact has supported all of its local IPs support their local proposal partners with different capacity partners in fundraising and development building supports for resource resource mobilization. tools) mobilization. Resource mobilization for specific capacity building and organizational development activities is limited. Capacity building is mostly included in work plans as staffing, training, infrastructure, and similar supports. Knowledge Documentation Team learning, including the sharing of Web sites are underused, management and sharing of experiences, knowledge, and skills lessons learned between individuals working in the organization, as well as cross-functional learning (e.g., sharing lessons between finance and program staff) is taking place. The regional clusters of Geneva Global NPI are platforms for a ‘Communities of Practice,’ where each participant comes to when they could be utilized to promote experience-sharing (such as communities of practice, Skype conversations, webinars) and critical “how to do” knowledge. “How to” knowledge resides in the USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT 11 Tools Application of the tool Strengths Gaps contribute/share traditional knowledge and tools as well as to learn. IPs also conduct experience-sharing at local and regional levels. In most organizations, documenting good practices is in place. Regional clustering by Geneva Global facilitates partnerships, networking, and linkages. Such clustering aims at integrating HIV/AIDS efforts for strategic response, which minimizes duplications, allows resource-sharing and strong linkage to ensure sustainable and effective programs. The implementers are encouraged to systematically link with each other, with other non‐ NPI implementers in the regions, and with government structures at all levels in the regions. field, but it is widely dispersed, and more needs to be done to make it available to a broad audience of local NGOs and CBOs. It was noted that awareness of, and demand for, quality data needs to be increased, and the data quality assurance needs to be improved. Monitoring and evaluation tools (including monitoring and evaluation framework, monitoring and evaluation planning, database and checklists) Monitoring and evaluation of programmatic activities All IPs have monitoring and evaluation frameworks, plans, databases, and supervision checklists. Some IPs, have designed interesting tools for collecting change stories, like PATH’s SCRHA. Some local organizations, such as Health Industries Distributors Association, have developed and applied their own framework for monitoring and evaluation. Databases for capacity building activities, specifically trainings, exist in all IPs and some local organizations. Common Features/Main Differences in Approaches Capacity-building Assessments The competency areas of organizational development provide a description of an NGO’s overall state of development. Many NGOs, however, will likely have one or more specific areas of interest that clearly relate to decisions they need to make in the near future, for example the need to develop a strategic plan, prepare a staff development plan, provide grants to sub–sub-partners. The purpose for which an organizational assessment will be carried out also determines how the analysis is carried out and what relative measures are used to determine the difference between the NGO’s current and past performance. When IPs use the assessment tools with their local partners, it is common for staff to be given the opportunity to self-reflect on the performance of their organization. In most organizations, the first step in the process was an orientation workshop to familiarize staff with the concepts of capacity building— 12 USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT often with an external consultant—the objective of capacity building, the tool, as well as the process of the actual assessment. As a next step, a self-assessment was conducted, using detailed forms with criteria for each element of organizational capacity building, with or without the help of an external facilitator. The outcome of this first step should result in a consensus reached about each criterion within the competency areas of organizational development. In view of turn-over in the local organizations that were included in the assessment, the team had difficulty in determining the extent to which these processes had been followed according to the guidelines of the tools, especially when it concerned reaching consensus on where the focus should have been with a view toward sustainability. This step should have been followed by the development of a capacity-building M&E framework, detailed capacity-building training plans, planning for partnerships and referrals, and quality assurance. Training plans were usually there, but the other outputs were only found in Geneva Global. It was found that generally IPs apply these tools to examine common areas of organizational capacity, such as leadership, management, administration, organizational resources, organizational structures and systems, and institutional linkages. Capacity-building Methods Ideally a capacity-building intervention plan includes a variety of strategies, including technical assistance, coaching and mentoring, training/skills enhancement, and infrastructure and system development. Summaries of how the organizations in Ethiopia are utilizing these methodologies are explained below. It was found that capacity building was often paired with training, and that therefore capacity-building plans often focus mainly on training, without offering a comprehensive and systematic approach to organizational development. Technical assistance is here defined as the provision of know-how in the form of personnel, training, research, and associated costs. Findings from the assessment indicate that technical assistance is often geared toward the rolling out of individual responsibilities aimed at achieving specific tasks in compliance with work plans and developing job descriptions. Technical assistance includes one-on-one mentoring or coaching, feedback on proposals, papers, or M&E plans, and guided application of skills while working on a task, such as joint problem solving or planning with a mentee. In many instances it took extensive probing to get the local organizations to recognize these as capacity-building strategies.11 Geneva Global’s staff was engaged with six selected implementers in a more profound technical assistance support along the criteria for graduation during the project duration. As a result of this technical assistance, IPs are expected to develop various plans and systems to strengthen their systems, and have improved/enhanced proficiency to manage PEPFAR/USAID funds as well as grants from technical donors. World Learning measures stages of NGO development using PIAI. The PIAI analyses NGO’s organizational development in four stages: start-up, developing, consolidating, and mature. It was found that trainings are mostly basic trainings, such as grant management, financial training, proposal and report writing, community mobilization, and training of trainers (TOT) with a view to cascading/sharing knowledge and skills and strengthening competencies at the lower level. Pre- and post￾test evaluations are usually carried out for TOT. However, in many instances there is no follow-up of training at regular intervals. As a consequence, training objectives may not be achieved. In addition, in the absence of central training databases, tracking of persons trained is often not possible. 11 It should be noted that the terms mentoring and coaching are being used interchangeably, although in practice it is mostly coaching, focusing on performance. USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT 13 Organizational development training may include HR planning and volunteer management, change/risk management, leadership training, and knowledge management. It was observed that the focus on organizational development differs per organization, partly depending on the needs (the larger local partners already had their strategic plans and systems in place), and partly because the IP was traditionally more focused on service provision, like Pathfinder or PATH. Some IPs (Pact, Geneva Global NPI, and World Learning GSM) helped their local counterparts to revise their bylaws, policies, and strategic plans. It was noted that (supportive) supervision is carried out by most IPs to make sure activities on the ground are running well, and that they are in compliance with agreed upon standards. There appears to be varying appreciation for the quality of supervision. In some instances it was said that it has a rather controlling and/or auditing character, whereas in other cases it really serves the purpose of problem solving. Supervision carried out by local organizations to the sub–sub- partners is generally perceived as more supportive, with attention paid to the real problems and efforts made to find solutions together with the sub–sub-partners. Joint monitoring visits provide a learning opportunity. A major challenge to supervision, communication, reporting, and community level capacity building is the often-limited infrastructure of local organizations, such as access to transportation, computers, or the Internet. Both IPs and local partners have put much effort into strengthening financial management. Reporting formats are in compliance with all IP and USAID regulations. All IPs have trained staff of local organizations and are regularly doing follow-ups on the application of financial and grant management tools. Grant managers have been included in different trainings on financial and grant management. All IPs and local organizations use computer applications (Excel, Peachtree, or QuickBooks) for financial and grant management activities. Some efforts have also been made to support local organizations in revising their financial policies and procedures for a more transparent system. For some, developing the community system is also important. TransACTION Save the Children has created community mobilization teams and provided basic training and TOT using TOCAT, thus increasing local capacity to address HIV/AIDS in the community. TransACTION works with local community-based and traditional institutions, Eddirs, and Ekubs to capacitate them to design, plan, implement, monitor, and evaluate HIV/AIDS related activities. Representatives of Eddirs and Ekubs are now members of HAC, and are providing prevention and care and support services for HIV-positive people and OVCs affected by HIV/AIDS. Evidence-based Tools and Approaches All the assessment tools being used have been designed elsewhere and have proven their value at the international level, both in for-profit and not-for-profits sectors, such as OCAT, which is widely used by USAID. These tools have proved to be appropriate to the Ethiopian context. For example, Pact’s organizational capacity assessment tool is a product of several years of research and field practice. “This comprehensive process brings together communities of peer organizations (or complex organizations with multiple departments/program offices) to reflect upon their performance and set strategies to broaden their impact and affect significant, positive and lasting change.”12 Different training materials in support of prevention, care, and support developed and used by other international stakeholders have been adopted by the IPs and local organizations. The assessment team found several examples in the field, which are presented in the following table. 12 Statement from interview of USAID/Ethiopia Implementing Partner PACT on September 18, 2011. 14 USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT Table 3. List of Different Materials Used by IPs and Local Organizations Training Material/Manual Sources Deutsche Stiftung Weltbevoelkerung (DSW) reproductive health manual Produced by DSW and local organizations, and IPs reproduce it for distribution MARPs manual Produced by Health Communication Partnership for smart journey project, and reproduced by IPs Home-based care providers guide Produced by MOH, adapted by local organizations OVC care giving basic package Produced by Population Service International (PSI), reproduced by local organizations and IPs Local M&E manual Produced by PSI, reproduced by local organizations and IPs Community sensitization workshop guide Produced by PSI, reproduced by local organizations and IPs Basic business skills training guide Different organizations produced their own basic business skill manual and guide Positive living training manual Produced by PSI, reproduced by local organizations and IPs Anti-AIDS clubs training manual Produced by DSW and local organizations, and IPs reproduce it for distribution Geneva Global Capacity Building Monitoring and Evaluation Framework Geneva Global NPI Ethiopia NPI Ethiopia Capacity Building Plan (October 2009– September 2010) Geneva Global NPI Ethiopia NPI Ethiopia Graduation Guideline Geneva Global NPI Ethiopia Pact Ethiopia OCAT Pact Ethiopia PATH/SCRHA Project National Implementing Partners Capacity Assessment Tool PATH SCRHA Project PATH/SCRHA Project National Civic Society Organizations Capacity Assessment Tool PATH SCRHA Project Pathfinder International Ethiopia—IFHP IP Organizations Financial and Material Management Policy manual Pathfinder International Ethiopia IFHP Save the Children U.S./ Ethiopia TransACTION program OCAT Save the Children US World Learning Ethiopia/ GSM PIAI manual World Learning World Learning Ethiopia/ GSM PIAI matrices World Learning World Learning Ethiopia/ GSM Supportive supervision manual/checklist World Learning Ethiopia USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT 15 More research is needed to test the effectiveness of these tools and approaches. Measuring, Monitoring, and Evaluating Improvements in Capacity Building Some of the IPs and their local partners use the assessment tools to rigorously track capacity-building progress. However, there is very little evidence that these assessments are included in M&E frameworks for capacity building. In general, capacity-building efforts are aligned to the cooperative agreement and PEPFAR indicators. In association with these indicators, tools have been developed to monitor improvement as a result of activities on quarterly basis. But there is no evidence of measuring capacity￾building improvements over the long term, which would indicate a desire to sustain these interventions over time, such as evaluations of leadership and management. Geneva Global uses a monitoring framework to measure progress against deliverables for each IP. They agree that monitoring change is a challenge. This is mainly done on a one-on-one basis, through review meetings and annual capacity-building plans. The monitoring framework includes process indicators and output indicators mainly, thought it is recognized that outcome indicators also need to be included. Approaches and methods, their strengths and weaknesses are summarized in Table 4 below. Table 4. Approaches/Methods to Capacity Building ­ 16 USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT Approaches/ Methods used by IPs Application Strengths Gaps and Local Partners Conducting Organizational Capacity Assessment To collect baseline data and monitoring progress All IPs perform organizational capacity assessment on local organizations. While some IPs conduct organizational capacity assessments by regular staff, others do it through experienced and external consultants. Some IPs provide feedback to their local counterparts and discuss the results of the assessment. Except for a select few, most local organizations don’t conduct organizational capacity assessment down to the lower level sub–sub grantees. Although some IPs use capacity assessment results for wider applications, such as identifying capacity gaps and addressing these through capacity building supports, some are conducting it to decide on the possibility of sub-granting local organizations. Capacity assessment in most cases is more “gap” based than “needs” based. Providing technical Assistance Provision of know how in the form of personnel, training, research, and associated costs Technical assistance is approached as a means for rolling out individual responsibilities. Technical assistance is often confused with supervision. When staff of IPs travel to the field and conduct on-the-spot checkups and discussions on certain activities, it is being seen as technical assistance. Technical assistance includes one-on-one Approaches/ Methods used by IPs Application Strengths Gaps and Local Partners mentoring or coaching, feedback on proposals, papers, or M&E plans, and guided application of skills while working on a task, such as joint problem solving or planning with a mentee. Mentorship and coaching are interchangeably used. However, there are many differences, mainly related to the intended outcome to a problem and the approaches taken to resolve that problem. Mentoring focuses on the individual/personal growth, career counseling, business, etc. Coaches focus on performance; there is a well￾defined goal based on the improvement of skills. Training Sharing knowledge and skills and cascading it down to the beneficiaries Training is basically the main activity for every program by both IPs and local partners, and is being provided as basic training and TOT Follow-ups on the outcomes of training provided is lacking in many instances. Absence of central databases for the people trained and types of training provided is a major constraint. Same staff from local organizations, and community and government offices take similar training by different providers. Assessment of training based on need is not commonly done by training organizers, (mostly IPs). Most training is geared toward complying with USAID regulations and achieving targets, rather than aiming at the overall development of the organization. The time allocated for TOTs and basic training is often too short, compromising the quality of the training. There was no evidence that post training evaluation is being done. Governmental stakeholders raised this as a USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT 17 Approaches/ Methods used by IPs Application Strengths Gaps and Local Partners big concern about training provided by most IPs and partners. Training organizers only evaluate during the training by pre- and post-tests to participants. There is mostly no follow-up on training . Supportive supervision Supervision provided Services for the compliance Controlling/auditing function for lower level and achievement of targets in in some organizations implementing a regular and timely manner staff/organizations through identifying and solving problems Infrastructural Support in purchasing Many IPs support critical areas In many cases, the development office machines, of infrastructural infrastructure of local equipments, vehicles, development. The purchase of organizations is limited and motor bikes, etc. computers and accessories, seen as a challenge in vehicles and office furniture addressing their growing has been done. needs, e.g., communication, reporting, and community￾level capacity building activities . Networking and Participation in All local organizations Several organizations actively partnerships umbrellas, networks, participate in one or more participate in networks and and consortia umbrellas (such as consortia, as well as Consortium of Christian Relief partnership forums. and Development Associations (CCRDA), networks such as National Network of HIV Positive People, and consortia such as Consortium of Reproductive Health Associations. Participation in networks, umbrellas, and consortia has helped local organizations to grow more while diversifying their knowledge, thereby reducing the duplication of efforts through better coordination, getting more funding, and sharing best practices. Besides the formal networks and partnerships, different forums are playing a great role in promoting knowledge, scaling up best practices, and improving programmatic integration. 18 USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT Approaches/ Methods used by IPs Application Strengths Gaps and Local Partners Development of Development, IPs usually produce and Development of training guidelines and manuals adaptation, translation, distribute facilitation materials, guidelines, and and reproduction of guidelines, training manuals, manuals that have been manuals and materials and other supportive materials produced by IPs or local in areas of HR, that are developed at the organizations are not HIV/AIDS prevention, global level and adapted to the necessarily in line with care and support. local context by themselves, national standards and or other IPs in the area, priorities. including government and local organizations. The MOH discovered that the quality of manuals, guidelines, Local organizations also adapt, and training materials is an contextualize, and reproduce area for improvement. different manuals, guidelines, and training materials for use by their staff, especially at the lower levels (e.g., for project In some cases, materials, manuals, and guidelines are not adapted to local needs. facilitators, peer educators, palliative care providers and other volunteers). Staffing supports Recruitment of staff IPs cover salaries and benefits IPs most often will only for local partners by of newly recruited staff and provide salaries for technical IPs cover the cost-share of some experts, and this has been key staff members. seen as a challenge for local Staff recruited and deployed to local organizations are supporting the local organizations to have staff for administrative- and finance￾related duties. organizations in many other IPs staffing support usually activities, such as proposal follows a project approach writing and resource rather than supporting the mobilization, in addition to organization in a wider single-project tasks to which context. they have been employed. Fellows’ assignment to Deployment of new PATH’s SCRHA, in However, due to their limited local organizations university graduates to partnership with the Ethiopian life experience, and lack of local organizations Society of Sociologists, and skills to change theory into Social Workers, and practice, it was noted that Anthropologists (ESSWA), support provided from the IP assigns newly graduated and ESSWA in this respect sociology and social work was insufficient. students from universities who have pre-service training. Community systems Development of IPs and local organizations strengthening informed, supportive have established multisectoral communities and community committees, such community‐based structures as HAC, PAC, WAC, and Children Support Committee. Committees include representation from the government sectors, the USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT 19 Approaches/ Methods used by IPs Application Strengths Gaps and Local Partners community, and other institutions in the area with the primary purpose of providing advisory and community mobilization assistance. Committees are conducting community mobilization, all at their levels, and beneficiary selection and problem solving at local levels, etc. Local partners provide sensitization, TOTs, as well as conduct regular review meetings and joint supervision. COMMON ORGANIZATIONAL CAPACITY STRENGTHS AND GAPS It was generally recognized that capacity building entails strengthening the ability of individuals and organizations to improve the performance of functions. The assessment, however, found that at the local levels, the focus of capacity building is perceived in different ways. For some, capacity building entails a means to comply with USAID regulations and includes technical training aimed at achieving project targets, therefore limiting capacity building to technical enhancement and skill-building at the individual level. Others recognize capacity building also as a means for organizational sustainability, thus including system development and improving the organizational culture by enhancing internal coordination; building skills and HR management to improve organizational performance; improving accountability; developing systems; planning; mentoring, and facilitation. In many instances capacity building is also perceived more broadly, working toward enabling environments through partnering and networking with other CSOs, government institutions, and the community. Most organizations recognized that “capacity building is a dynamic process of influencing and adapting to a continuously changing environment,”13 and that building capacity of local organizations is a step towards national ownership and sustainability. According to Pact, “capacity building is a means to an end, not an end in itself.” The most common capacity strengths and gaps among targeted CBOs in relation to the key areas of organizational development and the contributing factors for both strengths and gaps are here addressed. Contributing Factors for Strengths and Gaps Organizational Governance, Leadership, and Management It was found that all local organizations that were interviewed have bylaws, as required by government regulations for NGOs. In addition, internal policies, guidelines, strategic and operational plans are in place for the most part, or are being reviewed and elaborated. In several instances organizations mentioned the need for a clear purpose and strategy, especially those organizations that see their funding base is being threatened. 13 Formulated by World Learning. 20 USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT All local organizations included in the assessment have a board that is elected by their general assembly. Variation exists in how the boards function. Some have a hands-on-approach, putting themselves in a technical advisory role; others may play a more strategic role, scanning opportunities and conducting advocacy; and others just periodically meet without feeling much obliged to help the organization to improve its performance broaden its resource base, or advocate for specific causes. How an organization makes decisions is critical to its effectiveness, impact, and sustainability. Several local organizations mentioned that changes in board membership—changes that made these more appropriate and effective—contributed to increased transparency and accountability, which in turn minimized staff turnover. One organization mentioned that, “regardless of how strong the programs, how healthy the funding base, or how skilled the staffs are, if there is weak governance and an ineffective board, it will not be possible to establish a fully functioning organization.” Only some organizations included in this assessment were able to show the team that the board members are addressed through a defined capacity-building support (like OSSA). From the assessment, it appears that the contribution of the IPs in strengthening leadership and management has been limited, most often efforts made to this effect had already been undertaken by the local organizations themselves, often with technical assistance from other funding partners, especially where it concerned setting up boards and strategic planning. In FYLM (PATH affiliate), the assessment team also learned of a five-day leadership and management training that had been conducted. In general, however, the assessment team found that although there appeared to be improvement in terms of transparency and accountable management of human and financial resources within the local organizations, in several instances there was room for consensus building on management improvements, including active participation from all staff and sustained commitment from leaderships to transformation. Adopting a participatory process, whereby the members or constituents of an organization become committed to improving the functioning of their organization as they identify its needs and determine themselves how they can meet these needs, is by IPs and local partners alike considered an essential ingredient to institutional development. The assessment noted that this didn’t always work out on the ground. One of the challenges cited by Save the Children is “deeply rooted mistrust and un￾transparency” within the organizations. It is found that FBOs generally have a strong common vision and mission, which in several instances laid the foundation for establishing the organization and more democratic styles of management. At the same time, it was noted that some of these organizations are struggling in professionalizing their systems and management, as one organization mentioned, “bringing faith-based organizations into the developing world.” With a view to country ownership, technical assistance roles should be gradually shifted to local providers. Government partners expressed concern about costing and cost-effectiveness of technical training provided by IPs and local organizations, since approaches are most often not integrated into overall training plans of government, and capacity building is not always strategically targeting the government priorities identified by the 2010 roadmap toward a multisectoral response (building capacity of umbrella organizations, CSO networks, FBO networks), as designed by the MOH. On the other hand, the assessment found that local organizations working in community system strengthening are aligning their technical model of capacity building with national priorities. This is the case in strengthening local coordination bodies such as HACs and PACs, which are being trained as trainers and mentored in conducting their activities in the community. USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT 21 Financial and Grant Management It has been found that all IPs have put much effort in these particular areas of organizational capacity building, especially where it concerns compliance with USAID rules and regulations, with a view to meeting the objectives of the agreements. There is also evidence that program and finance staff received joint training to this effect, with a view to complement more specialized training and introduce accounting software to financial staff. Regular financial expenditure reporting and internal audit systems are usually in place. Some IPs provided technical assistance and coaching to local partners to revise their financial procedures and manuals. As a result of the more profound technical assistance that Geneva Global provides to selected implementers along the criteria of graduation during the project duration, their partners have strengthened their systems and have enhanced proficiency to manage PEPFAR/USAID funds, as well as grants from other donors. The assessment team found only few examples of flexibility in budget allocations for capacity building, whereby the local organization has its own budget for technical assistance and, thus, some autonomy in attracting support on a need basis (such as with World Learning). HR Management As a result of improved management, many organizations have updated their HR policy, regulations, staff benefits, and staff orientation. Most IPs provided technical assistance to strengthen HR management. Manuals, staff orientation booklets, and updated regulations were in place in most local organizations. However, many local organizations don’t have a comprehensive HR policy or a staff development strategy with HR master-training plans in place. In many organizations staff turnover remains a concern, especially at the senior management level. Although these turnovers may also be the result of organizational cultures, the functionality of the board, and the subsequent level of transparency, it should be noted that, in many instances, staff turnover is salary and benefit driven. In most FBOs, and Youth and PLWHA organizations, staff turnover is a lesser concern due to common motivation, shared vision, and larger commitment. To a certain extent many organizations remedy the potential loss of knowledge by training more than one individual in the same area and improving recruitment procedures. As one organization puts it, “every staff member should be replaceable.” Monitoring and Evaluation In a few instances, project cycle management training has been provided (for example by World Learning), where the local partner in Sebeta is considered to have lead to better understanding of the feedback loop of information, and therefore is able to handle setting up the organization’s management information system and to increase efficiency. The same organization conducts log frame analysis, using indicator-tracking tables and conducting yearly evaluations. Outside of these few cases, the assessment did not find this systematic approach being adopted elsewhere. Most IPs provided assistance in developing an M&E framework (in support of the overall operations), as part of a comprehensive M&E training. These frameworks include capacity-building activities, such as training and supervisory visits. However, measurement of interventions is mostly done against process and output indicators. It is recognized that indicators that can measure outcome and impact of capacity￾building interventions need to be included. Geneva Global observes that the final assessment at the close out of the NPI program will describe the changes in the organizations and provide feedback on the capacity-building assistance provided by Geneva Global. Across the board, supportive supervision, and quarterly and monthly review meetings are used to measure process indicators for the overall interventions. It was noted that supervision from the IPs often does more controlling than effective building capacity in local organizations. Joint review visits are 22 USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT carried out using monitoring checklists, but these mostly don’t focus on capacity building per se. Specific databases on capacity building are generally lacking, although the capacity to build a database is, in principle, there. Capacity for quality assurance at the lower levels is, to some extent, strengthened through technical assistance, standard setting, and guideline development, followed up by supportive supervision. To this end, most organizations use checklists to triangulate the information gathered from the reports. However, data quality assurance tends to be weak. This refers to integrity, methodological soundness, accuracy and reliability, serviceability, and accessibility of information. This is often compounded by the fact that data collection at regional and local levels performed by the different stakeholders is not sufficiently harmonized by HAPCO, and databases are often not shared, which increases the risks for double counting. Resource Mobilization and Infrastructure In light of dwindling external resources, resource mobilization—including having resource mobilization plans, writing proposals, etc.—becomes a crucial area for capacity building. Most national organizations participating in the assessment confirmed that they received technical assistance to that effect, although comprehensive exit strategies are mostly lacking, which in some cases led to gaps in service delivery, such as for OVC (WSG and OSSA, affiliates of Geneva Global, and Integrated Services for AIDS Prevention and Support Organization (ISAPSO), an affiliate of Pact). Bridging funds are generally lacking. Most of these organizations expressed a need for more systematic support in this area, particularly in developing proposals to compete for requests for applications/requests for proposals, and larger procurement. As one IP noted, “there is a lack of business orientation.” Infrastructural support, such as the purchase of computers and accessories, vehicles, office furniture, and other equipment has been mentioned by local partners as a basic area of capacity building that IPs have addressed. In many instances this support was considered very limited. Most organizations mentioned this as a major challenge to supervision, communication, reporting, and community-level capacity building, and thus as a threat to both achieving targets and the growth of the organization. Networking, Partnerships A network can be defined as an association of independent institutions with a shared purpose or goal, whose members contribute resources and participate in two-way exchanges or communications. The assessment noted that participating in networks created opportunities for local partners to grow (such as the PLWHA networks). There is evidence that IPs promote networking and partnering of local organizations; almost all local organizations reported that they participate in community and partnership networks. There are some innovative approaches, such as regional clustering of organizations working with the same IP (Geneva Global), whereby implementers working in the same regions of Ethiopia collaborate toward achieving a greater impact. This clustering facilitates partnerships, networking, and linkages. Such clustering aims at integrating HIV/AIDS efforts for strategic response, which minimizes duplication, and allows resource sharing and strong linkage to ensure sustainable and effective programs. The implementers are encouraged to systematically link with each other, with other non‐NPI implementers in the regions, and with government structures at all levels in the regions. Other examples include national partner quarterly meetings for program review and experience sharing. All local organizations participate in one or more umbrella organization(s), such as CCRDA, Consortium of Reproductive Health Associations, faith-based associations, networks of PLWHA, and women associations. GO/NGO forums at different levels, coordinated by the government, provide another opportunity for networking and knowledge sharing to minimize duplication of efforts, to promote programmatic integration, and to maximize sustainability. However, on several occasions the assessment USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT 23 team found that these efforts don’t fully prevent duplication, such as a targeted population receiving support from more than one organization. In another instance, the team found competition between the international IP and local partners over skilled staff. From the assessment, it appears that the HACs, PACs, WACs, and CSC are appreciative of the capacity building received from local organizations. An area for improvement is experience exchange among local coordinating bodies such as HACs, which could be facilitated through exposure visits. Knowledge Management Knowledge management is an important element of organizational development. The assessment teams found that knowledge management focus is on individual learning, dissemination, and the imparting of experiences and best practices. To a certain extent, team learning, whereby the sharing of lessons learned between individuals working in the organization and cross-functional learning (such as sharing lessons between finance and program staff) is taking place. Web sites are not utilized to promote experience sharing (such as communities of practice, Skype conversations, webinars), including those that share critical “how to do” knowledge. The “how to do” knowledge resides in the field, but it is widely dispersed and more needs to be done to make it available to a broad audience of local NGOs and CBOs. The regional clusters of Geneva Global NPI are platforms for a community of practice, where each participant comes to contribute/share traditional knowledge and tools as well as learn knowledge management. As NPI is currently effective in Addis Ababa, Amhara, and Oromia Regional States of Ethiopia, there are three regional clusters under PEPFAR/USAID/Geneva Global NPI. Other IPs conduct experience sharing at local and regional levels. Strategic Information and Documentation The goal of documentation is to capture info and share it with those who need it the most. Most organizations visited by the assessment teams had their electronic documentation in good order, and their hard copies in posters and leaflets. In several organizations, “mission” and “vision” were put up on the wall, visible to all staff, as was monthly information on service delivery. Apart from policies, plans, and guidelines, the team found good evidence of documented HR manuals and accounting software, as well as documentation, ledgers, and progress reports. Requests to underscore findings from the interviews were usually promptly followed up by documented evidence. Further, in reference to the new generation of indicators for PEPFAR II, overall reporting formats have been revised and staff trained in how to complete the forms. In most organizations, documenting good practices is in place. However, it was noted that awareness on and demand for quality data need to be increased. There was no evidence of more multi-dimensional analysis of information for decision-making, such as linking performance-related information with demographic change trends over time. Gender The team did not find much evidence of a systematic approach to integrating gender in capacity building approaches or on how gender specific data shape capacity building programming. There is a need to build capacity of organizations to systematically include gender considerations into their information systems that allows for measuring social, legal, health, and other indicators affecting the health of women, girls, and other at-risk populations, and for assessing their priority needs to inform design of projects and activities. Major Performance Gaps and Recommended Approaches to Bridging These Gaps Local organizations, beneficiaries, and IPs identified the following major performance gaps and listed what they perceived to be the most effective approaches to close these gaps. 24 USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT Governance and Management Performance Gaps  Many organizations still struggle with leadership and management issues, especially the local organizations. Decision-making needs to be more transparent, especially with a view to retaining staff.  Assistance to national partners remains mostly project-oriented, versus focusing on organizational development.  There is a need for effective exit strategies in order to avoid implementation gaps, which affect the existence of beneficiaries, such as OVC. Recommended Approaches for Closing Performance Gaps  More emphasis should be placed on improved governance. Members of the board need to be strategically elected, and capacitated to provide adequate oversight and support at the higher policy­ /decision-making level, and to conduct advocacy and resource mobilization. Senior management should be capacitated to “own” capacity building and drive it down through the organization, while local organizations’ aspirations and strategy should be reset to reach their full potential.  A deliberate proactive approach to capacity building and assessing capacity-building needs in relation to the entire organization, versus the need for a specific project component, should be adopted. A capacity-building framework should be expanded (building on the existing ones if any). This should have indicators in the following categories: partnerships; organizational policy/systems; organizational/institutional (board, strategic plan, effective leadership/management, financial management, HR management, monitoring and evaluation plans; knowledge management; service guidelines); individual/workforce (volunteers); and performance (toward graduation). The selection of capacity-building indicators should be guided by the understanding that organizational development interventions and all related actions are geared at sustainability of the organization.  IPs should provide more support in designing exit strategies at the initial planning stages. Financial and Grant Management Performance Gaps  In most instances, the capacity-building budgets are managed directly by the IP. This is mostly for training and technical assistance. This situation doesn’t contribute to greater ownership on the part of the local partners in recruiting technical assistance when the need arises. Recommended Approaches for Closing Performance Gaps  Create more capacity in local NGOs to manage their own capacity-building budgets and ensure more flexibility in the decentralization of capacity-building budgets HR Management Performance Gaps  A major challenge is to retain skilled staff Recommended Approaches for Closing Performance Gaps  Make changes to the organizational culture in a way that builds positively on a shared commitment of staff and volunteers to the mission, and strikes a balance between core values, beliefs, behavior norms, and the organization’s performance orientation  Develop HR policy/strategy and master training plans, linked to an incentive system Monitoring and Evaluation Performance Gaps  Limited understanding of the feedback loop of information and evidence-based decision-making USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT 25  Lack of rigorous evaluation and measurement of the effectiveness of programs and gains in social impact  Supportive supervision by the IPs vis-à-vis controlling/auditing  Systematic measuring gender concerns Recommended Approaches for Closing Performance Gaps  Build capacity in project cycle management, log frame analysis, and evaluation  Develop indicators to measure effect and impact of capacity building interventions  Strengthen capacity in data demand and use  Build capacity in local organizations to systematically include gender considerations into information systems Resource Mobilization and Infrastructure Performance Gaps  Lack of exit strategies, which poses the risk of serious disruptions, especially when several grants close out at the same time, such as with OSSA  Lack of vision/strategic support from board and senior management in this regard  Insufficient infrastructure support, especially affecting supervision and communication Recommended Approaches for Closing Performance Gaps  Provide structural and timely support to senior management and board in scanning new avenues, advocacy, and proposal writing  Conduct periodic stakeholder analysis and risk analysis  Develop a strategy to package best practices and market the organization  Ensure sufficient allocation for infrastructure Networking, Partnerships Performance Gaps  Need for mutual commitment to capacity building, along with shared expectations and accountability  Lack of demonstrated, measured increased leadership by local and national partners  Need to diversify and scale up local knowledge Recommended Approaches for Closing Performance Gaps  Explicitly recognize partnership efforts supporting capacity building and monitor these for their effectiveness and duplicability (such as regional clustering and umbrella networks)  Introduce incremental planning for a shift in role of IPs’ engagement to less involvement over time  Build on the lessons learned from community system strengthening (such as working with HACs and PACs) to strengthen partnerships with the non-health sectors when broadening social service packages  Strengthen partnerships with HAPCO and provincial and local government institutions in geographic mapping of services, stakeholders, target populations and dynamics of the epidemic, and in working toward alignment with national priorities Knowledge Management Performance Gaps  There is lack of other channels, in addition to the traditional ones (partnership meetings, good practices documentation) for experience sharing. 26 USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT Recommended Approaches for Closing Performance Gaps  Strengthen knowledge management by offering critical “how to do” knowledge versus “what to do” to a broad audience of NGOs and government organizations, e.g., offering exchange visits among local organizations  Introduce innovative ways for accessing and sharing information, such as webinars, through mobile phones, etc. System Documentation Performance Gaps  Dataflow and dependencies between individual systems are insufficiently being captured. Only in some instances were management dashboards found. Recommended Approaches for Closing Performance Gaps  The more rigorous use of management dashboards could address this gap. Main Challenges and Good Practices for Working with National Organizations This section is highlighting major challenges and good practices as perceived by the IPs and national stakeholders. Challenges: 1. Organizational development is still insufficiently recognized as a route to sustainability. Bringing together organizational development with technical capacity building often appears challenging for local organizations. Several attempts have been undertaken to strengthen organizational development in the local partners; however, in many instances the process has not been fully internalized. Regarding country ownership and sustainability, bringing the capacity of local partners to a level at which they are capable of mobilizing, designing, implementing, and monitoring development priorities, and effectively partnering with other stakeholders may pose a challenge. Major reservations heard by government stakeholders are: disintegrated approaches, different tools and standards of capacity building/training interventions; most of the trainings are project-based and donor-driven, and cease when the project closes; integration and harmonization of capacity-building efforts toward implementation of the “three ones” for HIV/AIDS. Quality assurance and demand for quality data tend to be a challenge in local organizations. Another challenge is building effective alliances in creating a critical mass at the community level for improving the quality of life of PLWHA and OVC, as well as their access to government services. In some instances sub granting to smaller CBOs appears challenging. Systematic approaches toward maturation and graduation are mostly not applied. Good Practices: 1. In many instances a culture shift is taking place through an increasing recognition of the critical linkage between organizational development and sustainability. Good practices of organizational development are found in organizations with an effective board, clear directions, needs based capacity-building plans, and a degree of financial flexibility to address these needs when necessary, such as with the World Learning partners. A crucial step toward generating more ownership and sustainability is building capacity within local organizations to support community system strengthening. In several instances there is good collaboration with government at the local levels through coordinating bodies, such as HACs, PACs, or WACs, which have multisectoral representation and interface with the community. It was found that most of the local partner organizations use collaborative approaches to transfer knowledge and USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT 27 skills to improve the quality and capacity of these community actors, including organizational strengthening; monitoring and supervision; and linkages, partnerships, and engagement with other actors in planning, coordination, and service delivery that are essential for effective and sustainable community responses. Most local partners provide supportive supervision and apply triangulation of reported data, which may contribute to data quality assurance. Good practices are exemplified by NGOs and CBOs adding new services and support for their constituencies, and for building a larger and more effective “voice” at different levels. Local organizations benefitted from partnering in a larger network or being affiliated with umbrella organizations. The more established organizations demonstrate well-elaborated financial management mechanisms, which are developed in longer-term agreements either with the IPs or through other funding agencies. These mechanisms enable them to channel funds to CBOs through small grants that require relatively simple proposals and accounting. Geneva Global is working with selected organizations toward graduation. In this respect, a series of yearly assessments on capacity and performance is being conducted. The total score collected from all capacity and performance assessment criteria will determine where organizations are on the road to graduation. The grading system is categorized by “excelling” (total score 95–100%), “performing” (total score 85–95%) and “growing” (total score 75–85%). In line with the assessment follow-ups, technical assistance is being provided to focus on those areas that need further improvement. CONCLUSION With advancements in surveillance and treatment, HIV/AIDS has been transformed from an acute health problem to a long-term global development issue. Intervention strategies now require a long-term multisectoral approach. In Ethiopia, the assessment team found that IPs have moved to a more systems￾based approach, in that they have replaced emergency-type interventions for HIV/AIDS with more comprehensive interventions that involve communities, local organizations, and networks in planning, implementation, and monitoring. To ensure success with this more complex community-wide strategy, the IPs studied in this assessment are employing practices to build capacity in their local partners at the individual/workforce level, at the organizational level, and at the system/institutional level, some including more comprehensive approaches than others. To Scale Up Capacity-building Strategies 1. Broadening the use of assessment tools and assuring the use of capacity-building action plans As previously noted, the different assessment tools and practices being used by IPs are evidence￾based and offer the potential to assist local organizations and capacity building providers in understanding and using the principles and practices of organizational capacity building. They allow for developing a process of designing, implementing, measuring, and documenting capacity-building interventions that are geared toward improving organizational performance and the skills and competencies of staff. The assessment concludes that efforts to build technical and managerial capacity at the workforce level are generally paying off, and have resulted in increased individual skills and competencies. These assessment tools are useful and should continue to be linked to action plans for improvements. 2. Using frameworks and assessment tools for on-going evaluation of capacity-building efforts 28 USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT Organizational assessments typically lead to the development of capacity-building strategies throughout each organization, which should be ongoing. These should include regularly scheduled assessments and evaluations, as well as training, new policy development, enhanced systems, and others. The conceptual framework in the Technical Brief (January 2011) of AIDSTAR-Two, and PEPFAR Capacity Building Framework (August 2011) describes how to build the foundation for stronger, more sustainable HIV/AIDS programs, organizations, and networks. Adopting such framework could help to better standardize capacity-building interventions and measure improvements. The organizational capacity-building framework presented by AIDSTAR-Two is organized around four components: organizational functions, organizational practices, standards, and indicators. The organizational functions (a set of core organizational features, which need to be present and functioning together effectively in a sustainable institution: mission, vision, and strategy; structure; management systems; partnerships, external relations, and networking; and leadership and governance). The organizational capacity-building framework used by Geneva Global is a step in the right direction in helping local organizations “graduate” and thereby mature to a level where they have adequate capacity and systems in place to directly access funding from external sources. 3. Expanding successful training programs The assessment team was informed of several beneficial training programs, including the training of community coordinating mechanisms such as HACs, PACs, etc. Some local organizations have lengthened the duration of their training courses, which allows for the building in of practicum. Programs to build leadership, management, and governance competencies should be provided to organizational or community teams to broaden their long-term value, particularly given the high turnover rates in local organizations and complex nature of the skill sets needed. 4. Increasing systems level capacity through partnerships and networking The assessment team found several good practices of systems-level support, such as strengthening community systems and collaboration with government stakeholders at the district and provincial level, and demand-generation approaches. Capacity building at this level involves multiple interventions across different sites and requires an alignment of different resources and activities. Successful system-level interventions require agreement by all parties to share information and to act in cooperative ways. Therefore organizations need to have clearly defined strategies and annual operational plans for policy engagement and advocacy with relevant stakeholders, based on the national policy context and the needs of civil society. In addition, IPs should invest in a wide range of networking and partnering work to improve mechanisms for sharing technical expertise and experiences. Approaches such as regional clustering, applied by Geneva Global, could be adopted by other organizations where relevant. 5. Improving HR management The assessment found HR management systems in place in most organizations, backed by HR policies and manuals. Capacity building in this area may include systematic documented processes supported by regular training and coaching for staff. More systematic performance planning and monitoring, as well as a systematic application of work groups in organizing the work and being held accountable for results may be a strategy worth stepping up. KEY RECOMMENDATIONS This section includes key recommendations for the capacity-building efforts (which should be continued as this work moves forward in Ethiopia) and additional strategies that need to be strengthened or considered. USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT 29 1. Promote the value of capacity building by linking funding requirements to plans, actions, and goals. This assessment found many different levels of ownership and leadership in capacity building and varying levels of demand-driven technical assistance. IPs should tie expectations for comprehensive capacity building plans to funding. They should also provide assistance in building the capacity of organizational and health system leaders, and make sure these leaders understand how to develop such plans, how to implement and monitor specific capacity-building activities, and how to evaluate and report progress. One of the underlying causes of the variance seen in how organizational leaders view capacity￾building efforts may be the lack of flexibility in budgeting for capacity building by the local organizations themselves; in most instances, it is the IP that releases the funding based on the work plans. Another important reason may be the way in which capacity-building assistance is offered and facilitated at the initial stages of capacity-building assessments and planning. As previously noted, organizational development depends on the commitment of the organization and its leadership, a good understanding of organizational development, and the involvement of all levels of the organization. Assessment at the field level did not provide sufficient evidence that these guiding principles for organizational development were always applied. More emphasis is needed on assisting organizations in understanding the synergistic relationships between the levels of capacity building and developing the primary functions of leadership and governance as part of organizational development. Leadership and governance are especially critical since they bind and promote the effectiveness of all the other functions. 2. Build in-country capacity to provide technical assistance in organizational development/capacity building Organizations need assistance in obtaining an operational understanding of capacity building and organizational development, including its principles, concepts, and practices. In several instances, the initial assessments were done with the assistance of an external expert, and the actual organizational development was followed up by the IP. In several local organizations leadership changed and the organizations that benefitted from the initial technical assistance were back to square one. To build more sustainable capacity, USAID may consider outside technical assistance for organizational capacity building to help develop in-country talent for organizational development work. The in￾country talent can work with local IPs, while being mentored by the outside technical assistance. The in-country talent could include a university (business school) or other in-country organizational development consultants. 3. Develop key indicators for capacity-building success As the strategies to strengthen capacities become more complex, the challenge of measuring effective capacity-building efforts intensifies. Another difficulty is that current quantitative M&E systems used by many organizations to measure outcomes of behavior-change interventions do not sufficiently capture the depth and breadth of organizational capacity-building activities. The capacity￾building framework also includes indicators to measure progress in meeting the standards for essential organizational practice. Adopting a systematic approach and metrics that assess efficiency, effectiveness, and impact would allow for measuring intermediate progress and longer-term results. In evaluating capacity-building efforts, it is critical to determine if the project made a difference. For capacity-building evaluation, it is important to determine not only if objectives were met, but how they were met, and how well. The standards of practice of organizational development should be confirmed and applied across the USAID/PEPFAR portfolio. Improved practice could include building an improved monitoring system to measure change or impact of organizational capacity building and information sharing on the “how” versus the “what” of capacity building. 30 USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT Moreover, more time and resources should be invested in baseline and follow-up research for capacity building, particularly in projects that are expected to demonstrate organizational behavioral change. Greater collaboration among IPs using compatible tools will allow to them generate a database of performance, and help to better understand what drives NGO effectiveness. Also needed is more systematic investment in establishing and maintaining a quality assurance system. In many instances, guidelines, protocols, and standard operating procedures exist; however, these could be more systematically backed up by an ongoing evidence-driven system for assessing and improving the quality of services to be measured by, for example, a written question and answer plan, periodic surveys to establish client satisfaction, or an adequate supervisory system. 4. Adopt more innovative approaches to capacity building and organizational development Areas of innovation to be expanded are geographic information systems, knowledge management, and data quality assurance. For enhanced knowledge management, a broader definition needs to be provided—a definition that encourages field staff to reach out to others working on similar issues and to share lessons on “how-to-do” through the traditional channels, as well as through widening access to the Internet, which would allow for using inexpensive social media tools such as Skype and webinars, in addition to the current practices of experience sharing through review meetings and regional cluster program review forums. 5. Strengthen community systems In focusing on community system strengthening and strengthening the interface with health and social systems, the program should invest more in expanding volunteer programs (such as linking up with community health workers) toward prevention, care, and support, and continued resource mobilization. 6. Include gender concerns in capacity building The assessment found little attention to gender concerns in the capacity-building approaches. Strategies for building capacity for mainstreaming gender in an organization’s overall strategy should be more prominently on the agenda when assessing capacity building and when developing capacity￾building plans. 7. Direct funding of local organizations USAID may consider direct funding through FOG and APS mechanisms for the more established organizations that demonstrated well-elaborated financial management systems, which enable them to further channel funds to CBOs through small grants that require relatively simple proposals and accounting. USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT 31 32 USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT APPENDIX A. SCOPE OF WORK Global Health Technical Assistance Project GH Tech Contract No. GHS-I-00-05-00005-00 Statement of Work USAID/Ethiopia: Implementing Partners’ Organizational Capacity Assessment (Revised:08-26-11) I. TITLE Activity: USAID/Ethiopia: Implementing Partners’ Organizational Capacity Assessment Contract: Global Health Technical Assistance Project (GH Tech), Task Order No. 01 II. PERFORMANCE PERIOD Evaluation should begin in late August 2011 depending on the availability of the selected consultants. Final revised draft report available by mid-November, tbd. III. FUNDING SOURCE USAID/Ethiopia IV. OBJECTIVES AND PURPOSE OF THE ASSINGMENT V. BACKGROUND Since 2004, USAID-Ethiopia has, in collaboration with the Government of Ethiopia (GoE), provided support through the Presidential Emergency Plan for AIDS Relief (PEPFAR) to strengthen organizations providing HIV/AIDS prevention, and non-clinical care and support services. These include support for orphans and other vulnerable children (OVCs), prevention among most at risk populations (MARPs), HIV counseling and testing, condom distribution, behavior change communication, prevention with positives (PwP), palliative care and support for persons living with HIV/AIDS (PLHIV) and support for income generating/economic strengthening activities. In addition to sub-granting to local organizations to provide these services and improving their technical capacity to improve the quality of service delivery, USAID, through its implementing partners (IPs) provided technical assistance to improve the organizational capacity of over 600 national and local civil society organizations (CSOs), non governmental organizations (NGOs), faith-based organizations (FBOs) and community-based organizations (CBOs). With a new five-year Health Sector Development Program IV (2010–2014), the country’s Growth and Transformation Plan (2011–2015) and the US Government (USG) Global Health Initiative (GHI) in place, USAID-Ethiopia will build upon the successes and lessons learned of its institutional strengthening interventions to date. Capacity building of health facilities and local organizations to improve the non￾clinical services they provide is an integral part of the health system strengthening approach under PEPFAR and the Global Health Initiative. It is critical to ensuring country ownership and sustainability of PEPFAR programs. ­ USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT 33 USAID is proposing an assessment of the organizational capacity interventions of select PEPFAR/USAID partners, the conclusion of which will inform programmatic decisions regarding future institutional capacity building approaches and activities. This statement of work (SOW) describes the purpose and the proposed approach/assessment methodology, timeline, and deliverables of the assessment. VI. SCOPE OF WORK Purpose of the Assessment The main objective of the assessment is to determine strengths, weakness and best practices in the local institutional capacity building approaches of select USAID implementing partners (IPs) and to provide a set of practical recommendations to the Mission for its future institutional capacity building strategy and activities. This will ensure that USAID supports evidence-based and proven effective programs. Objectives of the assessment: 1. Describe the various approaches, models and tools used by partners to design, implement, monitor and evaluate local capacity building activities. Assess whether these approaches, tools, etc. align with published or documented best practices and recommendations on local capacity building. Identify strengths, weaknesses, challenges, and best practices with respect to local capacity building interventions. Identify innovative capacity building interventions that can be reinforced and/or replicated in future programs and make recommendations on how USAID can standardize support to and M&E systems of IPs working on capacity building of local organizations. Specific evaluation questions to address in the assessment: 1. What methods and tools were used by the implementing partners to assess the needs of local organizations, design appropriate capacity building interventions based on identified gaps, implement the intervention, monitor and evaluate improvements in capacity? a. What key areas of capacity building/organizational development are being targeted by the IPs? b. What are the common features and main differences among partners’ approaches to capacity building? c. Are partners using tools and approaches that have been previously tested or based on standardized/evidenced based practices? d. How do IPs measure, monitor, and evaluate improvements in capacity over the long term to assure the sustainability of their interventions? Are there approaches to provide CQI, address staff turnover and diffusion of skills transfer and learning? What are the most common organizational capacity strengths and gaps identified in terms of human, institutional, material, financial, and technical and project management among targeted community based organizations? a. What do beneficiaries, IPs and other stakeholders perceive are the contributing factors for both strengths and gaps? b. What do beneficiaries and IPs perceive are the most effective methods/approaches for closing the performance gaps for each capacity improvement area? c. What are the main challenges and best practices for working with local organizations? What approaches to capacity building and organizational development should USAID reinforce or scale up in the future? a. Are there interventions/approaches to capacity building that should be discontinued? If so, why? 34 USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT b. Should USAID consider having a separate entity that provides technical assistance on organizational capacity building to all IPs in HAPN? If so, how should this be designed? c. Should USAID consider having capacity building integrated or mainstreamed among IPs? If so, what should USAID do to better standardize interventions and indicators/metrics to measure improvement among the IPs? What are specific recommendations for programming to include in the PEPFAR COP 12? d. Is there a tested and/or recommended M&E system for the organizational capacity building interventions? If yes, what are they? If no, what do you recommend? VII. METHODOLOGY Six implementing partners will be included in the assessment. These will include PACT, Geneva Global, World Learning, PATH and Save the Children that work on HIV/AIDS programs. These partners both sub-grant to local organizations to provide HIV/AIDS prevention, and non-clinical care and support services and provide assistance to strengthen the technical and organizational capacity of local organizations. The assessment will also include one partner that implements USAID’s reproductive, maternal, neonatal and child health-related activities that also undertake local organization capacity building (Pathfinder’s Integrated Family Health Program). The assessment will be carried out by a team of two independent, external consultants and one USAID/Washington Capacity Building technical advisor over six weeks. The assessment will use a mix of qualitative and quantitative methods such as key informant interviews, surveys, field observation/site visits, and a review of the implementing partners’ tools, reports and other materials. The specific methodology and draft tools will be developed by the evaluation team prior to in-country travel with input from USAID/E. The workplan will be approved by USAID prior to the start of field work. Two to three USAID/E staff will join the assessment team during the site visits. Implementing Partners will accompany the team on site visits as appropriate, but will not be present during interviews with the sub-grantees, stakeholders or beneficiaries. USAID/E staff and representatives from implementing partners will be responsible for arranging their own travel, logistics and other arrangements, as well as financial responsibility for their participation. Team Planning Meeting The assignment work will commence with a two-day Team Planning Meeting (TPM). This meeting will allow the team to meet with the USAID/E staff to be briefed on the assignment. It will also allow USAID to present the team with the purpose, expectations, and agenda of the assignment. In addition, the team will:  Clarify team members’ roles and responsibilities,  Review and develop final assessment questions,  Review and finalize the assignment timeline and share with USAID,  Finalize data collection methods, instruments, tools, guidelines and analysis plan,  Review and clarify any logistical and administrative procedures for the assignment,  Establish a team atmosphere, share individual working styles, and agree on procedures for resolving differences of opinion,  Develop a preliminary draft outline of the team’s report, and  Assign drafting responsibilities for the final report. USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT 35 Key Informant Interviews: Key informants to be interviewed include:  USAID Mission staff/focal persons including those working on health systems strengthening and capacity development.  USAID implementing partners— PACT, Geneva Global, World Learning, Save the Children, Pathfinder, and PATH staff and respective USAID project AOTRs/COTRs  Staff of selected 2-3 local organizations supported by each of the six IPs (actual number to sample is TBD).  Government of Ethiopia representatives including Federal and Regional HIV/AIDS Prevention and Control Organization (HAPCO) office staff, Federal Ministry of Health and Regional Health Bureaus, Kebele leaders and Woreda officials  Other PEPFAR partners that work with/through local organizations as necessary Focus Group Discussions: Focus group discussions can be used for interviewing local organization staff including management and administrative staff and those who have participated in capacity building activities (e.g., training and workshops) supported by the four IPs. Local sub-grantees of the IPs will be selected for the assessment on purpose taking samples of those that have low, medium and high performances after the capacity building interventions by the respective IPs. USAID/E will provide a detailed contact list of key informants and focus group participants to the consultants during the document review period so that appointments and interviews can be set up for the team’s arrival in-country. USAID/E will also provide a draft schedule for field visits including duration of stay at various sites to inform the team’s time in-country. Document/Material Review: USAID/E will provide consultants with the following background documents in preparation of the assignment.  IPs Cooperative Agreements  IPs PEPFAR Semi-Annual Reports  IPs Annual Reports  IPs Quarterly Reports  GOE HSDP IV  FHAPCO SPMII  GHI related documents  USAID trip reports summarizing past field visits to IPs  GOE Road Map for HIV/AIDS Prevention, Care and Treatment  Organizational capacity assessment and capacity building tools and materials used by the IPs(e.g. OCAT, PIAI---) Other methods may be used based on input from evaluation team. VIII. TEAM COMPOSITION AND PARTICIPATION USAID seeks three consultants – a Team Leader with experience assessing/evaluating USAID health system strengthening/policy and capacity development programs, a capacity building specialist with 36 USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT experience working with local Ethiopian organizations and an expert in USAID programming (PEPFAR/HIV AIDS highly desired) and a logistics coordinator. The USAID expert may be selected from USAID/Washington Capacity Building team. The Team Leader will be an international consultant who will agree to fulfill his/her responsibilities in six weeks, spending up to four weeks in-country, and will play a central role in guiding the assessment process. The consultant will hold conference calls with the other team member and USAID/E representatives before and after the visit to Ethiopia, and produce a draft followed by a final report for USAID/E. The consultant also presents findings of the assessment to USAID and IP staff as well as key stakeholders. The team leader should have previous experience leading evaluation teams. The Capacity Development Specialist will ideally be a local consultant with extensive human and institutional capacity development assessment, implementation and evaluation experience. Knowledge of technical capacity assessment with local organizations is essential. The consultant will be responsible for the field assessment, writing some sections of the report and providing overall assistance to the Team Leader. Ability to speak local language(s) and Ethiopian experience highly desired. The Logistics Coordinator will have experience managing complex evaluations within the development sector, such as coordinating business travel, field visits, and meetings. He/She will be responsible for developing the final schedule with USAID, the evaluators and IPs, making lodging and transportation arrangements, and scheduling meetings. A USAID representative will also participate on the team as the USAID/PEPFAR programming expert and will have in-depth knowledge of HIV AIDS programs under PEPFAR and capacity building best practices and models, USAID program requirements and experience working in USAID offices. The PEPFAR programming expert will be responsible for assisting the team lead as necessary, writing sections of the report and strategizing possible future program options for USAID/Ethiopia. Assessors’ Selection Criteria for Team Members (Maximum 100%) distributed as follows: Education (25%): An advanced degree in any of the social sciences with specific experience in Health System Strengthening/and Capacity Development with recognized organization for a minimum of seven years. (Master’s and above). Work Experience (35%): Minimum 6-10 years of progressively responsible experience in the design, implementation and evaluation of Health System Strengthening or local capacity development programs with demonstrated technical expertise and skills in public health and HIV/AIDS. Skills and Abilities (40%): Demonstration of strong analytical, managerial and writing skills are very critical for the assessment work. Exceptional leadership in coordinating, assigning the team with the appropriate responsibilities, communication, and interpersonal skills is absolutely critical. In addition, the team member must be able to interact effectively with a broad range of internal and external partners, including USAID/PEPFAR implementing partners, host country government officials, local organizations and beneficiaries. Must be fluent in English and have proven ability to communicate clearly, concisely and effectively both orally and in writing. Must be able to produce a quality document that can give direction and facilitate improvement to the local organizations and CSO networks organizational capacity development programs in Ethiopia IX. SCHEDULE AND LOGISTICS A six-week assessment is proposed with four weeks spent in-country and a desired start date in early Sept 2011 (in-country work to begin Sept 13). The consultants, working with the six IPs (with logistics support from the GH Tech logistics coordinator and USAID administrative and evaluation staff), will USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT 37 arrange all of the partner meetings and site visits. Associated travel and per diem costs will be covered under the contract for services. USAID will handle the logistics for in-country travel and meetings and the scheduling of the in-house debriefings and any trainings that occur as a result of the findings. Below is a list of the specific tasks to be accomplished by the consultant team, with an estimated level of effort and proposed timing for each task. Table 5. Level of Effort Activity Duration/LOE per consultant Proposed Dates (illustrative) Mission sends background documents August 26 Review background documents, draft work plan methodology and survey instruments. Hold planning call with GH Tech, evaluation lead and USAID/E and logistics coordinator (TBD). Team prepares draft work plan and fieldwork schedule. 5 days Aug 29-Sept 2 Sept 6 Travel to Ethiopia 2 days (depending on Point of origin) Sept 11-12 Participate in team planning meeting; team and USAID complete work plan and fieldwork schedule 2 days Sept 13-14 Conduct interviews, focus groups and site visits in Addis and regions 13 days Sept 15-27 Write draft report in-country and prepare for and conduct debriefings/trainings for USAID and IPs 5 days Sept 28-Oct 3 Presentation of preliminary findings and recommendations to USAID and IPs, sharing best practices and training on capacity building. Submit draft report. 2 days Oct 4-5 Depart Ethiopia 2 days Oct 5/6 Mission reviews and submits comments on draft report (10 working days/no LOE for consultants) Oct 18 Finalize Report—incorporate Mission comments 3 days Oct19-21 Total LOE 34 days/Team Leader 30 days/Team Member 20 days/Logistics Coordinator A six-day work week is approved while in-country. Note that there are two Ethiopian holidays Sept 12 (Ethiopian New Year) and (Meskal) during this time period. The US Embassy/USAID and most IP offices and all GOE offices will be closed on these days. A detailed meeting and site visit schedule will be developed by USAID and the logistics coordinator. 38 USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT X. DELIVERABLES  A methodology to address the evaluation questions including survey tools, interview guides and analysis plan prior to conducting fieldwork. The work plan and fieldwork schedule will be completed during the TPM and approved by USAID prior to initiation of any site visits or key informant interviews.  Two separate debriefing presentations will be made to USAID and IP staff prior to departure, sharing preliminary findings and recommendations, best practices on capacity building and a training on how to strengthen current approaches to capacity building as deemed necessary. Copies of the power point slides and any materials used during the debriefings will be left with USAID before departure.  Draft report: Analyzing data collected from interviews and field visits, the assessment team will provide USAID/E with a draft report prior to departure. USAID/E will provide one comprehensive set of written comments on the draft report to the team leader within 10 working days of receiving the report.  Final report: The evaluation team is then required to submit a final report within 5 working days after USAID provides its written feedback on the draft report. The report should be no longer than 30 pages (excluding annexes) and include an executive summary and key findings for each of the evaluation questions and recommendations. Copies of all survey tools should be included in the annexes. The report should be submitted electronically to USAID. NOTE: Due to potential procurement sensitive information in the report, any procurement sensitive material will be removed from the final report and compiled into an Internal USAID Memo that will not be published or circulated outside the Mission. Once the Mission approves the final unedited report, GH Tech will have the document edited and formatted and made 508 compliant consistent with USAID branding guidelines, and will provide the final report to the mission for distribution (5 hard copies and CD ROM). It will take approximately 30 days for GH Tech to edit/format and print the final document. This will be a public document and posted on USAID/DEC and the GH Tech website. XI. RELATIONSHIPS AND RESPONSIBILITIES GH Tech will coordinate and manage the evaluation team and will undertake the following specific responsibilities throughout the assignment:  Recruit and hire the evaluation team  Make logistical arrangements for the consultants, including travel and transportation, country travel clearance, lodging, and communications. The USAID/E will provide overall technical leadership and direction for the Evaluation Team throughout the assignment and will undertake the following specific roles and responsibilities: Before In-Country Work  Respond to any queries about the SOW and/or the assignment at large.  Consultant Conflict of Interest. To avoid conflicts of interest or the appearance of a COI, review previous employers listed on the CV’s for proposed consultants and provide additional information regarding potential COI with the project contractors or NGOs evaluated/assessed and information regarding their affiliates.  Documents. Identify and prioritize background materials for the consultants and provide them, preferably in electronic form. USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT 39  Site Visit Preparations. Provide a list of site visit locations, key contacts, and suggested length of visit for use in planning in-country travel and accurate estimation of country travel line items costs.  Lodgings and Travel. Provide guidance to the Logistics Coordinator and identify a USAID/E administrative assistant to work with the LC. During In-Country Work  Mission Point of Contact. Throughout the in-country work, ensure constant availability of the Point of Contact person and provide technical leadership and direction for the team’s work.  Meeting Space. Provide guidance on the team’s selection of a meeting space for interviews and/or focus group discussions (i.e. USAID space if available, or other known office/hotel meeting space).  Meeting Arrangements. USAID administrative staff will arrange meetings in house and work with the LC to arrange meetings with external stakeholders and site visit logistics.  Facilitate Contact with Implementing Partners. Introduce the Evaluation Team to implementing partners and other stakeholders, and where applicable and appropriate prepare and send out an introduction letter for team’s arrival and/or anticipated meetings. After In-Country Work  Timely Reviews. Provide timely review of draft/final reports and approval of the deliverables XII. MISSION CONTACT PERSONS Petros Faltamo Health Systems Strengthening Advisor Kristin Saarlas Evaluation Coordinator COST ESTIMATE A cost estimate will be provided by GH Tech. 40 USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT APPENDIX B. PERSONS CONTACTED LIST OF PEOPLE CONTACTED AS KEY INFORMANTS AT IP LEVEL USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT 41 Name of the Organization Name of Person Contacted Position in the Organization Save the Children USA TransACTION project Yosef Burka Chief of party Yonas Mekonnen Capacity building advisor Asayehegn Tekeste Capacity building manager Aynalem Abrha Grants officer Geneva Global/NPI Yilma Woldeyohannes Country representative and NPI-ETH program manager Melkie Tilahun Monitoring and evaluation coordinator & health program adviser Meron Kidane Capacity building & operations coordinator Sehale Fantahun Monitoring and evaluation analyst Fetlework Tegenie Financial analyst Daniel Melese Program coordinator Anteneh Bizuayehu Program coordinator Yohannes Amado Program coordinator Pathfinder International IFHP Dr. Mengistu Asnake, IFHP chief of party Metiku Giorgis Operations manager Dr. Kidest Lulu DTD for RHIFP Girma Seifu Finance, admin & logistic team leader Mehari Belachew Monitoring and evaluation officer Alemtsehay Mamo Grants manager PACT Tamiru Lega Director, CD Yemane Kejela HR and admin senior manager Lesly Mitchel Country representative Wubareg G/Kintos CD program officer Bezawit Bekele CD manager PATH/SCRHA Abenet Leykun, Chief of party Amelework Haileselasie Monitoring and evaluation specialist Ahmed Yusuf Progress monitoring assistant Dr. Deme Ergetie PC advisor Mitiku Lamma Economic strengthening specialist Tariku Teka ES advisor Tisae Mekonnen, Compliance evaluation advisor Name of the Organization Name of Person Contacted Position in the Organization Mulumebet Jemberu Grants and operations head Selam Girma Grants specialist Addisu Getahun Grants specialist Yemisrach Gezahegn, Palliative care specialist Asalif Demissie Community empowerment specialist World Learning/GSM Adele Djekoundade Project director/GSM Claude John Country director Simachew Yigzaw Capacity building advisor Abrha G/Tsadik Senior grants and compliance manager Gemechis Teferi Monitoring and evaluation specialist PEOPLE CONTACTED AS KEY INFORMANTS AT LOCAL PARTNERS LEVEL Name of the Organization Name of Person Contacted Position in the Organization Alem Children’s Support Organization, Bahir Dar Nor-ahun Bayeh Project coordinator Abrham Mulu Livelihood officer Etenat Dagnaw Finance and admin Biftu, Ambo Taddese meskela Board head Alemayehu Adella PAC chairperson Meseret Derso Accountant Dereje Araga Generalist Getu Abera Manager Roman Diriba Fellow Yewoinshet Masresha NIP executive director Biniam Mesfin NIP project supervisor Birhanu Desta NIP monitoring and evaluation officer GDAO Dereje T/Michael Project coordinator Meseret Derseh Accountant Yalemwork Beyene Sec. casher Assefa Almaw Driver Mulugata Aliyu Field officer—Bichena Muluken Atinafu Field officer—Dejen Yirsaw Mazengia Field officer—Gozamin HIDA Mathewos Taddese Monitoring and evaluation officer Sr. Tibebe Mulu Executive director Demeke Feissa Program manager 42 USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT Name of the Organization Name of Person Contacted Position in the Organization Ephrem Shiferaw Fundraising expert Lemma Demo Finance and admin off. Andualem Tesfaye Project coordinator Zemedikun Worku OVC officer Tadios Kebede CBHC program coor. Beza Girls Anti AIDS Club Ibrahim Yasin M&E officer Heregewoing Asmamaw Book keeper Zemenu Adis Mena HQ coordinator Getachew Belay Fellow Bashaw Getachew Fellow Amare Merka Program coordinator Kalid Yimer Grant officer Abdulhakim Husein Capacity building office Mekonnen Ali Generalist Dawud Muhe Generalist Mania Mohammed Board chairperson Birtukan Belayneh Volunteer REST, Wukro Gebrehiwot Hailu Coordinator Tsehainesh Abay Health dept. head Meherete-Ab Atsebeha IFHP project officer Kiflom Gidey Youth counselor Mebratey Alemayehu Youth volunteer Tesfahun Dagnew Youth volunteer Meseret G/Hiwot Youth volunteer YEGEB Zewdinesh Tesfaye Program coordinator Joshua Alemayehu National director Martha Fantahun Accountant Askale Demissie Social worker Matias Abebe Project officer Win Souls for God Sosena Shigeraw Project coordinator Biruk Getaghew Project manager Sewit Ketama Finance director Hawi Badasa Program coordinator Addis Merlaku Accountant USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT 43 Name of the Organization Name of Person Contacted Position in the Organization Mengistu Bunaro Human resources manager Gizachew Ayka Coordinator GZDA Ayele Etifu Project site coordinator Dawit Zerga Project site coordinator Yonas Teklegiorgis Project site coordinator Derye Tekim GZDA program coordinator Feleke Lemna GZDA activity manager Fair Yecommunity Limat Mahiber Zebib Kubsa Finance officer Carol Yohannes Program manager HFC Miyiku Lemma SCRHA economic secretariat specialist Berhamo Desta M&E officer HFC Tezera Likissa M&E officer YZMCCO Abraham Degu Program manager Yoseph G/Mariam Project officer Abreham Teshome Community worker (CW) Getachew Yirga CW Sagni Ayana Nurse Zenash Shrwaye CW Bemnet Aychegtew CW Tizita Tsehaye Accountant Mesibu Kebede Managing director Zelalem Ashenafi Admin & finance manager Propride, Diredawa Anneleye Fantahun Care & support officer Berhanu Habetewold HAC coordinator Kebele Teferi Aberia ES officer Jonatan Chali Kebele manager Dawit Mekonnen Senior finance officer Mesfen Tesfaye CM/CB officer Femas Mohammed HAC member Medidim, Adama Tamire Ahale Head finance & admin Dereje Beredede HR Mekonnen Head PR & advocacy Aida Abdalla Finance & admin manager Eshetu Bekere Monitoring and evaluation manager 44 USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT Name of the Organization Name of Person Contacted Position in the Organization Alemnew Asebe Communication officer Amare Ayalew Monitoring and evaluation officer Tadersse Dagefa Program coordinator OSSA, Harar Kalid Ahmed Manager/program officer Asfanu Teshome Ass program officer Tsedet Melaku Finance officer Dr. Ibrahim Yusuf Program manager EECMY, Hossana Ato Solomon Shafamo President of EEMYC/SCS Amanuel Shiferaw Cluster coordinator Tesfaye Balushie IFHP program coordinator Zenebech Haile Woreda officer Tamirat Minassie Accountant Habtamu Eritro Director of development and social service commission(DASC) CVDA Dagne Xlegasu Program manager Yesemzer Bazozew Accountant at branch office Alemnesh Zena Acting executive director ISAPSO Beletu Mengistu Executive Director Tilahun Wondimu Impact mitigation team leader & OVC project coordinator Workenesh Ajema Field project officer USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT 45 46 USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT APPENDIX C. LIST OF ATTENDEES IPS CAPACITY ASSESSMENT OUT BRIEFINGS ATTENDEES FROM USAID/ETHIOPIA October 5, 2011 Meri Sinnitt, Office Chief, HAPN Sheri-Nouane Duncan-Jones, Deputy Office Chief & HIV/AIDS Team Leader, HAPN Jeanne Rideout, Health Team Leader, HAPN Warren Leishman, Legal Advisor, DIR Kristin Saarlas, HAPN Elina Sverdlova, HAPN James Wang, Civil Engineer, HAPN Dr. Samuel Hailemariam, HAPN Dr. Yirga Ambaw, HAPN Sileshi Kassa , HAPN Samson Oli, HAPN John McKay, PRM Guda Alemayehu, HAPN Petros Faltamo, HAPN Alemnesh Hailemariam, HAPN Suzie Jacinthe, HAPN TDYer Noreen Mucha, HAPN TDYer Padmaja Shetty, HAPN Melissa Freeman, HAPN TDYer Awoke Tilahun, PRM Semunegus Meherete, HAPN USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT 47 48 USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT APPENDIX D. KEY INFORMANT INTERVIEW GUIDES ETHIOPIA IMPLEMENTING PARTNERS ORGANIZATIONAL CAPACITY ASSESSMENT KEY INFORMANT INTERVIEW GUIDE Government Stakeholders: MOH/HAPCO (Federal level and in each respective regions) Introduction Good morning/afternoon, thank you for taking the time to talk to us! We are (name) and the purpose of our visit is to conduct an organizational capacity assessment for USAID/Ethiopia. USAID would like to have better understanding of the strengths, weakness and best practices in the institutional capacity building approaches of the implementing partners. We would also ask you for your recommendations to the Mission for its future institutional capacity building strategy and activities. Two teams are travelling over the country in the conduct of this assessment. We would like to ask you a few questions regarding capacity building methods and tools that the Implementing Partners and their sub grantees are using and we would appreciate to hear what you perceive as their strengths and weaknesses. Our discussion should last about an hour. We will be taking notes. Your comments will be summarized with the comments from the other interviews we do and will be used for a report we share with USAID/E. First, do you have any questions for us? Name of the Office (MOH/HAPCO)_____________________________________ Name of the key informant ____________________________________________ Position of the key informant __________________________________________ Contact Address (Tel_____________________ Email________________________) Questions 1. Has your office been involved in organizational capacity building activities provided by XXX (specific IP and/or local partners in the area)? Could you list them? 2. Were you involved in identifying CB needs? 3. Who are the primary beneficiaries of your organizational capacity building activities? 4. How does your office monitor and evaluate organizational capacity building activities of the implementing partners? USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT 49 5. Are PEPFAR supported HIV/AIDS prevention and care activities by different implementing partners in line with the GOE’s strategic directions? 6. Do you think that organizational capacity building should be a focus area? 7. Which types of organizational capacity building activities does your office recommend for future actions? 50 USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT ETHIOPIA IMPLEMENTING PARTNERS ORGANIZATIONAL CAPACITY ASSESSMENT KEY INFORMANT INTERVIEW GUIDE Local Partners(CBOs/NGOs/FBO/CSOs): Staffs/Representatives Local Partners Introduction Good morning/afternoon, thank you for taking the time to talk to us! We are (name) and the purpose of our visit is to conduct an organizational capacity assessment for USAID/Ethiopia. USAID would like to have better understanding of the strengths, weakness and best practices in the institutional capacity building approaches of the local partners. We would also ask you for your recommendations to the Mission for its future institutional capacity building strategy and activities. Two teams are travelling over the country in the conduct of this assessment. You have been identified as a respondent due to your affiliation with this program. We would like to ask you a few questions regarding methods and tools that you are using, and what you perceive as capacity strengths and weaknesses in organizational capacity. Our discussion should last about one and a half hour. We will be taking notes. Your comments will be summarized with the comments from the other interviews we do and will be used for a report we share with USAID/E. First, do you have any questions for us? Name of the Local Partner______________________________________________________ Name of the key informant person(s)________________________________________________ Position of the key informant(s) _______________ ____________________________________ Contact Address (Tel_____________________ Email________________________) Methods and Tools 1. When did you join the program? 2. How do you define capacity building? 3. What key areas of capacity building/organizational development have been targeted. (Leadership & management, structure, governance, roles & responsibilities, decision making framework, planning, financial, HR, information management, Quality Assurance, revenue generation, service delivery?) 4. Since when? 5. Were you trained as an individual, or was this part of organizational development strategy? 6. To what extent have CB training activities resulted in changes in your organization? 7. Why or why not? (barriers and facilitators). USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT 51 8. Are the CB activities meeting your needs in capacity building and training; what more is needed? 9. Does your organization collaborate with local government and/or local networks? How? Reporting? Specific capacity building assessments and activities? 10. How are you documenting and analyzing what works and what doesn’t? Do you have indicators for measuring (organizational) CB? 11. How do you measure, monitor and evaluate improvements in capacity over the long term to assure the sustainability of their interventions? 12. Is staff turn over a problem in your organization; how do you address that? 13. How do you address skills transfer and learning? Strengths and Gaps: 14. Which organizational activities now being carried out by you were you not able to do before working with the IP? 15. What could the IP you’re working with done differently to build your capacity? 16. Do you have an idea what causes the main challenges? Capacity Building and Organizational Development approaches: 17. Is your portfolio of capacity building and training activities meeting the needs of your organization and beneficiaries? 52 USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT ETHIOPIA IMPLEMENTING PARTNERS ORGANIZATIONAL CAPACITY ASSESSMENT KEY INFORMANT INTERVIEW GUIDE Implementing Partners (IPs): Staffs/Representatives of IPs Introduction Good morning/afternoon, thank you for taking the time to talk to us! We are (name) and the purpose of our visit is to conduct an organizational capacity assessment for USAID/Ethiopia. USAID would like to have better understanding of the strengths, weakness and best practices in the institutional capacity building approaches of the implementing partners. We would also ask you for your recommendations to the Mission for its future institutional capacity building strategy and activities. Two teams are travelling over the country in the conduct of this assessment. You have been identified as a respondent due to your affiliation with this program. We would like to ask you a few questions regarding methods and tools that you are using, and what you perceive as capacity strengths and weaknesses in organizational capacity. Our discussion should last about an hour. We will be taking notes. Your comments will be summarized with the comments from the other interviews we do and will be used for a report we share with USAID/E. First, do you have any questions for us? Name of the IP______________________________________________________ Name of the key informant person(s) _______________________________________ Position of the key informant in the IP____________________________________ Contact Address (Tel)_____________________ Email________________________) Organizational Capacity and Related Issues Methods and Tools 1. Did you join the program before or after the program started? 2. How do you define capacity building and how is it positioned in your organization (integrated, treated as a separate component, individual level, institutional etc)? 3. Is capacity building geared towards reaching a certain level of sustainability and part of your exit strategy? 4. What key areas of capacity building for organizational development are being targeted by your organization? (Leadership & management, structure, governance, roles & responsibilities, decision making framework, planning, financial, HR, information management, Quality Assurance, revenue generation, service delivery?) 5. Is it included in your agreement (with USAID, and with local partners) USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT 53 6. How do you address staff turnover (skills transfer and learning)? 7. How do you identify needs to capacity building in your organization? 8. What needs assessment tools are being used? Is this tool and the approach your organization is using evidence based? Who does the assessment? 9. Does your organization collaborate with GOE/local government entities/ CSO networks? How? Reporting? Specific capacity building assessments and activities? 10. How does your organization measure, monitor and evaluate improvements in capacity over the long term to assure the sustainability of their interventions? (M&E system) 11. What more could you be doing in M&E for CB? Strengths and Gaps: 12. Could you describe the main challenges for working with local organizations (government, CSO/CBO) specific to capacity building? 13. Could you list some best practices to CB? 14. Do you have an idea what causes the main challenges? 15. Which practices should be scaled up? Are there any that should be discontinued? 16. How is your M&E system documenting and analyzing what works and what doesn’t? Do you have indicators for measuring organizational CB? 17. Do you have any suggestions on effective methods/approaches for closing the performance gaps in 1) management & leadership; 2) HR; 3) financial management; institutional development; 4) technical aspects? Capacity Building and Organizational Development Approaches: 18. Are there capacity building interventions/ approaches that should be discontinued? If so which and why? 19. Are there ways that capacity building could be better integrated among IPs? What could USAID do to make this happen? What implications do you think this can have? 20. Is your portfolio of capacity building and training activities meeting the needs of your organization and beneficiaries? 54 USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT APPENDIX E. ETHIOPIA IMPLEMENTING PARTNERS ORGANIZATION CAPACITY ASSESSMENT FOCUSED DISCUSSION GUIDE ETHIOPIA IMPLEMENTING PARTNERS ORGANIZATIONAL CAPACITY ASSESSMENT FOCUSED DISCUSSION GUIDE Staffs of local partners that participated in CB activities Introduction Good morning/afternoon, thank you for taking the time to talk to us! We are (name) and the purpose of our visit is to conduct an organizational capacity assessment for USAID/Ethiopia. USAID would like to have better understanding of the strengths, weakness and best practices in the institutional capacity building approaches of the implementing partners. We would also ask you for your recommendations to the Mission for its future institutional capacity building strategy and activities. Two teams are travelling over the country in the conduct of this assessment. We would like to ask you a few questions regarding capacity building methods and tools that the Implementing Partners and their sub grantees are using and we would appreciate to hear what you perceive as their strengths and weaknesses. Our discussion should last about an hour. We will be taking notes. Your comments will be summarized with the comments from the other interviews we do and will be used for a report we share with USAID/E. First, do you have any questions for us? Participants Information: Location: ____________________________ Name of Participant Organization Position in the organization USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT 55 Questions for discussion: 1. In what type of capacity building activities have you participated in the last 2 years (Trainings, Experience sharing, Mentoring, TA)? And how? 2. What tools and approaches were used? 3. What have you been able to do with the CB assistance that you received that you could not have been able to do otherwise and how did you apply that? 4. To what extent have CBT activities resulted in changes in your organization? 5. Are the CBT activities meeting your needs in capacity building and training and what more is needed? 6. Are there areas of innovation that should be given more emphasis and which are these? 56 USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT APPENDIX F. CAPACITY BUILDING FRAMEWORK Capacity Outputs Capacity Outcomes Capacity (Competencies (Performance and Strategy Inputs and Efficiencies) Effectiveness) Health Impact Capacity Building Strategy and Partnerships Capacity Building Activities System/Policy Outputs Organizational Outputs Individual Workforce Outputs System/Policy Capacity Organizational Capacity Individual Workforce Capacity Sustained Performance Over time Improved health outcomes External Country Context Cultural, Social, Economic, Legal Community Behaviors and Use of Services USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT 57 58 USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT APPENDIX G. DRAFT OF WORK PLAN IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT DRAFT WORK PLAN Draft 9/15/2011 Introduction: GH Tech is pleased to submit this preliminary work plan as Deliverable #1. The Work Plan has been broken down into eight items. A. Background B. Purpose/Objectives C. Data Sources and Analysis (evaluation methodology) D. List of organizations/individuals included in interviews E. Evaluation timeline F. Responsibilities of team members G. Outline of the draft report H. Illustrative key interview guides for key informant groups We hope that this meets your expectations and look forward to any comments or suggestions for modifications. A. BACKGROUND In July 2011, USAID requested an assessment of the organizational capacity interventions of select PEPFAR/USAID partners. The main objective of the assessment is to determine strengths, weaknesses and best practices in the local institutional capacity building approaches of the selected implementing partners and to provide a set of practical recommendations that will facilitate programmatic decision making regarding future institutional capacity building approaches and activities. A three member consultant team was hired through GH Tech and met with the USAID-E team on Tuesday, September 13, 2011. The goal of the meeting was to clarify the purpose, expectations and agenda of the assignment. In addition, the team conducted its own 2 day Team Planning Meeting and decided on the following sources of data and methodologies to be used in the assessment. The team is going to be reinforced by an external USAID/Washington Capacity Building technical adviser and two USAID/E staff who will join the assessment during the site visits. B. PURPOSE OF THE ASSESSMENT The main objective of the assessment is to determine strengths, weakness and best practices in the local institutional capacity building approaches of select USAID implementing partners (IPs) and to provide a set of practical recommendations to the Mission for its future institutional capacity building strategy and activities. This will ensure that USAID supports evidence-based and proven effective programs. USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT 59 Objectives of the Assessment: 1. Describe the various approaches, models and tools used by partners to design, implement, monitor and evaluate local capacity building activities. Assess whether these approaches, tools, etc. align with published or documented best practices and recommendations on local capacity building. 2. Identify strengths, weaknesses, challenges, and best practices with respect to local capacity building interventions. 3. Identify innovative capacity building interventions that can be reinforced and/or replicated in future programs and make recommendations on how USAID can standardize support to and M&E systems of IPs working on capacity building of local organizations. C. DATA SOURCES AND ANALYSIS Five sources of information will be used during this project evaluation. These include key archival documents and reports, in-depth key informant interviews, focus group discussions, a brief survey and direct observation. Aside from Addis Ababa, 5 Regions and one City Administration will be visited for conducting case studies of the IPs and their sub grantees. The first source of information will be key documents and reports created by the IPs to describe the progress of their CB activities over the past years, country strategic plans and other relevant documentation, the CB tools, training offerings and collaborative learning forums. Documents considered critical to this study include but are not limited to IPs PEPFAR Semi-Annual Reports, IPs Annual Reports, IPs Quarterly Reports, GOE HSDP IV, FHAPCO SPMII, GHI related documents, USAID trip reports summarizing past field visits to IPs, GOE Road Map for HIV/AIDS Prevention, Care and Treatment, Organizational capacity assessment and capacity building tools and materials used by the IPs (e.g. OCAT, PIAI, PICAT) The second source of information for this review will be from key informant interviews. Interview sessions will be conducted face-to-face and will follow a semi-structured format using an interview guide but will allow for relevant, unplanned discussions, where relevant focus group discussions will be conducted. Notes from the interviews will be filled in by the consultants to assure completeness immediately following the end of each discussion. All interview notes will be analyzed to identify themes in the comments related to the overarching evaluation questions. Interview Guides to be used in the analysis of the interview data are included in this work plan. Site visits will also provide an opportunity for observation of use of CB products and tools. They also will allow for focus group discussions of individuals who have participated in training programs. Focus group discussions will be facilitated by one consultant using a semi-structured format. A focus group guide identifies questions to be asked about the effectiveness of the design of the training program and about the impact of the program on the individual and his/her organization. Notes will be taken by a second consultant during the group discussion. The notes will be analyzed in tandem with the analysis of the key informant interview data. D. ORGANIZATIONS/INDIVIDUALS TO BE INTERVIEWED Key informants to be interviewed include:  USAID Mission staff/focal persons including those working on health systems strengthening and capacity development  USAID implementing partners—PACT, Geneva Global, World Learning, Save the Children, Pathfinder, and PATH staff and respective USAID project AOTRs 60 USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT  Staff of selected 3 local organizations supported by each of the six IPs (actual number to sample is TBD)  Government of Ethiopia representatives including Federal and Regional HIV/AIDS Prevention and Control Organization (HAPCO) office staff, Federal Ministry of Health and Regional Health Bureaus Focus group discussions will be used for interviewing local organization staff including management and administrative staff and those who have participated in capacity building activities (e.g., training and workshops) supported by the four IPs. Local sub-grantees of the IPs were selected for the assessment on purpose taking samples of those that have low, medium and high performances after the capacity building interventions by the respective IPs. Criteria for selection of the CBOs/NGOs were: 1. NGOs/CBOs that are sufficiently significant to make an impact 2. Capacity building (CB) improved networking and institutional capacity for service delivery (high, medium, low) 3. CB improved networking and institutional capacity for sustainability (high, medium, low) 4. CB strengthened data demand and use (high, medium, low) 5. CB contributed to successfully generate, organize, process &disseminate useful knowledge in support of operations (high, medium, low) E. EVALUATION TIMELINE Key tasks and deliverables will be accomplished according to the timeline provided below: Dates Task/Deliverable Sept 4 Team begins review of documents and travel to Ethiopia Sept 13 -14 Team planning meeting, Addis Sept 14 Workplan delivered to USAID for review Sept 17 Team revises work plan and data collection instruments and continues review of documents and determines additional data needs Sept 15-Sept 16 Conduct interviews in Addis Ababa Sept 17 Begin data analysis before visiting the field Sept 19-Sept 29 Field visits Sept 30-Oct 4 Finalize data analysis, Develop findings and recommendations, Write draft report Oct 4 Prepare power points for presentations Oct 5 Presentation of preliminary findings and recommendations to USAID and IPs, sharing best practices and training on CB Oct 6 JR and KS depart Oct 18 Feed back on draft report from USAID Oct 19-21 Finalize report F. RESPONSIBILITIES OF TEAM MEMBERS Janneke Roos, Team Leader, is responsible for overall team organization and coordination, management of teamwork schedule to ensure tasks are completed according to the workplan, and interface with the USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT 61 USAID management team. She is also responsible for managing report writing and completion and submission of all deliverables. Substantively, she will focus on program wide capacity building issues affecting overall performance of the IPs and their sub grantees, client satisfaction and future directions. Together with the Capacity Development Specialist, she will conduct field-based interviews. Alemneh Tadele, the team’s Capacity Development Specialist, will take responsibility for review and analysis of all capacity building and training aspects of the evaluation, in addition to providing inputs and comments on other aspects of the evaluation and share responsibility for preparation of the final report. Dinsry Berhanu, the team’s Logistics Coordinator will be responsible for developing the final schedule with USAID, the evaluators and IPs, making lodging and transportation arrangements, and scheduling meetings. Kenneth Sklaw, USAID/Washington Capacity Building technical adviser will assist in conducting the assessment and provide input in the final report. 62 USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT G. OUTLINE OF THE DRAFT REPORT Table of Contents Acknowledgements I. Executive Summary II. Recommendations III. Introduction: A. Problem statement B. Purpose C. Assessment Questions D. Methodology E. Limitations to methodology IV. Capacity Building Approaches, Methods and Tools (Client satisfaction, alignment to needs, performance against CB plans) A. Findings and conclusions B. Recommendations V. Organizational Capacity Strengths and Gaps A. Findings and conclusions B. Recommendations VI. Opportunities and Future Directions A. Findings and conclusions B. Recommendations Appendices SOW Persons Contacted Itinerary Assessment instruments References USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT 63 H. KEY INFORMANT INTERVIEW GUIDES 64 USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT APPENDIX H. DRAFT OF EVALUATION METHODOLOGY Evaluation Question Relevant Documents Key Informants Analytical Method/ Key Questions Key Responsible Person 1. Methods and tools used by the IPs to assess 1) needs of local organizations, 2) design appropriate capacity building interventions based on identified gaps, 3) implement the intervention, monitor and evaluate improvements in capacity 1.a. What key areas of capacity building/organizational development are being targeted by the IPs Workplans & Progress reports IPs To what extent is CB focusing on the individual level? All team members for To what extent is CB focusing on the organizational level (Leadership & interviews Management, Structure (governance, roles & responsibilities, decision making framework), Systems (planning, financial, HR, information management, QA, revenue generation), service delivery? To what extent is CB focusing on the wider Needs assessments/baselines IPS, Host country, USAID, NGO/CBO How have you determined the areas of capacity building? To what extent participated NGOs/CBOs in the identification process. Training evaluations IPs, Host country, USAID What have you been able to do with this assistance that they would not have been able to do otherwise? NGO/CBO IPs Would you use these methods again? Why or why not? Host country USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT 65 Evaluation Question Relevant Documents Key Informants Analytical Method/ Key Questions Key Responsible Person CBO/NGO How did you apply the CB/training that you received? Can you give me an example of how you have used your skills for evidence-based decision￾making? To what extent have CBT activities resulted in organizational change? CBO/NGO Did it meet your needs? How did it improve your performance? IPs Was there anything you were dissatisfied with. (interactions, technical, process, knowledge) CBO/NGO USAID Donors Is assistance aligned with the needs of the country? USAID Host country 1.b. What are the common features and main differences among partners approaches to capacity building Capacity building plans; Management assessments; Financial reports; Progress reports USAID & IPs Identify common features and main differences in CB approaches IPs Assess effectiveness of the different approaches on improving the 7 elements in the capacity framework vis-à-vis good practices elsewhere Ips Document extent of balance between funding sources and type of activity financed and effect. NGOs/CBOs Extent to which transfer of learning occurs across elements IPs, CBOs/NGOs What are your activities/tools, etc to cover gender adequately? 1.c. Are partners using tools and approaches that have been previously tested or are based on standardized/evidence based practices Workplans; annual reports; capacity building plans; examples of good practices IPs, USAID, other donors, CBOs/NGOs How do you test whether CB/OD products/methods/tools are appropriate to the local context (e.g., content and format)? Are you learning from other donors, other local stakeholders, etc. and sharing your experiences? 66 USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT Evaluation Question Relevant Documents Key Informants Analytical Method/ Key Questions Key Responsible Person Website Downloads; email; phone USAID; IPs; Host country; Donors With what other people/project/groups should the IP have been partnering but was not, and what were the obstacles Host country; USAID; Donors; IPs How do you collaborate with partners to make the national health strategy work in the area of health system strengthening; do you sit in a working group or other collaborative bodies? Are you part of a community of practice (COP)? 1.d. How do IPs measure, monitor and evaluate improvements in capacity over the long term to assure the sustainability of their interventions? Are there approaches to provide CQI, address staff turnover and diffusion of skills transfer and learning? MIS, CQI protocols (for data collection, data analysis, reporting), HR management system, knowledge management IPs; CBOs/NGOs; USAID Describe your system to measure, monitor and evaluate capacity building effectiveness, effects and impact? Do you have CQI protocols (for data collection, analysis and reporting)? Do you have HR management plans/guidelines; do you have a knowledge management system, How do you respond to feedback? 2. What are the most common organizational capacity strengths and gaps identified in terms of human, institutional, material, financial, technical and project management among targeted CBOs 2.a. What do beneficiaries, Interviews, self USAID; host Are the CB products/methods/tools for OD meeting your needs (content IPs, and other assessment? , lessons country, and format)? List them? Are you able to apply the knowledge/skills to stakeholders perceive are learned donors, IPs, your work? How did it improve your performance? the contributing factors CBOs/NGOs What do you like and dislike about them? What are the areas of for both strengths and improvement that should be given more emphasis? and gaps? 2.b. What do beneficiaries Self-assessment CBO/NGO; Questionnaire and Ips perceive are the IP How do you respond to the feedback? most effective methods/ approaches for closing the performance gaps for each capacity area? USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT 67 Evaluation Question Relevant Documents Key Informants Analytical Method/ Key Questions Key Responsible Person 2.c. What are the main challenges and best practices for working with local organizations? Focus group, documented lessons learned IPs Discuss in focus group 3. What approaches to capacity building and organizational development should USAID reinforce or scale up in the future? 3.a. Are there CB interventions/ approaches that should be discontinued. If so why? Lessons learned USAID; host country; IPs; CBO/NGO Do you have evidence that a particular intervention did not result in a change in the way you work? What are your capacity building goals for the next year? What activities (as a result of capacity building) now being carried out would you expect to/want to yield result by the end of the project? (On technical level and on organizational level). 3.b. Should USAID consider having a separate entity that provides TA on organizational capacity building to all IPs in HAPN? If so, how should this be designed? Web downloads of best practices; documentation of lessons learned by other stakeholders (donors) USAID, host country, other donors, IPs What could be the advantages of having a separate entity that provides TA on OD to all IPs (think about value for money)? Should this be a national entity, a regional entity or an international one? 3.c. Should USAID consider having CB integrated or mainstreamed among IPs? (get consensus about the concepts and their implications) Focus group, documented lessons learned USAID; host country; IPs; CBO/NGO What could be the advantages of having CB mainstreamed among IPs? If so what could USAID do to better standardize interventions and indicators/metrics to measure improvement among IPs; What are your specific recommendations for programming to include in the PEPFAR COP 12? 3.d. Is there a tested and/or recommended M&E system for the organizational capacity building interventions? If yes what are they? If not what do you recommend M&E plan IPs; CBOs/NGOs Do you have a performance information system in place. Do you conduct DQA at regular intervals? Can you give a concrete example of evidence based decision making? 68 USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT Key Responsible Person Evaluation Question Relevant Documents Key Informants Training What is the quality of the Training evaluations Training Observation of training activities; what are the standards for the training trainings (workshops and participants & curriculum and how were they developed? Are former participants using distance learning) based orgs the skills they learned in training? Why or why not? (barriers and on the available evidence facilitators) (for example, evaluations by the participants including trainers, and others)? To what extent is the USAID; List activities. Are these activities meeting your needs in capacity building project’s portfolio of CBOs/NGOs; and training and what more is needed? capacity building and IPs; host training activities meeting country the needs of stakeholders? Analytical Method/ Key Questions USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT 69 70 USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT APPENDIX I. ITINERARY ETHIOPIA IMPLEMENTING PARTNERS’ ORGANIZATION CAPACITY ASSESSMENT Sept. 12–Oct. 5, 2011 Itinerary Monday, Sept. 12 Janneke Roos arrives in Addis at 7:25AM on ET Flt# ET707 Taxi ride to Sidama lodge Tuesday, Sept. 13 7:30am–8:30am—Pick up Alemneh from home and Janneka from hotel and head to Jupiter Hotel￾Cazanchise. Pick up by Medir Travel 8:30am–9:30 am—Janneke, Alemneh and Dinsry meet to get organized and share information at Jupiter Hotel Cazanchise Board room. 9:30am–12:30pm—Meeting with USAID Agreement Officer Technical Representative (AOTRs) 12:30pm–1:15pm—Lunch break 1:15pm–5:00pm—Team meeting 5:00pm–6:00pm—Drop off a hotel and home Wednesday, Sept. 14 7:30am–8:30am—Pick up Alemneh from home and Janneka from hotel and head to Jupiter Hotel￾Cazanchis 8:30am–11:15am—Team planning meeting 11:15am–12:00pm—Drive to USAID and security check 12:00pm–1:00pm—Meeting with AOTRs over lunch 1:00pm–2:00pm—Briefing meeting with USAID staff 2:00pm–2:30pm—Drive back to Jupiter Hotel 2:30pm–5:00pm—Team planning meeting 5:00pm–6:00pm—Drop off at hotel and home USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT 71 Thursday, Sept. 15 Meeting with implementing partners at their respective offices 7:30am–8:30am—Pick up Alemneh from home and Janneke from hotel and drive to Save the Children US/TransAction (Old Airport, near Bisrate Gebriel) (betekirstian) 9:00am–11:00am—Meeting with Save the Children US/TransAction 11:00am–12:45pm—Early lunch 12:45am–1:30pm—Drive to Geneva Global (Mina Building, Wello Sefer) 1:30pm–3:15pm —Meeting with Geneva Global 3:15pm–3:45pm—Drive to Integrated Family Health Program IFHP (Old Airport, near Bisrate Gebriel) 3:45pm–5:30pm—Meeting with Integrated Family Health Program IFHP 5:30pm–6:30pm—Drop off at hotel and home Friday, Sept. 16 Meeting with implementing partners at their respective offices 8:00am–9:00am—Pick up Alemneh from home and Janneke from hotel and drive to PACT (Bole Japan Embassy on the way to the ring road) 9:00am–11:00am—Meeting with PACT 11:30am–12:30pm—Lunch 12:30pm–1:00pm—Drive to PATH (Bole, Getu Building back side 2nd floor) 1:00pm–2:45pm—Meeting with PATH 3:15pm–5:00pm—Meeting with World Learning/GSM (near Meskel flower) 5:00pm–6:00pm—Drop off at hotel and home Saturday, Sept. 17 Team in Addis Sunday, Sept. 18 - Thursday, Sept. 29 Team 1 & 2 on site visit. Refer to Annex A and B below Friday, Sept. 30 Team in Addis–Report writing Saturday, Oct. 1 Team in Addis–Report writing Sunday, Oct. 2 Team in Addis–Report writing 72 USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT Monday, Oct. 3 - Team in Addis–Report writing Tuesday, Oct. 4– De-briefing at USAID Time TBD Wednesday, Oct. 5 – Workshop/debrief with the IPs Debrief with USAID (TBD) Thursday, Oct. 6–Janneke and Ken depart USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT 73 ANNEX A Team 1 Site Visit Sunday Sept. 18–Thursday Sept. 29, 2011 Addis Ababa, Dire Dawa, SNNPR, Harar and Oromia region Team members—Janneke Roos and Petros Faltamo (USAID–AOTR) Sunday, Sept. 18 9:00am–4:00pm–Pick up team members from hotel and home (old airport area) and drive to Yirgalem (325km from Addis) 4:00pm–4:30pm—Hotel check-in Monday, Sept. 19 7:00am–8:30am—Hotel check-out and drive to site in Yirgalem 8:30am–11:30am—Meet with CVDA staff 11:30am– 1:00pm—Lunch and drive to Hawassa (40km from Yirgalem) 1:00pm–2:00pm—TBD (SNNPR HAPCO) 2:00pm–6:00pm—Drive to Adama 6:00pm–6:15pm—Hotel check-in Tuesday, Sept. 20- 7:30am–8:30am—Hotel check-out and drive to site in Adama 8:30am–11:30am—Meeting with Mekdim 11:30am–12:30pm—Drive to Bishoftu 12:30pm–1:30pm—Lunch 1:30pm–2:00pm—Drive to site in Bishoftu 2:00pm–4:00pm—Meet with Fair Yecommunity Limat Mahiber 4:00pm–6:00pm—Drive back to Addis and drop off at home/hotel Wednesday, Sept. 21 8:00am–9:00am—Pick up from Hotel/home and drive to Win Souls for God Evangelical Ministries (WSGEM) Geneva Global Inc.(IP) 9:00am–12:00pm—Meet with WSGEM 12:00pm1:00pm—Lunch 1:00pm–2:00pm—Drive to Networks of Networks of HIV Positives in Ethiopia (NEP+) World Learning/GSM (IP) 2:00pm–5:00pm—Meeting with NEP+ 74 USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT 5:00pm–5:30pm—Drive to Jupiter Hotel—Cazanchis 5:30pm–7:30pm—Team meeting 7:30pm–8:30pm—Drop off at hotel/home Thursday, Sept. 22 6:00am–8:30am—Pick up from home/hotel and drive to Wolkite (158km from Addis) 8:30am–11:30am—Meet with Guraghe Zone Development Assosiation (GZDA) World Learning/GSM (IP) 11:30am–12:30pm—Lunch 12:30pm–2:00pm—Drive to Sebeta 2:00pm–5:00pm—Meet with Yezelalem Minch Children and Community Organization (YZMCCO) World Learning/GSM (IP) 5:00pm–6:00pm—Drive back to Addis and drop off at hotel/home Friday, Sept. 23 6:00am–10:00am—Pick up from home/hotel and drive to Hossana (Achamo) (232km from Addis) 9:00am–1:00pm—Meet with EECMY/SCS IFHP (IP) 1:00pm–2:00pm—Lunch 2:00pm–6:00pm—Drive back to Addis and hotel/home drop off Saturday, Sept. 24 – Team in Addis Sunday, Sept. 25 5:30am–6:00am—pick up from home and hotel and drive to the airport. 7:00am–8:00am—Flight to Dire Dawa (ET200) Hotel check-in Monday, Sept. 26 - 8:30am–9:30 am—Drive to site in Dire Dawa 9:30am–12:30pm—Meet with ProPride Transaction (IP) 12:30pm–1:30pm—Lunch 2:00pm–4:00pm—TBD (HAPCO) 4:00pm–5:00pm—Drive back to hotel Tuesday, Sept. 27 6:30am– 8:30am—Hotel check-out and drive to Harar 8:30am–11:30am—Meeting with Social Services Against AIDS (OSSA) Geneva Global (IP) USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT 75 11:30am–1:00pm—Drive back to Dire Dawa airport 2:15pm–3:00pm—Flight back to Addis (ET327) 3:00pm–4:30pm—Airport pick up and drop off at home/hotel in Addis Wednesday, Sept 28 Holiday Thursday, Sept. 29 7:00am–9:00am—Pick up from hotel/home and drive to Tulubolo (81km from Addis) 9:00am–12:00pm—Meeting with ISAPSO PACT (IP) 12:00pm–3:00pm—Lunch and drive back to Addis 3:30pm–5:00pm—TBD 5:00pm–6:00pm—Drop off 76 USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT ANNEX B Team 2 Site Visit Sunday Sept. 18–Thursday Sept. 29, 2011 Addis Ababa, Amhara, Tigray, and Oromia—Team members Alemneh Tadele and Seleshi Kassa Sunday Sept. 18 12:00pm–1:00pm—Pick up from Hotel/home (around Atoboos tera) and drive to Airport. 3:00pm–4:00pm—Flight to Bahir Dar (ET140) 4:00pm–5:00pm—Airport pick-up and drive to hotel. Hotel check-in Monday, Sept. 19 8:00am–9:00am—Drive to site#1 in Bahir Dar 9:00am–12:00pm—Meeting with YeEthiopia Gubae Egziabiher Betkirstian Limat Mahiber (EGEBLM) Geneva Global (IP) 12:00pm–1:00pm—Lunch 1:00pm–2:15pm—Meeting with Amhara HAPCO (TBD) 2:15pm–2:30pm—Drive to site#2 in Bahir Dar 2:30pm–5:30pm—Meeting with Alem Children Support Organization (ACSO) PACT (IP) 5:30pm–6:00pm—Drive back to hotel Tuesday, Sept. 20 6:00am–10:00am—Drive to site in Bichena (about 200km) 10:00am–12:00pm—Meeting with Guhion Development Aid Organization (GDAO) IFHP (IP) 12:00pm–1:00pm—Lunch 1:00pm–2:00pm—Continue meeting with GDAO 2:00pm–6:00pm— Drive back to Bahir Dar and hotel drop off Wednesday, Sept. 21 6:00am–7:00am—Hotel check-out and drive to airport. 8:30am–9:20am—Flight back to Addis (ET126) 9:30am–10: 30am—Airport pick up and hotel/home drop off 10:30am–1:00pm—Lunch/break 1:00pm–2:00pm—Hotel/home pick-up and drive to site in Addis 2:00pm–5:00pm—Meeting with Hiwot Integrated Development Association (HIDA) PACT (IP) 5:00pm–5:30pm—Drive to Jupiter Hotel—Cazanchis USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT 77 5:30pm–7:30pm—Team meeting 7:30pm–8:30pm—Drop off at hotel/home Thursday, Sept. 22 9:00am–4:00pm—Hotel/home pick up and drive to Kombolcha (375km from Addis) 4:00pm–5:00pm—Hotel check-in Friday, Sept. 23 8:00am–10:00am—Hotel pick up from Kombolcha and drive to site in Dessie. (25km from Kombolcha) 10:00am–12:00pm—Meeting with Beza Girls and Anti AIDS Club (Beza GAAC) PATH (IP) 12:00pm–1:30pm—Lunch 1:30pm–2:30pm—Continue meeting with Beza GAAC 2:30pm–4:30pm—Drive back to hotel in Kombolcha. Saturday, Sept. 24 9:00am–5:00pm—Drive to Mekele (about 405km from Kombolcha) 5:00pm–6:00pm— Hotel check in in Mekele Sunday, Sept. 25 Team in Mekele Monday, Sept. 26 6:30am–8:00am— Hotel check out and drive to site in Wukro (40km from Mekele) 8:30am–11:30am—Meeting with REST IFHP (IP) 11:30am–1:00pm—Lunch and driving to Tigray HAPCO (TBD) 1:00pm–2:00pm—Meeting with Tigray HAPCO 2:00pm–2:30pm—Drive to Mekele Airport 4:50pm–6:15pm—Flight back to Addis (ET105) 6:30pm–7:30pm—airport pick up and hotel/home drop off Tuesday, Sept. 27 7:00am–9:00am—Hotel/home pick up and drive to site in Ambo. 9:00am–12:00pm—Meeting with Biftu PATH (IP) 12:00pm–5:00pm—Drive back to Addis, lunch, home/hotel drop off Wednesday, Sept. 28 Holiday Team in Addis Thursday, Sept. 29 9:00 am–11:00am—TBD (Meeting with FHAPCO) 2:00pm–4:00pm—TBD ( Meeting with Oromia HAPCO) 78 USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT APPENDIX J. REFERENCES Please note that while not all references in this annex are mentioned in the body of the report, all documents listed here have been reviewed for the purposes of the Ethiopia Implementing Partners’ Organizational Capacity Assessment. Assefa, Berhanu “Monitoring and Evaluation training for WL/GSM grantees – Training Report.” December 2010. “Capacity Building Workshop Report: Phase I Grantees.” Addis Ababa, Ethiopia. August 2009. Federal Democratic Republic of Ethiopia Ministry of Health. “Health Sector Development Program IV 2010/11 – 2014/15.” Final Draft. August 2010. Federal HIV/AIDS Prevention and Control Office (FHAPCO). “Draft Road Map for Intensifying the Multi-sectoral HIV/AIDS Response in Ethiopia 2010/11 – 2014/15.” Addis Ababa, Ethiopia. December 2010. Federal HIV/AIDS Prevention and Control Office and Federal Ministry of Health. “Strategic Plan II – For Intensifying Multisectorial HIV and AIDS Response in Ethiopia 2010/11 – 2014/15.” Addis Ababa, Ethiopia. February 2010. Federal Ministry of Health. “Human Resource for Health Strategic Plan, Ethiopia 2009-2020.” Version 3. Geneva Global, Inc. “Capacity Building Program – Draft Graduation Guideline.” June 15, 2010. Geneva Global, Inc and Impact Association for Social Services and Development (IMPACT). “Fixed Obligation Sub Grant Agreement Number GG-ETH-059-IMPACT-03.” October 2009 – September 2010. Geneva Global, Inc and Ye Ethiopia Guba Egziabher Betekrstyan Lemat Mahber (EGEBELM). “Fixed Obligation Sub Grant Agreement Number GG-ETH-058-EGEBELM-058-03.” October 2009 – September 2010. Geneva Global, Inc. “New Partners Initiative—Ethiopia Capacity Building Plan: October 2009 – September 2010.” November 2009. Geneva Global, Inc. “New Partners Initiative—Ethiopia Capacity Building Program Annual Report: October 1, 2009 – September 30, 2010.” October 2010. Geneva Global, Inc. “New Partners Initiative Ethiopia – Geneva Global Capacity Building Monitoring & Evaluation Framework.” HGSM Pre-Away Survey. November 2009. “Introduction to Organizational Capacity Development” – Pact Organizational Development Toolkit. Pact’s Learning Series Publications. January 2010. Livelihood and Food Security Technical Assistance. “Assessment of USAID/PEPFAR’s Economic Strengthening Programs in Ethiopia.” August 2010 – October 2010. Livelihood and Food Security Technical Assistance. “Assessment of USAID/PEPFAR’s Economic Strengthening Programs in Ethiopia—Annex.” August 2010 – October 2010. Management Sciences for Health. “Management and Organizational Sustainability Tool – A Guide for Users and Facilitators.” Third edition. 2000. USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT 79 “Muslim Agencies Recharging Capacity against HIV/AIDS M-ARCH Revised Work Plan.” April 2007 – March 2008. Pact Ethiopia. “Organizational Capacity Assessment Tool.” Developed by Pact in collaboration with other NGOs, with financial support from USAID. “Pact Ethiopia, PEPFAR Ethiopia FY 06 COP Activity Work Plan Template on Youth & Children with Health Options Involving Community Engagement Strategies – Y-Choices.” Addis Ababa, April 2006. Pact Services Organizational Capacity Assessment (OCA) Profile. PATH, Dawn of Hope Ethiopia Association, Hope for Children, Mekdim Ethiopia National Association, and the Organization for Social Services for AIDS. “Strengthening Communities’ Responses to HIV/AIDS – Ethiopia Performance Monitoring Plan (PMP).” Version 1.1. April 2009 – March 2012. September 30, 2009. PATH “Strengthening Communities’ Response to HIV/AIDS (SCRHA) Project Civil Society Organizations (CSOs) Capacity Assessment Tool.” September 2009. Pathfinder International “Implementing Partner Organizations (IPOs) – Financial and Materials Management Manual.” Addis Ababa, Ethiopia. April 2006. Pathfinder International, John Snow and Consortium of Reproductive Health Associations. “Integrated Family Health Program (IFHP) Quarterly Report: July – September 2010 and Year 2 Annual Summary Report: October 2009 – September 2010.” Addis Ababa, Ethiopia. November 2010. Pathfinder International, John Snow and Consortium of Reproductive Health Associations. “Integrated Family Health Program (IFHP) Year 3 – 3rd Quarterly Report: April – June 2011.” Addis Ababa, Ethiopia. August 2011. Pathfinder International – Ethiopia. “Organization Capacity Assessment (OCA) – WeSMCO Final Report.” Pathfinder International. “Pre-Award Survey.” August 2002. Save the Children USA – USAID TransACTION Program. “TransACTION Organizational Capacity Assessment Tool (TOCAT).” June 2010. “SCRHA Project National Implementing Partners (NIPs) Capacity Assessment Tool.” September 2009. “Strengthening Communities’ Responses to HIV/AIDS Annual Work Plan April 1, 2011 – March 31, 2012.” TransACTION. “Prevention and Care Services for At Risk Mobile Populations Year III Annual Work Plan and Timeline. May 2011 – April 2012.” TransACTION. “Expected Results 4.” Venture Philanthropy Partners by McKinsey & Company. “Effective Capacity Building in Nonprofit Organizations.” 2001. World Learning Ethiopia – Grant Solicitation and Management Project. “Grant Administration and Financial Management Supportive Supervision Checklist.” Addis Ababa, Ethiopia, November 2009. World Learning Ethiopia – Grant Solicitation and Management (GSM) Program Profile. August 2011. 80 USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT World Learning. “Participatory Institutional Analysis – An NGO Capacity Building Tool.” December 2010. World Learning. “Participatory Institutional Analysis Instrument Matrices.” Other Background Documents: Notes from Medico Social Development Organization (MSDAO) Supervision Visit. February 2011. “Checklists for Spot Checking.” PATH Scope of Work. USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT 81 82 USAID/ETHIOPIA: IMPLEMENTING PARTNERS’ ORGANIZATIONAL CAPACITY ASSESSMENT REPORT For more information, please visit http://resources.ghtechproject.net - - Global Health Technical Assistance Project 1250 Eye St., NW, Suite 1100 Washington, DC 20005 Tel: (202) 521 1900 Fax: (202) 521 1901 www.ghtechproject.com