11 PPL LER M&E IDIQ Contract No. AID-OAA-I-15-00015 Contract No. AID-603-TO-16-00001 June 23, 2016 This document was prepared for review by the United States Agency for International Development. It was prepared by Banyan Global under the PPL LER M&E IDIQ, Task Order AID-603-TO-16-00001 EXTERNAL EVALUATION AND PERFORMANCE MONITORING SERVICES FOR “STRENGTHENING HIV/AIDS INTERVENTIONS IN DJIBOUTI PROJECT” FINAL REPORT 2016 This publication was produced for the United States Agency for International Development, USAID Contract Number AID-OAA-I-15-00015/AID-603-TO-16-00001. It was prepared by Banyan Global and John Snow Inc. (JSI) through the PPL LER Monitoring and Evaluation IDIQ. Implemented by: Banyan Global 1120 20th Street NW, Suite 950 Washington, DC 20036 Phone: +1 202-684-9367 Fax: +1 202-697-5020 Disclaimer: This report is made possible by the support of the American People through the United States Agency for International Development (USAID). The contents of this report are the sole responsibility of Banyan Global and do not necessarily reflect those of the USAID or the United States Government. Project Description: The External Evaluation and Performance Monitoring Services for “Strengthening HIV/AIDS Interventions in Djibouti Project” is being implemented by Banyan Global and its partner John Snow Inc. (JSI). The objective of this task order is to provide USAID/Djibouti with an independent performance evaluation of the Capacity Building for HIV/AIDS Services Project (ROADS II). Managed by USAID/Djibouti, ROADS II is a performance based award whose purpose is to reduce HIV/AIDS transmission among the most vulnerable populations living in and around the Djibouti-Ethiopia transport corridor. Targeted populations receive access to HIV/AIDS detection and treatment services as well as training on the adoption of safer sex behaviors. USAID/Djibouti will utilize the findings of the evaluation in its determination as to whether the local sub-partners’ organizational, performance and financial capacities were improved by the project, as articulated in the project agreement. Recommended Citation: Banyan Global and John Snow, Inc., 2016. External Evaluation and Performance Monitoring Services for “Strengthening HIV/AIDS Interventions in Djibouti Project” Final Report. USAID. External Evaluation and Performance Monitoring Services for “Strengthening HIV/AIDS Interventions in Djibouti Project” Task Order. EXTERNAL EVALUATION AND PERFORMANCE MONITORING SERVICES FOR “STRENGTHENING HIV/AIDS INTERVENTIONS IN DJIBOUTI PROJECT” FINAL REPORT 2016 PPL LER M&E IDIQ Contract No. AID-OAA-I-15-00015 Contract No. AID-603-TO-16-00001 Submitted to: Mansour Ahmed Contracting Officer Representative USAID Djibouti External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 i TABLE OF CONTENTS ACRONYMS........................................................................................................................................................................... ii TABLES.................................................................................................................................................................................. iii FIGURES ................................................................................................................................................................................ iv EXECUTIVE SUMMARY..................................................................................................................... 1 PROJECT BACKGROUND ........................................................................................................................................... 1 EVALUATION OBJECTIVES.......................................................................................................................................... 1 METHODOLOGY......................................................................................................................................................... 1 KEY FINDINGS ............................................................................................................................................................. 3 CONCLUSIONS AND RECOMMENDATIONS............................................................................................................ 6 STRENGTHENING HIV/AIDS INTERVENTIONS IN DJIBOUTI PROJECT........................................... 9 EVALUATION OBJECTIVES AND QUESTIONS.................................................................................. 9 EVALUATION METHODOLOGY AND DESIGN ............................................................................... 10 METHODOLOGY....................................................................................................................................................... 10 EVALUATION DESIGN .............................................................................................................................................. 10 SAMPLING .................................................................................................................................................................. 13 LIMITATIONS................................................................................................................................. 14 KEY FINDINGS ............................................................................................................................... 15 HIV/AIDS SERVICE DELIVERY KNOWLEDGE ........................................................................................................ 18 CAPACITY BUILDING................................................................................................................................................ 26 CONCLUSIONS & RECOMMENDATIONS........................................................................................ 28 SERVICE DELIVERY..................................................................................................................................................... 29 CAPACITY BUILDING................................................................................................................................................ 30 ANNEX A: DATA COLLECTION TOOLS ........................................................................................ 32 ANNEX B: ORGANIZATIONAL CAPACITY TOOL COMPONENTS................................................ 102 ANNEX C: OBSERVATION AND EVALUATION BY ORGANIZATION ............................................ 103 ANNEX D: Summary Of Key Findings by Evaluation Question............................................ 130 External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 ii ACRONYMS AIDS Acquired Immunodeficiency Syndrome AOR Agreement Officer’s Representative ART Antiretroviral Therapy ARV Antiretroviral AZT Azidothymidine CBTA Capacity Building Technical Advisor ET Evaluation Team FBO Faith-Based Organization FDG Focus Group Discussions FSW Female Sex Workers HCT HIV Counseling and Testing HIV Human Immunodeficiency Virus HMIS Health Management Information Systems IV Intravenous Therapy KHB HIV Screening Test Kits KII Key Informant Interviews LDTD Long Distance Truck Drivers MoH Ministry of Health NAC National AIDS Commission NGO Non-Governmental Organization NPS National Program Strategy NVP Nevirapine PEPFAR The U.S. President's Emergency Plan for AIDS Relief PLS National HIV Management Program PLWHIV People living with HIV/AIDS ROADS II Strengthening HIV/AIDS Interventions in Djibouti Project SC Steering Committee TA Technical Assistance TB Tuberculosis UN United Nations UNGASS UN General Assembly Special Session on HIV/AIDS USAID United States Agency for International Development External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 iii TABLES Table 1: Evaluation Sample Table 2: Summary of Capacity Building Table 3: Evaluation Sample Table 4: Sample of Key Respondents Table 5: Evaluation Sites Table 6: Knowledge by Beneficiary Group Table 7: Beneficiary Perceptions Table 8: Beneficiary Knowledge and Perceptions on HIV Testing Table 9: Beneficiary Perceptions of HIV Testing Table 10: PLWHIV Perceptions Table 11: Number of People Receiving ARV Table 12: Number of Health Facilities with HIV/AIDS Products and Equipment External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 iv FIGURES Figure 1: Importance for a Health Provider to do HIV/AIDS Counseling and Testing Figure 2: Trends in Trained Counselors and HCT Services Provided Figure 3: The Five C's Of HIV Counseling and Testing Figure 4: Condom Distribution and New HIV Infections Figure 5: Post Counseling Messages Figure 6: Changes in Trained Community and Social Workers Providing Preventive Messages Figure 7: Health Provider Knowledge on When Different Population Should Repeat HCT Figure 8: Health Provider Knowledge When a Pregnant/Breastfeeding Should Repeat HCT Figure 9: Changes on the HIV Positive Cases Figure 10: PK12 Safe Stop Resource Center Achievement In 2015 Figure 12: Number of Patients Receiving Malaria Services In 2015 Figure 12: HIV Patient Management and Care Requirements at Labor, Delivery and Post Natal Services External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 1 EXECUTIVE SUMMARY PROJECT BACKGROUND The U.S. President's Emergency Plan for AIDS Relief (PEPFAR) Strengthening HIV/AIDS Interventions project under the ROADS II technical assistance program was implemented specifically to support the Government of Djibouti (GoDj) to reduce HIV/AIDS transmission among long-distance truck drivers (LDTD), female sex workers (FSW), girls and young women engaging in risky sexual behavior primarily for economic gain, and other vulnerable populations living in and around transport corridor communities. The project was led by FHI 360 who partnered with UNFD, ASO, RNDP+, Secretariat Executif de la lutte contre le SIDA, le Paludisme et la Tuberculose, PLS Sante, Massaba, PATH and Humanity Voice to implement the project from 2013–2017. In May and June 2016, the Evaluation Team (ET) conducted an evaluation of the Strengthening HIV /AIDS Interventions in Djibouti Project (ROADS II). The evaluation assessed project effectiveness with respect to achieving project targets related to service delivery and current capacity and performance of the Ministry of Health (MOH) and its partners to manage the National Program Strategy (NPS) and to provide HIV/AIDS prevention, treatment and supportive care services. The evaluation has also generated specific recommendations for the design and scope of future program activities to address the critical health needs of the target population. The evaluation was designed to determine which interventions are working effectively and to assess the likelihood that the project will reach its intended outcomes by the end of the implementation period (2017) including the transition of activities to local partners. To perform the evaluation, the ET reviewed project documentation and performance reports, interviewed 25 health providers, 7 NGO managers with along their 12 key program staffs as well as 62 beneficiaries to understand the strategies, processes and the outcomes of the project. The evaluation approach was guided by a research strategy that was developed and reviewed with key project partners and funders and guided by a local Steering Committee established by the MOH. EVALUATION OBJECTIVES Based on guidance established by USAID, the objectives of the evaluation were: • To describe and assess the changes in organizational capacity among NGOs, the GoDj Executive Secretariat, and the National HIV Management Program (PLS) to deliver services and manage HIV interventions; • To examine the link between improved local capacity and reduced risk of HIV/AIDS among key populations (including reduced transmission and improved health); • To determine the effectiveness of the program management/coordination and communication of the ROADS II Project. METHODOLOGY Evaluation Design. The ET was comprised of a Team Leader, Capacity Building Expert and two local consultants with appropriate evaluation and language skills and experience in public health program Types of Stakeholders Interviewed: • Health providers • NGO managers • Key program staff • Beneficiaries o Truck Drivers o Adolescents o PLWHIV o Sex Workers External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 2 design and management in Djibouti. The team employed a mixed methods strategy, drawing on existing quantitative and qualitative data and conducted primary qualitative data collection to describe and assess the observable changes made in strengthening the capacity of HIV/AIDS managers and service providers and institutions working to reduce the risk of infection and provide services to key populations. The qualitative methods were also use to gain an understanding of the perceptions of HIV/AIDS service availability, quality and effectiveness from the perspective of providers, managers and beneficiaries. The team conducted fieldwork over two weeks in Djibouti, working closely with local stakeholders to ensure relevance of the evaluation and sharing preliminary findings before departing the country. The strategy and approach utilized for this evaluation was vetted by the JSI Institutional Review Board Evaluation Respondents. The ET selected the sampling group for the evaluation across eight ROADS II project intervention sites that offered community- and facility-based HIV prevention, care and support assistance. The evaluation focused on six sites within Djibouti City, one site in Dikhil region as well as PK12. These sites were selected based on their exposure to the ROADS II project’s technical assistance (TA) interventions, their location along the transit corridor and/or the scale of ROADS II project activities that were implemented within that particular site. The ET collected the data for the evaluation through: 1) key informant interviews with the NGOs and government institution managers and key staff as well as with project trained health providers; 2) focus group discussions with beneficiaries including truck drivers, adolescents, sex workers, and PLWHIV; 3) HIV/AIDS healthcare facility observations; and 4) a desk review of project and program documents and routine data for research ethics. The ET designed, piloted, and refined a set of data collection tools including interview guides adapted for each of the following key respondent groups: • ROADS II project implementing partner and sub-partners’ staff; • Project trained staff from Government institutions and local HIV/AIDS organizations (NGOs, FBO and Civil Society); • Project-trained HIV/AIDS service providers at health facilities and supervisors of outlets distributing condoms; and • PLWHIV, vulnerable population and people at risk as beneficiaries of the ROADS II project services and technical support including sex workers, truck drivers, and youth. These tools can be found in Annex A. The sample size for the evaluation totaled 109 respondents, as shown in table 1. TABLE 1: EVALUATION SAMPLE Key Respondent Group Male Respondents Female Respondents Total No. of Respondents Implementing Partner and Sub-Partner Staff 12 7 19 Government Institutions and local HIV/AIDS Organizations 1 2 3 HIV/AIDS Service Providers 11 14 25 PLWHIV, Vulnerable Populations and People at Risk 33 29 62 TOTAL 57 52 109 External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 3 KEY FINDINGS The major findings of the evaluation of the Strengthening HIV/AIDS Interventions in Djibouti Project (ROADS II), relate to health worker knowledge in the context of service delivery and to local organizational capacity to manage HIV programs and service delivery. In terms of overall performance, the evaluation found a positive trend in the number of people receiving HIV counseling and testing (HCT) services and an increase in the detected HIV positive cases from among those tested (23% to 41.9% from 2013 to 2014), an increase in the provision of ARV to HIV patients since 2011, and a decrease in new HIV infections from 36% in 2013 to 30.6 % in 2015. Based on routine data collected from the PK12 safe stop resource center and displayed at the center, the center introduced services in 2015, recorded that 790 persons received HIV services. Among those clients, 383 accepted HCT services and two were found to be HIV positive. In 2015, however, the proportion of positive cases detected by the program dropped to 35.1%. This decline could be explained in a number of ways, including an increase in the number of HCT clients, successful awareness raising and behavior change at population level including increased condom use, HIV positive clients who were lost to follow up, or other reasons. Additional research is required to track epidemiological trends, behavior change, and the effectiveness of the routine reporting systems to estimate the contribution of the project to population level outcomes or explain the decline in cases detected. Finally, trends in condom distribution were overall very positive. The use of condoms as a prevention strategy led to an increase in the condom distribution from 23.1% in 2013 to 42.8% in 2015 based on the total number of condoms distributed over this period. Notable, however, are the evaluation findings that 1) document the levels of health worker knowledge and how this knowledge has translated into increased service delivery and beneficiary awareness related to HIV prevention and treatment; and 2) document local organizational capacity in areas such as governance, administration, human resources, financial management, organizational management, program management, and program performance. SERVICE DELIVERY Health worker knowledge about HIV services counseling, testing and treatment. In terms of improving service delivery at the project sites, the project focused on providing equipment and health commodities, increasing condom distribution, training of health providers to increase the number of people receiving services including counseling and testing and ART. The ET assessed health worker knowledge required to provide services focusing on counseling practices, steps in the counseling process, content of counseling and post counseling messages for specific groups (general population, youth, pregnant women) and patient management and care during labor, delivery, postnatal services. The ET found that health worker knowledge related to basic voluntary HIV Counseling and Testing (HCT) was adequate overall but found particular gaps in knowledge and areas that require specific attention. The majority of health providers interviewed clearly understand the importance of conducting HCT. The significance of identifying HIV positive persons and prioritizing the prevention of new infections was stated by 81% and 85.1% of health providers, respectively. In addition, the transfer of knowledge among trained counselors to clients was reflected among beneficiaries who testified to the importance of counseling and testing and the increase in the number of people accepting and receiving the HCT services (from 20% in 2013 to 46% in 2015). In contrast, health worker understanding of the HCT steps was limited and incomplete. Slightly more than 50% of health providers reported the steps of “consent, confidentiality and counseling” as part of External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 4 HCT. The remaining two steps – reporting correct results and making connections to other HIV services – were mentioned by less than the half of the providers interviewed. The evaluation found gaps in knowledge related to providing the required set of counseling messages to all clients during counseling and testing. The majority of health workers reported that all people with negative or positive HIV test results should receive post-HCT services. More than 90% of the health providers interviewed reported knowledge of key post counseling messages, including counseling clients on the risk of contracting HIV infection and the use of condom for safe sex. However, only half of the respondents stated the need for counseling on HIV testing for partners, repeating testing after 3 months for negative persons and safe feeding and infant prophylaxis for PMTC services. Less than 10% of the health providers named advice on contraception and fertility planning as one the key messages for clients following HCT. A small proportion of respondents reported knowledge on the periodicity of repeating HCT for HIV negative persons and different population categories. The timeframe of 6 to 12 months for the general population, adolescents and sex workers was reported by less than half of health providers. Sixty six percent of respondents reported that pregnant women should be retested following HCT and 61.9% reported this knowledge related to breastfeeding women. Although only medical doctors were trained to provide treatment to HIV patients, more than 60% of health providers know the 3 out of the 8 requirements for pregnant and breastfeeding women management and care during labor, delivery and postnatal services. More than half the respondents reported knowledge on the use of iron, folate and calcium supplementation, the provision of ARV and the need for counseling on safe sex. Counseling for pregnant and lactating women on family planning and contraception was only mentioned by 19% of respondents. Beneficiary knowledge and perceptions. The ET assessed beneficiary perception of the program along with knowledge and awareness on HIV, HCT and program services. Based on the findings from focus group discussion, more than half of the respondents reported that unprotected sexual intercourse, sexually transmitted disease and condom misuse were among the reasons to undergo HCT services. Nearly three quarters of FGD beneficiary respondents reported that they had accepted HCT and were given their results within a short period of time along with the support and care of the health provider. However, they had initially found it difficult to accept HIV testing in the first place because of the fear of being stigmatized and discriminated by their families and communities. The majority of the beneficiaries groups interviewed considered the home-based HCT more confidential and discrete than the facility-based HCD; the exception was adolescents and truck drivers who perceived home-based counseling as a breach of confidentiality between them and their families. Respondents also noted a preference for offering integrated services particularly at the PK12 safe stop resource center, because they were reluctant to be seen at a center that was designated for HIV care only. Finally, as noted above, the transfer of knowledge among trained counselors was reflected by beneficiaries who testified to the importance of counseling and testing. Project M&E data also showed an increase in the number of people accepting and receiving HCT services (from 20% in 2013 to 46% in 2015) based on data found in project reports. The evaluation also found that there were specific attitudes and beliefs that were unique to particular beneficiary groups. For example, truck drivers reported that, prior to marriage, individuals should be External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 5 counseled and tested for HIV. Young adolescents reported that victims of rape and traffic accidents should be offered HCT services. A key beneficiary population - PLWHIV - reported that health providers were sometimes not supportive and caring during the provision of ARV. Moreover, PLWHC had only received food supplementation once following the discovery of their positive status. Despite the project’s efforts to increase the number of trained community workers and social workers (from 31.3% in 2013 and 49.1% in 2014), the project’s M&E data shows almost no change in the percentage of the people reached through preventive messages (34.8% in 2013 and 32.7% in 2015). Equipment Availability. The required products and equipment for HCT and treatment services were not fully available in all visited health facilities. Only the HIV screening test kits (KHB) and the confirmatory test kits (Stat Pack) were available in all facilities. Tie Break Test Kits were available in 5 health facilities. Preventive equipment such as containers for disinfecting instruments, containers for blood soaked waste, and gloves were available in all facilities. CAPACITY BUILDING The ET found that among all the NGOs and government institutions assessed, governance capacity regarding mission/vision, organizational structure, composition and responsibilities and legal status was acceptable. The exception to this finding was the absence of succession planning in all of the project￾supported NGOs and government institutions for continuing program activities beyond the life of the ROADS II intervention. This capacity component is particularly weak for the HIV/AIDS Secretariat due to the transitioning of the institution into its new structure. Please see table 2 for a capacity building indicator summary. TABLE 2: SUMMARY OF CAPACITY BUILDING Indicators 2011 2012 2013 2014 2015 Targets Results Targets Results Targets Results Number of community workers trained 560 233 353 321 235 146 Number of active national TWGs committees (Gender TWG, MARPs-Key pop TWG, M&E TWG) 3 3 3 3 3 3 Number of quarterly coordination meetings convened by the Executive Secretariat 4 1 4 4 4 4 Number of HCT sites established 6 6 8 8 9 9 Number of people provided HCT and HIV test results 5400 5,250 10,560 9186 11,830 12,131 Number of people reached with the prevention messages 31,850 32,349 31,355 30,197 37,300 30,350 Number of condoms distributed ND 96,280 ND 142,06 6 ND 178,780 Number of new infection cases 248 610 NA 594 ND 551 ND 506 The operational policies, procedures and systems of local organizations supported by the project are strong. It was apparent that these institutions (Executive Secretariat, Massaba, RNDP+ and ASO) had improved their systems significantly in some areas, but not in others. The weakest elements of External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 6 organizational capacity related to human resources management, particularly staff performance and personnel policies. There were considerable gaps in the use of job descriptions, recruitment procedures, and staff retention among the majority of the NGOs; government institutions were particularly weak in this area. Regarding financial management, skills in auditing were lacking. However, organizations had improved in many general financial management systems – such as accounting systems, policies and financial procedures, budgeting, financial documentation and internal controls); the exception was RNDP+ which has weak internal controls and financial reporting. The organizations included in this evaluation were found to have acceptable standards and procedures related to organizational management but improvement is needed in the areas of resource mobilization. In program management, areas such as donor compliance, technical reporting and referral also require improvement. The evaluators found limited evidence that these organizations possess skills in these areas. With respect to project interventions, efforts should focus on monitoring, quality assurance and supervision in the context of performance management. The project’s efforts to build the capacity of local NGOs and FBOs resulted in: • The establishment of 3 Technical Working Groups as platforms to discuss and exchange project achievements, areas for improvement; • An increase from 1 in 2013 to 4 in 2014 in the number of coordination meetings among implementers led by the Government AIDS, TB and Malaria Executive Secretariat to discuss progress and challenges in the implementation of the activities and to propose solutions to better organize and coordinate efforts; • An increase from 6 in 2013 to 9 in 2014 of HCT sites to improve the quality of services and also increase the awareness of HIV prevention and accessibility; • An increase from 96,280 in 2014 to 142,066 in 2014 and to 178,780 in 2015 of the condoms distributed by the project to promote safe sex; and • A decrease of new HIV infection cases from 594 in 2013 to 551 in 2014 and to 506 in 2015. Currently the NGOs and FBOs supported by the project report their achievements and findings to only the ROADS II project. CONCLUSIONS AND RECOMMENDATIONS The evaluation of the Strengthening HIV /AIDS Interventions in Djibouti Project (ROADS II) revealed many positive changes in the health service delivery performance and the institutional capacity of the NGOs (FHI 360, UNFD, RNDP+, ASO, Massaba) and Government Institutions (Executive Secretariat, PLS Sante) that are addressing the health needs of populations living and working along the Ethiopia￾Djibouti transportation corridor. Notably, the knowledge of health providers related to HCT and service delivery practices is growing. Moreover, based on beneficiary focus group findings, health workers counseling and outreach appears to have contributed to improved awareness among the target population about HIV infection, preventive behaviors and care and treatment services. Indeed the acceptability of HCT services has increased considerably among beneficiaries, which was reflected in the increased number of condoms distributed and increases in the number of clients receiving HCT. In comparison to the results of the baseline organizational evaluation conducted by the project, the findings of this evaluation indicate that the overall capacity of local institutions appears to have improved consistently over time. The increased capacity to manage HIV interventions resulted in improved performance in overall coordination of service delivery and monitoring of services provided at the External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 7 community level. Despite this progress, however, this evaluation revealed the need for improvement in several aspects of organizational capacity, mainly in internal control/auditing, field activity monitoring, quality assurance and succession planning. To further strengthen and improve the project’s work to reduce the transmission of HIV, the ET recommends that the project take the actions described below. SERVICE DELIVERY • Positive trends on increasing access to essential HIV counseling care and treatment should be reinforced with additional investment in service delivery. The project should increase the number of health providers offering HIV treatment and care by training nurses and midwives and updating protocols to allow these cadres perform treatment services. • Based on the fact that the population reached through project messages did not increase (from 34.8% in 2013 to 32.7% in 2015), it is recommended that efforts should be made to increase the knowledge of health providers on HCT services. This includes the repetition of HCT services in different population groups, the requirements for HIV patient management and care, and the understanding of the different steps of HCT to re-enforce the already successful efforts to raise awareness of the population related to HIV prevention, care and treatment • Resource centers currently provide mainly HIV services and few other healthcare services and support. As a result, communities see the centers as mainly treating HIV positive patients. It is recommended that the program integrate HIV services into other health and community support services in the PK12 Safe Stop Resource Center to increase the use of services and to eliminate the stigmatization around users of the center services. • In group discussions, some beneficiaries expressed the belief that HIV positive persons should be isolated from their families and communities to prevent the spread of HIV. Therefore, the project should take steps to improve the understanding among beneficiaries of steps for HIV prevention and encourage positive attitudes towards PLWHA. • In HIV programs, a key strategy is to closely monitor infected persons to help them to live healthy life styles and to prevent the transmission of the virus to others. Since the project support is limited to prevention, the Ministry of Health should introduce more rigorous and high quality treatment services and follow-up strategies, particularly to monitor HIV positive cases and address the decline in positive cases (from 49.1% in 2014 to 35.1% in 2015), which may be due to loss during follow-up or death. • Information, monitoring and evaluation systems overall require strengthening among partner institutions as does the use of data to guide program strategy. Reporting to the national health information system from local partners is limited and limits government capacity to track national program efforts, identify gaps and report on successes. The program should invest in improved surveillance and monitoring systems to track clients and understand and use rigorous evidence around health seeking, preventive and adherence behavior. The use of a unique identifier-based tracking system for following itinerant populations has proven successful in others settings as has introducing ways for projects to map sexual and social networks to improve capacity to reach potential clients. • Links between health facilities and community groups are limited which is a missed opportunity to bolster program effectiveness and engendering shared accountability for program performance. The project should establish a liaison between the implementing local NGOs and FBOs and health facilities to enable supportive supervision and integration of community collected data into the national health information system. External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 8 CAPACITY BUILDING • Continued and tailored capacity building interventions for local organizations is required based on the assessment of performance gaps to further strengthen organizational systems for the administration of NGOs and government institutions. Successful organizations should mentor fellow organizations to build on successful local models. • Increasing performance in field operations is at the core of all service delivery and community outreach efforts. The majority of capacity building efforts with NGOs and government institutions should focus on improving the quality of field activities and systems that support them including monitoring and tracking clients, use of data to improve quality and performance; and effective and regular supervision to better support their personnel and continually improve their skills. • Continued support is required to enhance the composition and responsibilities of the newly transitioning HIV/AIDS Executive Secretariat and improve its capacity to coordinate the works of all implementing partners including the project supported local NGOs and the United Nations projects on HIV/AIDS interventions. • To enable organizations to manage and nurture their human resources, which are essential to the effective operations of health service delivery and community outreach, the project should institute specific and tailored procedures for staff performance and personnel policies that are suitable to NGOs and government institutions. • Although financial systems have improved, technical support is needed to build in internal control capacity for all project-supported NGOs and government institutions to improve the use and monitoring of their finance and budget activities, to align with financial policies and procedures, and to attract additional external funding. In general, NGOs must be supported to develop strategies to improve their resource mobilization and increase their capacity to sustain the implementation of their HIV/AIDS interventions. • Tailored capacity building strategies such as coaching and mentoring are highly effective in motivating local organizations to take ownership of their own performance improvement. These kinds of interventions are required to bring organizations up to a standard that allows them to comply with donor policies and procedures, produce rigorous technical reports, and ensure effective referrals for their clients. Focused efforts to address many of the findings of this evaluation will enhance the effectiveness of the project to ensure sustained change in local capacity and to address the needs of vulnerable populations along the Djibouti – Ethiopia transportation corridor. Key recommendations are focused on: the need for increased understanding of the epidemiological and behavioral trends; improved use of monitoring and tracking to follow clients through the cascade of HIV/AIDS services; continued efforts to improve service delivery and community organizational capacity to reach transient as well as settled local populations and to engage them in addressing the needs of the community and sharing accountability for reducing the transmission of HIV infection. The ROADS II project made progress in reducing the HIV/AIDS infections among key populations. However, to further strengthen and sustain these efforts, the implementation of the above recommendations requires the participation of all HIV/AIDS stakeholders. The dissemination of the evaluation results will serve as a platform for learning and sharing ROADS II best practices and experiences and also to discuss and adopt agreed prioritization areas and develop standardized strategies. External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 9 STRENGTHENING HIV/AIDS INTERVENTIONS IN DJIBOUTI PROJECT The U.S. President's Emergency Plan for AIDS Relief (PEPFAR) Strengthening HIV/AIDS Interventions project under the ROADS II technical assistance program was implemented specifically to support the Government of Djibouti (GoDj) to reduce HIV/AIDS transmission among long-distance truck drivers (LDTD), female sex workers (FSW), girls and young women engaging in risky sexual behavior primarily for economic gain, and other vulnerable populations living in and around transport corridor communities. The Banyan/JSI team led an evaluation of the Strengthening HIV /AIDS Interventions in Djibouti Project (ROADS II) in May and June 2016 to determine the effectiveness of this project. The evaluation assessed the progress and current effectiveness toward achieving project targets related to capacity and performance of the Ministry of Health (MOH) and its partners to manage the National Program Strategy (NPS) and to provide HIV/AIDS prevention, treatment and supportive care services. The evaluation was designed to determine which interventions are working effectively and to assess the likelihood that the project will reach its intended outcomes by July 31, 2016, including the transition of activities to local partners. To perform the evaluation, the evaluation team (ET) reviewed in depth project documentation and performance reports, interviewed stakeholders as well as clients and beneficiaries to understand both the processes and the outcomes of the project. The team interviewed 25 health providers, 7 NGO managers with along their 12 key program staff as well as 62 beneficiaries to understand the strategies, processes and the outcomes of the project. The evaluation approach was guided by a research strategy that was developed and reviewed with key project partners and funders and guided by a local Steering Committee established by the MOH and consisting of collaborating UN agencies and other partners. EVALUATION OBJECTIVES AND QUESTIONS Based on guidance established by USAID, the objectives of the evaluation were: • To describe and assess the changes in organizational capacity among NGOs, the GoDj Executive Secretariat, and the National HIV Management Program (PLS) to deliver services and manage HIV interventions; • To examine the link between improved local capacity and reduced risk of HIV/AIDS among key populations (including reduced transmission and improved health); • To determine the effectiveness of the program management/coordination and communication of the ROADS II Project. In the context of these objectives, the evaluation addressed the following main question: Types of Stakeholders Interviewed: • Health providers • NGO managers • Key program staff • Beneficiaries o Truck Drivers o Adolescents o PLWHIV o Sex Workers External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 10 1. To what extent has ROADS II’s Technical Assistance (TA) to NGOs and system in managing the NGO network, resulted in increased capacity to carry out their mandate to deliver HIV/AIDS Services targeted at vulnerable populations? 2. To what extent has ROADS II’s technical assistance (TA) to the Executive Secretariat and to the National HIV/AIDS control program increased their capacity to manage the nation’s sustainable HIV/AIDS prevention, care and support interventions for key and other vulnerable populations? 3. To what extent has the capacity strengthening resulted in reduced risk of HIV/AIDS among key and other vulnerable populations? 4. How successful has the program been in reducing HIV/AIDS transmission and improving health of the targeted beneficiaries? 5. Are the original project assumptions still valid and will they provide sufficient guidance for appropriate programmatic and technical assistance decisions? 6. What are the strengths and weaknesses of the ROADS II management, coordination and communication processes? EVALUATION METHODOLOGY AND DESIGN METHODOLOGY The evaluation applied a mixed methods strategy, drawing on existing quantitative and qualitative data and conducting primary qualitative data collection. This approach allowed evaluators to describe and assess the observable changes made in strengthening the capacity of HIV/AIDS managers and service providers and the link between capacity changes, service delivery to clients and ultimately reduced risk and transmission of HIV infections among vulnerable populations as a result of the ROADS II TA. The qualitative methods also enabled improved understanding of the perceptions of HIV/AIDS service availability, quality and effectiveness from the perspective of providers, managers and beneficiaries. EVALUATION DESIGN The ET was comprised of a Team Leader, Capacity Building Expert and two local consultants with appropriate evaluation and language skills and experience in public health program design and management in Djibouti. The team employed a mixed methods strategy, drawing on existing quantitative and qualitative data and conducted primary qualitative data collection to describe and assess the observable changes made in strengthening the capacity of HIV/AIDS managers and service providers and institutions working to reduce the risk of infection and provide services to key populations. The qualitative methods were also use to gain an understanding of the perceptions of HIV/AIDS service availability, quality and effectiveness from the perspective of providers, managers and beneficiaries. The team conducted fieldwork over two weeks in Djibouti, working closely with local stakeholders to ensure relevance of the evaluation and sharing preliminary findings before departing the country. The strategy and approach utilized for this evaluation was vetted by the JSI Institutional Review Board. FIELD DATA COLLECTION The methodology for the evaluation is comprised of various steps and tools, which are described below. It also draws on the evaluation team’s experience implementing and evaluating USAID HIV/AID programs and local capacity building initiatives. The Team Leader and the Capacity Building Technical Advisor (CBTA) worked in two teams with two local consultants to gather and review relevant documents and conduct data collection in the selected sites, as well as data analysis. The data collection External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 11 activities included document review, review of National AIDS Control Program (PLS) and ROADS II data, focus group discussions and observation, program data review, and key informant interviews (KIIs). The document review explored information contained in the project and NGO partner reports, strategic operational plans, and the national strategic plan to provide insight on service delivery and management processes. KIIs enabled evaluators to understand the perceptions and experiences of the trained managers and service providers about current HIV/AIDS program management and service delivery capacity and assess the capacity built in local organizations. Focus group discussions provided in￾depth information from service providers and beneficiaries about their experiences and impressions of the programs and services. Observations and assessments were also conducted to assess program operations, equipment/materials and program/project activities against acceptable standards. Data were extracted from key documents and recorded in a central database. PLS and project data were reviewed and analyzed with the support of the research assistant to document trends over time. The team recorded and transcribed interview transcripts and field observations daily, and held team meetings to synthesize findings from fieldwork using an iterative approach that generated additional questions and deeper probes over time. The interview protocols, questions and tools align with the areas of inquiry set out in Request for Task Order Proposal Number SOL-603-16-000005, which are reflected in the evaluation objectives listed above. The finalization of these interview guides and questions took place before fieldwork began after piloting them with representatives of each respondent group. The team adhered to ethical standards for conducting the program evaluation as well ensuring that tools and evaluation procedures took into account gender, stigma and other sensitivities. These tools will be made available for conducting follow up studies as needed. DATA COLLECTION TOOLS The team designed, piloted, and refined a set of data collection tools including adapted interview guides for each of the following key respondent groups: • ROADS II project implementing partner and sub-partners’ staff; • Project trained staff from Government institutions and local HIV/AIDS organizations (NGOs, FBO and Civil Society); • Project-trained HIV/AIDS service providers at health facilities and supervisors of outlets distributing condoms; and • PLWHIV, vulnerable population and people at risk as beneficiaries of the ROADS II project services and technical support including sex workers, truck drivers, and youth. These five data collection tools are described below and presented in Annex A. External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 12 Health Worker Knowledge and Capacity Tool 1 is a Key Informant Interview (KII) guide for facility-based health workers, developed to assess the capacity and knowledge of trained health workers related to service delivery (preventive, care and supportive interventions) and commodity management. Health Facility Assessments Tool 2 is an facility-level inventory assessment checklist used to observe and record the availability of the equipment and materials required for service delivery and to assess the management of the HIV/AIDS commodity procurement and supply (stock out, expiration) of the project partners (prime and sub-partners, the NGOs and the health facilities). Tool 3 is a PLS and ROADS II TA Project Routine Data Collection and Aggregation Form. This form was used to review quantitative routine data produced by the project and found in the project database and reports and the PLS database. Beneficiary and Health Worker Perception Tool 4 is a Focus Group Discussion guide. The evaluation team conducted five focus groups representing different beneficiaries. The purpose of the focus groups was to determine beneficiary experiences with services and community-level interventions. Focus groups were conducted with each of the following beneficiary groups: female sex workers, truck drivers, PLWHIV, and youth. The teams conducted one group discussion with each group except youth who were interviewed in Dikhil as well as Djibouti City. The focus groups included at least 7 to 10 participants identified via the in-country networks of the project’s key personnel from recommendations from USAID/Djibouti staff. The research team invited at least a dozen individuals to participate in each group to guarantee that there is a minimum of 7 respondents. The sex workers and truck drivers and PLWHIV and youth groups were separated to ensure that the team captures data on the unique perspectives of each group experience, in particular on their access to opportunities and experiences of service delivery, and community-based interventions. Local consultants or interviewers with key language skills were engaged to conduct these discussions. The research team posed some questions at the beginning of each focus group to gather data on the specific demographic characteristics of potential interest. All participants in the focus group discussion were 18 years or older, including youth respondents. All groups were homogenous by both sex and their key population type (i.e., female sex workers were not in the same FGDs as young women) except for youth and PLWHIV who requested that the group discussions involve both genders. Organizational Capacity Tool 5 is an Organizational Capacity Assessment Tool which is similar to the tool used by FHI 360 for their baseline. It was administered to project managers including the prime and sub-partners, the NGOs and the HIV/AIDS National program to explore their capacity to coordinate and oversee the HIV/AIDS service delivery. It also assessed the availability of HIV/AIDS management, supervision and training guidelines and tools. The assessment components focus on the following areas of organizational capacity: • Governance and Leadership • Finance and Administration • Management of Human Resources • Financial Management • Organizational Management • Program Management External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 13 • Project Performance Management For detailed understanding of the scoring used please see Annex B. SAMPLING EVALUATION POPULATION As noted above, the team collected the data for the evaluation through: 1) key informant interviews with the NGOs and government institution managers and key staff as well as with project trained health providers; 2) focus group discussions with beneficiaries including truck drivers, adolescents, sex workers, and PLWHIV; 3) HIV/AIDS healthcare facility observations; and 4) a desk review of project and program documents and routine data for research ethics. Key respondents. A sample of respondents was selected from different stakeholder groups to ensure adequate understanding of the progress made in strengthening the individual and organizational capacity to reduce the risk and transmission of HIV/AIDS. The evaluation reviewed sub-partner and cross border activities in this setting, conducted interviews with sub-partner representatives, key vulnerable population groups (i.e., FSWs and their partners, youth, mobile workers including truck drivers and their assistants, and low income women and community members who interact with them), and government officials, and visited the Safe Stop Resource Center, constructed by DP World. The number of respondents interviewed for the evaluation totaled 105, as shown in table 3. The number of respondents by specific category is shown in Table 4. TABLE 3: EVALUATION SAMPLE Key Respondent Group Male Respondents Female Respondents Total No. of Respondents Implementing Partner and Sub-Partner Staff 12 7 19 Government Institutions and local HIV/AIDS Organizations 1 2 3 HIV/AIDS Service Providers 11 14 25 PLWHIV, Vulnerable Populations and People at Risk 33 29 62 TOTAL 57 52 109 TABLE 4: SAMPLE OF KEY RESPONDENTS Study Population Number Trained health providers 25 Trained Managers 7 Beneficiaries Sex workers 10 Truck drivers 12 Adolescents (Djibouti and Dikhil) 32 PLWHIV 8 All the HIV/AIDS managers of the ROADS II project, sub-partners and government institutions were included in the evaluation. All the five main PLWHIV groups (Oui à la Vie, Association Arrey, Nouvel Espoir, Association Kalmo and Association Al Kamar) were reached via focus group discussion. External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 14 Key Stakeholder Interviews. Key stakeholder interview were comprised of USAID Project Managers, Government of Djibouti counterparts, and major donors supporting related projects. These interviews also targeted FHI 360 as well as representatives of program and key community groups. The team also interviewed key informants suggested by the USAID/Djibouti Project Staff and the Ministry of Health based on their exposure to the project and their position and influence within the public health and social welfare community in Djibouti. EVALUATION SITES Site Selection. The evaluation team selected the sites for the evaluation across the eight ROADS II project intervention sites that offered community and facility-based HIV prevention, care and support assistance. The evaluation focused on six sites total within Djibouti City (five sites including PK12) and Dikhil region (one site). These sites were selected based on their exposure to the ROADS II project’s technical assistance (TA) interventions, their location along the transit corridor and/or the scale of ROADS II project activities that were implemented within that particular site (Table 5). TABLE 5: EVALUATION SITES Health Facilities Partners CS PK12 FHI 360 PK 12 Safe Stop Resource Center Secrétariat Exécutif de Lutte contre le SIDA, le Palu et la TB CS Balbala 2 PLS Sante Hôpital DiKhil UNFD CS Einggela MASSABA Hôpital Dr Chakib Saad Omar ASO Arhiba RNDP+ Yonis Toussaint Khar Bourhane ETHICAL CONSIDERATIONS The evaluation protocol received approval from USAID/Djibouti and JSI’s Institutional Review Board to conduct this evaluation. All participants in the evaluation were informed about the purpose of the evaluation, that their participation was voluntary and that sharing their thoughts and perceptions about the program would not influence their access to services. They were also informed that the results of evaluation will be used to increase and strengthen the management and provision of HIV/AIDS services. The participant names were not collected and no other steps were used to identify individual respondent. Access to the evaluation databases and the paper-based completed tools and interview transcripts was limited to the evaluation team and was password protected. The paper-based filled tools were kept in locked file boxes. All members of the study team (Team Leader, CBTA, and two consultants) were trained on the protection of human subjects and the ethical conduct of research, through the online course entitled “Protecting Human Research Participants” offered by the National Institutes for Health: (https://phrp.nihtraining.com/users/login.php) LIMITATIONS The limitations associated with this evaluation report are as follows. The evaluation team chose health provider respondents from a smaller pool of individuals than expected because fieldwork for this External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 15 evaluation coincided with an external HIV/AIDS training of health providers. Thus, during the facility visits, the evaluation team was able to interview only the ROADS II-trained health providers that were available at that time. The local team conducted four additional interviews at three additional health facilities to address this shortcoming. The team increased the number of participants in focus group discussions and relied on program monitoring and reports to enhance the database for the evaluation. An additional limitation relates to recall bias associated with respondents’ reported experiences and observations. Clearly, time did not permit data collection from non-project service delivery sites to allow comparison of project sites with non-project sites. Nor was it possible to validate completely routine HMIS and program monitoring data reported here and used to highlight the changes occurring over time during the project period. KEY FINDINGS The main findings of the evaluation of the Strengthening HIV/AIDS Interventions in Djibouti Project (ROADS II) relate to 1) local organizational capacity to manage HIV programs and service delivery, focusing on the Executive Secretariat and to the National HIV/AIDS control program and the NGO network as well as health worker knowledge in the context of service delivery and beneficiary perceptions of service delivery; and 2) the links between capacity building and reduced transmission and improved health of target beneficiaries and other vulnerable populations. These findings are discussed below and described in detail in this section. Annex D summarizes key findings by evaluation question, noting the data sources from which findings have emerged. QUESTIONS 1, 2, 3: INCREASED CAPACITY TO DELIVER SERVICES AND MANAGE THE NATIONAL HIV PROGRAM AND REDUCING RISK OF HIV/AIDS. The evaluation produced findings that help explain how and why changes in organizational performance occurred as a result of the technical assistance project and suggests steps for reinforcing progress and addressing performance gaps through new program strategies. Detailed findings presented below: 1) document the levels of health worker knowledge and how this knowledge has translated into increased service delivery and beneficiary awareness related to HIV prevention and treatment; 2) document local organizational capacity in areas such as governance, administration, human resources, financial management, organizational management, program management, and program performance; and 3) provide observations about project-related influences on HIV program performance. In terms of capacity to delivery services and raise awareness of HIV prevention and accessibility of services, the number of HCT sites increased from 6 in 2013 to 9 in 2014. Health worker training related to understanding the importance of conducting HIV counseling and testing; identifying HIV positive persons and prioritizing the prevention of new infections; conducting counseling and training with clear links to the increase in the number of people accepting and receiving the HCT services; initial increases in detection of new HIV positive cases from among those tested; and an increase in the provision of ARV to HIV patients. It is not possible to link these service delivery improvements to trends in HIV case detection without additional epidemiological data. Counseling and testing efforts to raise awareness among beneficiaries were revealed in focus group discussions (FGDs) with female sex workers, adolescents, and truck drivers who consistently reported knowledge on the importance of knowing their HIV status, the use of condoms to prevent future infections and the need to support HIV positive people with services and community interventions, among others. External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 16 The organizational capacity assessment of civil society, NGOs and government institutions (Executive Secretariat, PLS) revealed positive trends in their readiness and ability to manage and implement HIV interventions. For example, trends in condom distribution were overall very positive - with condom distribution increasing over time. Service delivery sites were also equipped with basic materials for providing care and the Secretariat has introduced regular coordination and collaboration mechanisms (see below). Specific findings with respect to each organization’s internal capacity in administration and governance, human resource management, financial management, organizational and program management and program performance management are found below. Consistent progress and adequate capacity and readiness were found, such as: (1) significant improvements in governance capacity related to vision/mission, organizational structure, composition and responsibilities and legal status; and (2) operational policies, procedures and systems, systems for staff development, use of job description and recruitment scores are available and are increasingly being used appropriately. In contrast, there was widespread variation in capacity related to information systems, systems for managing staff performance and use of personnel policies. Weaknesses in human resource management partly stem from the use of government procedures rather than independent approaches human resource management that are suited to private sector and non-governmental organization. A complete transfer of skills took place related to financial management with the support of FHI 360. Organizations scored highly for most of the components under this category, though there is some variation. One organization demonstrated limited performance for internal controls and financial reporting. All organizations have low scores for audit capacity, a system that is required to receive funding directly from donor organizations. Introduction and strengthening of an audit system will increase the credibility of these organizations considerably. Organizational planning was found to be acceptable and organizations have mapped stakeholders and taken steps to work collaboratively with them. There are opportunities for FHI360 to strengthen communication among managers and staff. Finally, with respect to program management, there are systems in-place among partner organizations to ensure compliance with donor requirements however capacity for independent technical reporting varies considerably. Procedures and training with respect to gender and cultural sensitization are in￾place in all organizations Referral systems are in general poorly established as there are no referral agreements with government, private sector and civil society partners to ensure that clients that are referred to service delivery points, or have access to required treatment services. Each organization has a plan for community mobilization but they do not systematically engage communities and leaders in planning and decision making. For project performance management, organizations scored well related to performance management standards including the use of operations manuals, introduction of quality assurance practices and monitoring and evaluation plans. Use of data for program management is effective however supervision is consistently limited across organizations. QUESTION 4: REDUCING HIV TRANSMISSION AND IMPROVING HEALTH. In terms of overall performance related to reaching vulnerable populations, the evaluation found an increase in the number of HCT sites from 6 in 2013 to 9 in 2014; an increase in the awareness of HIV prevention and accessibility of services; a positive trend in the number of people receiving HIV counseling and testing (HCT) services; and an increase in the detected HIV positive cases from among those tested (23% to 41.9% from 2013 to 2014). These are all a reflection of the organizational investments made by FHI360 to build program management and service delivery capacity. There has also been an increase in the number of people on ARV from 1,082 in 2011 to 1,945 in 2015. There is also a trend in increased access to HIV counseling, testing and treatment services in 2015. For example, based on routine data collected from the PK12 Safe Stop Resource Center, which introduced services in 2015, 790 persons received HIV services, of which 383 accepted HCT and two were found to be HIV positive. Among them 72.9% were male and 17.1% were female. Among the men counseled, 48.5% accepted voluntary HIV testing. External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 17 Men represented 61.9% of the total persons accepting voluntary HCT services and receiving HIV test results. Despite these positive trends in reaching vulnerable populations through the project, 2015 data indicate that the overall proportion of positive cases detected through HCT dropped to 35.1% from 41.95 in 2014, which raises questions about project performance and patient tracking. This decline could be explained in a number of ways, including an increase in the number of HCT clients overall, successful awareness raising and behavior change at population level including increased condom use and resulting in reduced HIV transmission, or HIV positive clients who were lost to follow up, among others reasons. Additional research is required to track epidemiological trends, behavior change, and the effectiveness of the routine reporting systems to estimate the contribution of the project to population level outcomes or explain the decline in cases detected. QUESTIONS 5 AND 6: PROJECT ASSUMPTIONS AND MANAGEMENT COORDINATION. The program theory introduced during ROADS II proposed the use of technical assistance to improve the capacity of civil society, NGOs and government institutions (i.e., the Executive Secretariat and PLS) to manage and implement HIV interventions as a strategy for reaching vulnerable populations living in and around the Djibouti-Ethiopia transport corridor and to reduce HIV/AIDS transmission among them. The intent of the FHI360 program was to establish and build lasting change in local organizational capacity to address the needs of these populations. This evaluation found that targeted populations have received access to and have increased use of HIV/AIDS detection and treatment services as well as training on the adoption of safer sex behaviors as a result of project interventions, mainly health worker training and support to improved health service delivery. In addition, local organizational capacity improvement is on course toward sustained change in program governance, management and service delivery. Both government and local sub-partners’ organizational, performance and financial capacities improved in several important areas based on the ET’s assessment. Looking forward, there are several aspects of the technical assistance strategy that should remain in force and other aspects that require adjustment to improve the effectiveness of USAID’s investment in the National HIV/AIDS Control Program. With respect to program management, coordination and communication the evaluation team observed progress with respect to stakeholder coordination and technical collaboration among the key partners who are committed to improving HIV service delivery and reducing HIV transmission in vulnerable populations through local capacity building. The program has established three technical working groups to exchange, discuss and review HIV intervention progress and strategies and increase the frequency of coordination meeting (led by the Executive Secretariat) to provide opportunities for stakeholders to discuss the progress and the challenges of improving national HIV/AIDS program performance. Capacity of the PLS Sante to train and supervise the health providers and community and social workers on HCT and treatment services has improved. Going forward, the evaluation team recommends that the ES and its stakeholders focus coordination efforts at the implementation level, building stronger relations between the community implementing partners and health facilities to better support and supervise community workers, engender a shared sense of accountability and ownership among communities and health and community service providers, and feed learning from this level back into program strategy. A practical step in this direction is to ensure that all program data are integrated into HMIS reporting and that data are fed back to the community to engage community level service providers and actors in addressing gaps and building on positive performance. In addition, progress in reaching program targets and acknowledging community and health worker efforts through bulletins or newsletters aimed at the general public would increase public awareness of program strategies and progress and bolster support for the program. External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 18 Lastly, the program should invest in improved surveillance and monitoring systems to track clients and understand and use rigorous evidence around health seeking, preventive and adherence behavior. The use of a unique identifier-based tracking system for following itinerant populations has proven successful in others settings as has introducing ways for projects to map sexual and social networks to improve capacity to reach potential clients. Clearly tracking the transient populations found in the Djibouti context is more challenging than tracking settled populations. However, it will be difficult to understand the effectiveness of program strategies without better data systems and commitment to using these data routinely to refine implementation strategies for reducing HIV transmission. Other recommended actions are presented in detail in the final section of this report. HIV/AIDS SERVICE DELIVERY KNOWLEDGE COUNSELING AND TESTING PRACTICES HEALTH WORKER KNOWLEDGE ABOUT HIV SERVICES COUNSELING, TESTING AND TREATMENT. To improve service delivery, the project focused on providing equipment and health commodities, increasing condom distribution, training of health providers to increase the number of people receiving services including counseling and testing and ART, and improving community-based prevention messages. The evaluation team assessed health worker knowledge required to provide services focusing on counseling practices, steps in the counseling process, content of counseling and post counseling messages for specific groups (general population, youth, pregnant women) and patient management and care during labor, delivery, postnatal services as well as other aspects of service delivery. Overall, the ET found that health worker knowledge related to basic voluntary HIV Counseling and Testing (HCT) was adequate but found particular gaps in knowledge and practice areas that require specific attention. For example, as depicted in Figure 1, the majority of health providers interviewed clearly understand the importance of conducting HCT. The significance of identifying HIV positive persons and prioritizing the prevention of new infections was stated by 80% and 88% of health providers, respectively. In addition, the transfer of knowledge among trained counselors to clients was reflected among beneficiaries who testified to the importance of counseling and testing and the increase in the number of people accepting and receiving the HCT services (from 20% in 2013 to 46% in 2015). FIGURE 1: IMPORTANCE FOR A HEALTH PROVIDER TO DO HIV/AIDS COUNSELING AND TESTING BENEFICIARY KNOWLEDGE AND PERCEPTIONS. The ET assessed beneficiary perceptions of the program along with knowledge and awareness on HIV, HCT and program services. Based on the findings from focus group discussion, more than half of the respondents reported that unprotected sexual intercourse, sexually transmitted disease and condom misuse were among the reasons to undergo HCT services. Nearly three quarters of FGD beneficiary respondents reported that they had accepted HCT and were given their results within a short period of time along with the support and care of the health provider. However, they had initially found it difficult to accept HIV testing in the first place because of the fear of being stigmatized and discriminated by their families and communities. Table 6 provides a summary of beneficiary knowledge. As noted above, the transfer of knowledge among trained counselors was reflected by beneficiaries who testified to the importance of counseling 80 88 48 60 Identify HIV positive persons Provide HIV infection prevention Provide Treatment Provide support and care External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 19 and testing. Project M&E data also showed an increase in the number of people accepting and receiving HCT services (from 20% in 2013 to 46% in 2015) based on data found in project reports. TABLE 6: KNOWLEDGE BY BENEFICIARY GROUP GLOBAL ACHIEVEMENT PROGRESS. Since 2013, the ROADS II project has trained 130 counselors. The proportion of counselors that were trained in 2013 and 2015 is 46.2% and 44.6% respectively, as depicted in Figure 2. As a result, there has been a progressive increase in the number of people who have received HCT services which rose from 20% in 2013 to 46% in 2015 (Figure 2). FIGURE 2: TRENDS IN COUNSELORS TRAINED AND HCT SERVICES PROVIDED COUNSELING AND TESTING REQUIREMENTS HEALTH PROVIDERS KNOWLEDGE. The requirement for conducting HCT is taught as the five C’s: Consent, Confidentiality, Counseling, Correct test results, Connection to prevention treatment and care services. In contract to health worker understanding of the importance of HCT, health worker understanding of the HCT steps was limited and incomplete (Figure 3). Slightly more than 50% of health providers reported the steps of “consent, confidentiality and counseling” as part of HCT. The remaining two steps – reporting correct results and making connections to other HIV services – were mentioned by less than the half of the providers interviewed. The evaluation found gaps in knowledge related to providing the required set of counseling messages to all clients during counseling and testing. FIGURE 3: THE FIVE C’S OF HIV COUNSELING AND TESTING Beneficiaries Importance of HIV/AIDS Counseling and Testing Sex Workers • To know Health status • To prevent STIs and HIV • To be able to continue doing the activity Young adolescents • To avoid transmitting HIV to others after unprotected sexual intercourse • To Know HIV status • Change behavior Truck Drivers • Monitor health status to live longer with their little children PLWHIV • Knowing HIV status is essential for health • To avoid danger • To have a normal life • Behavior change 46.2 9.2 44.6 20 35 46 2013 2014 2015 Trained Counselors (130) People having received HCT services (26567) External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 20 BENEFICIARY PERCEPTIONS. The majority of the beneficiary groups interviewed considered the home-based HCT more confidential and discrete than the facility-based HCD; the exception was adolescents and truck drivers who perceived home-based counseling as a breach of confidentiality between them and their families. Respondents also noted a preference for offering integrated services particularly at the PK12 Safe Stop Resource Center, because they were reluctant to be seen at a center that was designated for HIV care only. Adolescents and PLWHIV favored integrated services as a way of preventing mother-to-child transmission; sex workers also saw an opportunity for providing HIV testing when they visit the health facility for STI treatment. The evaluation found that there were specific attitudes and beliefs that were unique to particular beneficiary groups. For example, truck drivers reported that, prior to marriage, individuals should be counseled and tested for HIV. Young adolescents reported that victims of rape and traffic accidents should be offered HCT services. A key beneficiary population - PLWHIV - reported that health providers were sometimes not supportive and caring during the provision of ARV. Moreover, PLWHC had only received food supplementation once following the discovery of their positive status (see Table 7). Despite the project’s efforts to increase the number of trained community workers and social workers (from 31.3% in 2013 and 49.1% in 2014), the project’s M&E data show almost no change in the percentage of the people reached through preventive messages (34.8% in 2013 and 32.7% in 2015). TABLE 7: BENEFICIARY PERCEPTIONS Beneficiaries Home based HCT HCT integrated into other services Sex Workers • Discrete • Confidential • In health facility, lack of discretion. If others see someone doing HCT, they may think the person is positive • Good as may have STI and not think about HIV Young adolescents • Opportunity during Community HIV sensitization to propose the test to individuals • Not good, as my family may know my status • PMTCT (save the child) • Good, as often have unprotected sex, Truck Drivers • Do not prefer as it is uncomfortable and my family may know my status • Better since some people are scared to go to health facilities • Remind other the availability and accessibility of the HCT service • Contribute to stopping the progress of HIV transmission PLWHIV • Discrete (Will help people hiding to be tested) • To contribute to decreasing HIV infection • For PMTCT will help to avoid child transmission • To encourage scared people, • To access treatment and psychological support and care 68 56 68 16 40 Consent Confidentiality Counseling Correct test results Connections to care, treatment and prevention services External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 21 GLOBAL ACHIEVEMENTS. Following the HCT services, health workers provide clients with condoms for safe sex. The project has distributed condoms via implementing sub partners and health facilities. Distribution increased considerably from 2013 to 2015 (Figure 4) which may have contributed to the decrease of new HIV infections from 36% to 30.6% (as a percentage of all HIV infections over three years). FIGURE 4: CONDOM DISTRIBUTION AND NEW HIV INFECTIONS TARGETS AND POST COUNSELING MESSAGES HEALTH PROVIDER KNOWLEDGE. The majority of health workers reported that all people with negative or positive HIV test results should receive post-HCT services. More than 80% of the health providers interviewed reported knowledge of key post counseling messages, including counseling clients on the risk of contracting HIV infection and the use of condom for safe sex. However, only half of the respondents stated the need for counseling on HIV testing for partners, repeating testing after 3 months for negative persons and safe feeding and infant prophylaxis for PMTCT services. Less than 10% of the health providers named advice on contraception and fertility planning as one the key messages for clients following HCT (see Figure 5). FIGURE 5: POST COUNSELING MESSAGES BENEFICIARY PERCEPTIONS. With reference to the HIV preventive messages provided to communities, the groups were asked to share the reasons for HIV testing. Experience with unprotected sexual intercourse was stated by all beneficiaries. The sex workers perceive the misuse of condoms and an untrustworthy sexual partner as reasons for undertaking HIV testing. The adolescents reported that the following groups should present for HIV testing: victims of rape and traffic accidents, people with STIs; sex workers and truck drivers. Truck drivers noted that people at risk of HIV infection include sex workers, vulnerable adolescents, immigrants, military personnel. They also reported that a person should be tested for HIV before getting married. PLWHIV highlighted physical weakness and partner HIV status as reasons for HIV testing. All beneficiary groups highlighted the need to provide moral and psychological support as well the social acceptance of HIV status and encouragement to adhere to treatment to HIV positive individuals. Table 8 presents beneficiary knowledge related to testing and post-test counseling. TABLE 8: BENEFICIARY KNOWLEDGE AND PERCEPTIONS ON HIV TESTING Beneficiaries Reasons for someone to be tested for HIV Attitudes and behavior with HIV infected persons 23.1 34.1 42.8 2013 2014 2015 Condom distributed (417126) 84 84 8 48 52 56 Risks for HIV Transmission Safe sex and use of condoms Contraception and fertility planning HIV testing for partners and children Repeat HIV testing after 3 months for negative persons PMTCT (safe feeding and infant prophylaxis) External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 22 Beneficiaries Reasons for someone to be tested for HIV Attitudes and behavior with HIV infected persons Sex Workers • Unprotected sexual intercourse • Pregnant women • For Health Check up • Condom remained inside sex after sexual intercourse • Lack of confidence of customers that pay highly • Moral and psychological support • Discretely be away from them • Advise and encourage them to take ARVs Young adolescents • Unprotected sexual intercourse • Rape victims, Traffic accidents • Persons with HIV symptoms • Close person become HIV positive • STI infected persons • Pregnant women • Misused condoms • Exposed persons (sex workers, truck drivers) • Moral and psychological support • Company and advises • Encourage Medication and to accept the disease • Isolate the HIV infected persons Truck Drivers • Know their HIV status • Bad behavior • Before getting married • STI infected persons • Exposed persons (Sex workers, vulnerable adolescents, Immigrants, Military personnel) • Moral and psychological support • Advise to stop Smoking, drinking alcohol, eating khat • Encourage medication PLWHIV • Unprotected Sexual intercourse • Physically weak • STI infected persons • Know their HIV status • Moral and psychological support • Encourage compliance to medication and to accept the disease GLOBAL ACHIEVEMENTS. The project’s defined a strategy to increase the coverage of the community￾based messages on HIV testing, care and treatment and subsequently trained community and social workers. Despite the increase of trained community and social workers, the proportion of people reached with prevention messages barely changed over the project period from 34.8% in 2013 to 32.7% in 2015 (See Figure 6). FIGURE 6: CHANGES IN TRAINED COMMUNITY AND SOCIAL WORKERS PROVIDING PREVENTIVE MESSAGES COUNSELING AND TESTING FREQUENCY HEALTH PROVIDER KNOWLEDGE. A small proportion of respondents reported knowledge on the periodicity of repeating HCT for HIV negative persons and different population categories (see Figure 7). The timeframe of 6 to 12 months for the general population, adolescents and sex workers was reported by less than half of health providers. Seventy two percent of respondents reported that 31.3 49.1 19.6 34.8 32.5 32.7 2013 2014 2015 Trained Community and Social workers (744) People reached through prevention messages ((92896) External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 23 pregnant women should be retested following HCT and 60% reported this knowledge related to breastfeeding women (see Figure 8). FIGURE 7: HEALTH PROVIDER KNOWLEDGE ON WHEN DIFFERENT POPULATIONS SHOULD REPEAT HCT FIGURE 8: HEALTH PROVIDER KNOWLEDGE WHEN A PREGNANT/BREASTFEEDING SHOULD REPEAT HCT BENEFICIARY PERCEPTIONS. The majority of the beneficiary interviewed in the FGDs that had been tested for HIV, received their results within a short period of times. Those who were accompanied by close relatives or friends received strong support and care from the health providers. All beneficiaries reported that they found it difficult to be tested for the first time due to fear of being found HIV positive and experiencing discrimination or stigmatization within their communities and families (see Table 9). TABLE 9: BENEFICIARY PERCEPTIONS OF HIV TESTING Beneficiaries Experience with the first time HIV Testing Difficulties to undergo for HIV testing Sex Workers • Had HIV test within a year • Provided results immediately or the next day • Accompanied with relatives, friends or employer • Not accompanied • Health providers very supportive • Scared to be positive and forced to abandoned the work • My relatives do not know the work and being positive, they will be isolated by the family and other people Young adolescents • Had a group HIV testing after sensitization campaign • Had individual HIV testing • Provided results immediately or the next day • One a one in the provider office • Providers very supportive • Stressed to be HIV positive • Calm because did have bad behavior • Scared but also reassured that ARV exists 60 52 64 40 48 36 General population Adolescents Sex Workers Every 3 months 6 - 12 months depending on risks 72 48 36 60 Every 3 months throughout pregnancy At Labor/delivery At the 6 weeks EPI visits Every 3 months throughout breastfeeding External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 24 Beneficiaries Experience with the first time HIV Testing Difficulties to undergo for HIV testing Truck Drivers • Some were tested more than 1 year ago and others within the year • Provided results immediately • Not accompanied • Health providers were courteous and supportive • Scared to be positive and not to get married • Scared to be positive and not to be able to work for the family feeding • Scared of what will happen after being positive PLWHIV • All were HIV tested • Provided results immediately or the next day • Health providers were very supportive and very close • To be discriminated and stigmatized GLOBAL ACHIEVEMENTS. Between 2013 and 2014, the number of positive diagnoses increased significantly from 23% to 41.9% but decreased to 35.1% in 2015. The increase has likely emerged from the increased number of HIV counseled and tested persons and increased condom use. However, additional research is required to track epidemiological trends, behavior change, and the effectiveness of the routine reporting systems before determining the reason that reported cases declined in 2015. FIGURE 9: CHANGES ON THE HIV POSITIVE CASES PK12 Safe Stop Resource Center achievements. The PK12 Safe Stop Resource Center was located near one of the crowded suburbs of Djibouti City and the main Ethiopian truck stop area to provide HIV counseling and testing to population. In 2015, among 790 persons received care at the center. Among them 72.9% were male and only 48.5% accepted the voluntary HIV testing. Men represent 61.9% of the total persons accepting the voluntary HCT services and receiving HIV test results while only 38.9% of the clients were women. The frequency of malaria services provision over the same period was low and almost all patients visiting for malaria services tested positive for malaria (see Figures 10 and 11). HIV/AIDS services FIGURE 10: PK12 SAFE STOP RESOURCE CENTER ACHIEVEMENTS IN 2015 Malaria services FIGURE 11: NUMBER OF PATIENTS RECEIVING MALARIA SERVICES IN 2015 20 35 46 23 41.9 35.1 2013 2014 2015 People having received HCT services (26567) Tested positive (1340) 72.9 29.1 61.1 38.9 48.5 Male Female Total Frequented the center (790) Received HIV test and result (383) Total Tested with results (790) External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 25 TREATMENT HEALTH PROVIDER KNOWLEDGE. Although only medical doctors were trained to provide treatment to HIV patients, more than 60% of health providers know the 3 out of the 8 requirements for pregnant and breastfeeding women management and care during labor, delivery and postnatal services. Sixty eight percent of the respondents reported knowledge on the use of iron, folate and calcium supplementation, the provision of ARV and the need for counseling on safe sex. Counseling for pregnant and lactating women on family planning and contraception was only mentioned by 24% of respondents (see Figure 12). For the treatment of the patients, all medical doctors refer to the ARVs treatment algorithm for all categories of patients displayed in the consultation room. FIGURE 12: HIV PATIENT MANAGEMENT AND CARE REQUIREMENTS AT LABOR, DELIVERY AND POST￾NATAL SERVICES BENEFICIARY PERCEPTIONS. During their visits at the health center, the PLWHIV complained that do not always receive the care and support needed from health providers. However ARVs are always available. These beneficiaries also reported that they only received food supplement one time (see Table 10). TABLE 10: PLWHIV PERCEPTIONS GLOBAL ACHIEVEMENTS The project reports that the number of people receiving ARVs increased every year since 2011 (see Table 11). TABLE 11: NUMBER OF PEOPLE RECEIVING ARV Indicator 2011 2012 2013 2014 2015 Infected HIV persons receiving ARVs 1082 1327 1427 1593 1945 AVAILABILITY OF THE HIV EQUIPMENT, PRODUCTS AND MATERIALS 28 1 27 5 0 5 Received for Malaria services With RDT Negative With RDT positive Djiboutian Non Djiboutian 68 52 60 32 44 36 56 24 Iron, folate and calcium supplementation Hemoglobin Testing Provision of ARV Diagnosis Prevention and Mgt of Opportunistic infection Counseling on infant feeding Safe sex Family planning and contraception Beneficiaries Access to treatment and other services PLWHIV • The medicine are always available • Health providers are not always supportive • Received only once food External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 26 The required products and equipment for HCT and treatment services were not fully available in all visited health facilities. Only the HIV screening test kits (KHB) and the confirmatory test kits (Stat Pack) were available in all facilities. Tie Break Test Kits were available in 8 health facilities. Preventive equipment such as containers for disinfecting instruments, containers for blood soaked waste, and gloves were available in all facilities (see Table 12). TABLE 12: NUMBER OF HEALTH FACILITIES WITH HIV/AIDS PRODUCTS AND EQUIPMENT Products and Equipment Availability • HIV Screening Test Kit (KHB) • Confirmatory Test Kit (Stat Pack) • Containers for disinfecting instruments • Conteneur pour les déchets imbibés de sang • Gloves All 9 Health Facilities • Tie Break Test Kit (Unigold) • Fixed dose combination (AZT/3TC/NVP) • Reagent for disinfection • Aprons 85 Health Facilities • Nevirapine syrup 74 Health Facilities • AZT syrup • 3 TC tablet (Lamuvidine) • Oxytocin • IV antibiotics 63 Health Facilities CAPACITY BUILDING When the ROADS II project began, FHI 360 conducted a baseline organizational assessment focusing on three components of organizational capacity: • Organizational capacity to successfully conduct internal operations • Programmatic capacity to design, plan, implement, monitor and evaluate programs • Technical capacity to deliver quality HIV/AIDS and related health services. All collaborating organizations and FHI360 itself were assessed during a baseline organizational assessment. The following data presents the results of the organizational capacity assessment conducted for this evaluation; it is organized by assessment category and compares results from each organization. For a detailed analysis by category and individual organization and scoring approach, please see Annex C. The following tables show the organization’s score (y-axis) within the various components (x-axis) constructing each category that was assessed. GOVERNANCE Overall, all the local organizations engaged in the ROADS II project have improved their understanding of vision and mission. In addition, assessment results demonstrate that FHI 360 succeeded in strengthening organizational capacities of both NGOS and government structures who reached their 0 1 2 3 4 Vision / Mission Structure org. Comp. et resp. Statut juridique Plan de relève FHI MASSABA PLSS RNDP+ ES UNFD ASO External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 27 sustainability goals in the sense that are equipped with basic documents that define and guide organizational operations. However, several organizations require more investment related to establishing effective Board of Director composition and defining Board responsibilities, securing legal status, and defining clear succession plans. ADMINISTRATION All organizations assessed scored highly with respect to Operational Policies, Procedures and Systems. In contract, there was widespread variation in capacity related to information systems. FHI 360 introduced procedures manuals (standard operating procedures manuals) but did provide similar documentation and guidance materials for policies and procedures related to basic information systems. HUMAN RESOURCE MANAGEMENT FHI 360 worked with each organization to develop job descriptions, transparent and coherent recruitment directives, and data on vacant positions, and trained the staff in counseling, financial management and monitoring and evaluation (M&E). Scores varied by administration component. In all organizations, the staff development, job description and recruitment scores are very high but staff performance management, personnel policies remain weak. Personnel skills development scores were high for all partners except PLSS and ES remain weak overall. Since many of the human resources management policies are modeled on the government’s policies, most organizations are unable to develop independent and sustainable approaches human resource management. FINANCIAL MANAGEMENT A complete transfer of skills took place related to financial management with the support of FHI 360. Organizations scored highly for most of the components under this category, though there is some variation. RNDP+ shows a lower performance for internal controls and financial reporting. All organizations have low scores for audit capacity. No institution has an audit system in place – which is often required to receive funding directly from donor organizations. 0 1 2 3 4 Politiques, procédures et systèmes opérationnels Systèmes d’information FHI MASSABA PLSS RNDP+ ES UNFD ASO 0 1 2 3 4 Fiches de poste Recrutement Niveau et rétention du pers. Pol. d’emploi Gestion de la perf. du pers. Dvpt. des compét. du pers. FHI MASSABA PLSS RNDP+ ES UNFD ASO 0 1 2 3 4 FHI MASSABA PLSS RNDP+ ES UNFD ASO External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 28 ORGANIZATIONAL MANAGEMENT FHI 360 trained staff on best practices in organizational planning. However, processes are lacking to ensure that staff can continue to learn and advance these skills. FHI 360 has assisted these organizations in mapping expectations of their stakeholders and takes steps to work collaboratively with them. There are opportunities to strengthen communication among managers and staff. Note that these indicators, represent capacity areas that are needed to ensure organizational sustainability including (resource mobilization and change management). PROGRAM MANAGEMENT There are systems in place among partner organizations to ensure compliance with donor requirements, however capacity for independent technical reporting varies considerably among organizations. FHI 360 submits reports to donors on behalf of these partners. In terms of managing referral services, there are established reference sites such as in PK12. However FHI 360 has not defined referral agreements with government, private sector and civil society partners to ensure that clients that are referred to service delivery points and have access to required treatment or other health or support services. Each organization has a plan for community mobilization but they do not systematically engage communities and leaders in planning and decision making. Procedures and training with respect to gender and cultural sensitization are in-place in all organizations. The components that require additional investment include: donor compliance, technical reporting and in some instances referral systems. PROJECT PERFORMANCE MANAGEMENT Organizations scored well related to performance management standards including the use of operations manuals, introduction of quality assurance practices and monitoring and evaluation plans. FHI 360 has provided some guidance on conducting supportive supervision but the majority of organizations lacked effective and consistent supervisory capacity. CONCLUSIONS & RECOMMENDATIONS The evaluation of the Strengthening HIV /AIDS Interventions in Djibouti Project (ROADS II) revealed many positive changes in the health service delivery performance and the institutional capacity of the 0 1 2 3 4 Plan. Strat. Plan op. Mobil. des ress. Strat. comm Gestion du changt. Gestion des conn. Invent. des pp Comm. Int Prise de déc. FHI MASSABA PLSS RNDP+ ES UNFD ASO 0 1 2 3 4 Comp. donateurs Rapports tech Orientation Particip. commun. Culture, genre et handicap FHI MASSABA PLSS RNDP+ ES UNFD ASO 0 1 2 3 4 Suivi des act. de terrain Normes Assur quali. Supervision S&E FHI MASSABA PLSS RNDP+ ES UNFD ASO External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 29 NGOs (FHI 360, UNFD, RNDP+, ASO, Massaba) and Government Institutions (Executive Secretariat, PLS Sante) that are addressing the health needs of populations living and working along the Ethiopia￾Djibouti transportation corridor. Notably, the knowledge of health providers related to HCT and service delivery practices is growing. Moreover, based on beneficiary focus group findings, health workers counseling and outreach appears to have contributed to improved awareness among the target population about HIV infection, preventive behaviors and care and treatment services. Indeed the acceptability of HCT services has increased considerably among beneficiaries, which was reflected in the increased number of condoms distributed and increases in the number of clients receiving HCT. In comparison to the results of the midterm evaluation conducted by the project, the findings of this evaluation indicate that the overall capacity of local institutions appears to have improved consistently over time. The increased capacity to manage HIV interventions resulted in improved performance in overall coordination of service delivery and monitoring of services provided at the community level. Despite this progress, however, this evaluation revealed the need for improvement in several aspects of organizational capacity, mainly in internal control/auditing, field activity monitoring, quality assurance and succession planning. To further strengthen and improve the project’s work to reduce the transmission of HIV, the ET recommends that the project take the actions described below. SERVICE DELIVERY • Positive trends on increasing access to essential HIV counseling care and treatment should be reinforced with additional investment in service delivery. The project should increase the number of health providers offering HIV treatment and care by training nurses and midwives and updating protocols to allow these cadres perform treatment services. • Based on the fact that the population reached through project messages did not increase (from 34.8% in 2013 to 32.7% in 2015), it is recommended that efforts should be made to increase the knowledge of health providers on HCT services. This includes the repetition of HCT services in different population groups, the requirements for HIV patient management and care, and the understanding of the different steps of HCT to re-enforce the already successful efforts to raise awareness of the population related to HIV prevention, care and treatment • Resource centers currently provide mainly HIV services and few other healthcare services and support. As a result, communities see the centers as mainly treating HIV positive patients. It is recommended that the program integrate HIV services into other health and community support services in the PK12 Safe Stop Resource Center to increase the use of services and to eliminate the stigmatization around users of the center services. • In group discussions, some beneficiaries expressed the belief that HIV positive persons should be isolated from their families and communities to prevent the spread of HIV. Therefore, the project should take steps to improve the understanding among beneficiaries of steps for HIV prevention and encourage positive attitudes towards PLWHA. • In HIV programs, a key strategy is to closely monitor infected persons to help them to live healthy life styles and to prevent the transmission of the virus to others. Since the project support is limited to prevention, the Ministry of Health should introduce more rigorous and high quality treatment services and follow-up strategies, particularly to monitor HIV positive cases and address the decline in positive cases (from 49.1% in 2014 to 35.1% in 2015), which may be due to loss during follow-up or death. • Information, monitoring and evaluation systems overall require strengthening among partner institutions as does the use of data to guide program strategy. Reporting to the national health information system from local partners is limited and limits government capacity to track External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 30 national program efforts, identify gaps and report on successes. The program should invest in improved surveillance and monitoring systems to track clients and understand and use rigorous evidence around health seeking, preventive and adherence behavior. The use of a unique identifier-based tracking system for following itinerant populations has proven successful in others settings as has introducing ways for projects to map sexual and social networks to improve capacity to reach potential clients. • Links between health facilities and community groups are limited which is a missed opportunity to bolster program effectiveness and engendering shared accountability for program performance. The project should establish a liaison between the implementing local NGOs and FBOs and health facilities to enable supportive supervision and integration of community collected data into the national health information system. CAPACITY BUILDING • Continued and tailored capacity building interventions for local organizations is required based on the assessment of performance gaps to further strengthen organizational systems for the administration of NGOs and government institutions. Successful organizations should mentor fellow organizations to build on successful local models. • Increasing performance in field operations is at the core of all service delivery and community outreach efforts. The majority of capacity building efforts with NGOs and government institutions should focus on improving the quality of field activities and systems that support them including monitoring and tracking clients, use of data to improve quality and performance; and effective and regular supervision to better support their personnel and continually improve their skills. • Continued support is required to enhance the composition and responsibilities of the newly transitioning HIV/AIDS Executive Secretariat and improve its capacity to coordinate the works of all implementing partners including the project supported local NGOs and the United Nations projects on HIV/AIDS interventions. • To enable organizations to manage and nurture their human resources, which are essential to the effective operations of health service delivery and community outreach, the project should institute specific and tailored procedures for staff performance and personnel policies that are suitable to NGOs and government institutions. • Although financial systems have improved, technical support is needed to build in internal control capacity for all project-supported NGOs and government institutions to improve the use and monitoring of their finance and budget activities, to align with financial policies and procedures, and to attract additional external funding. In general, NGOs must be supported to develop strategies to improve their resource mobilization and increase their capacity to sustain the implementation of their HIV/AIDS interventions. • Tailored capacity building strategies such as coaching and mentoring are highly effective in motivating local organizations to take ownership of their own performance improvement. These kinds of interventions are required to bring organizations up to a standard that allows them to comply with donor policies and procedures, produce rigorous technical reports, and ensure effective referrals for their clients. Focused efforts to address many of the findings of this evaluation will enhance the effectiveness of the project to ensure sustained change in local capacity and to address the needs of vulnerable populations along the Djibouti – Ethiopia transportation corridor. Key recommendations are focused on: the need for increased understanding of the epidemiological and behavioral trends; improved use of monitoring and tracking to follow clients through the cascade of HIV/AIDS services; continued efforts to improve service delivery and community organizational capacity to reach transient as well as settled local External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 31 populations and to engage them in addressing the needs of the community and sharing accountability for reducing the transmission of HIV infection. The ROADS II project made progress in reducing the HIV/AIDS infections among key populations. However, to further strengthen and sustain these efforts, the implementation of the above recommendations requires the participation of all HIV/AIDS stakeholders. The dissemination of the evaluation results will serve as a platform for learning and sharing ROADS II best practices and experiences and also to discuss and adopt agreed prioritization areas and develop standardized strategies. External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 32 ANNEX A: DATA COLLECTION TOOLS External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 33 HEALTH FACILITY KII INTERVIEW GUIDE Number: Location: --------------------------------------------------------------- Partner organization:…………………………………………………………………. Hello, I am {NAME} here to discuss the (LOCAL NAME OF PROJECT) supported by USAID/Djibouti. We are here to understand your work related to HIV testing, treatment and prevention {IF NECESSARY, ADD DETAILS}. By listening to people who are working within the health services and with clients around HIV care, treatment, and prevention, we hope to learn about areas where you have been successful and discuss opportunities for improvement. You have been selected a health provider from this clinic and we would very much like to hear about your opinions and experiences. Before we get started, I would like to explain to you how the interview works. Informed consent Your participation in the interview and in every aspect of the study is completely voluntary. If some questions are difficult or make you uncomfortable, we can skip them. You may also ask me to clarify any questions if you do not understand them. You may also decide to stop the interview at any time. All of the information that you provide for the study will be kept completely confidential. We record your responses, but the questionnaire will not have your name on it, and your responses to our questions are identified only by a number, never by name. The study will not be used to evaluate this facility or to report what the providers tell us to those in the administration. If you have questions or concerns after we are finished, you may contact {CONTACT PERSON AT THE COMMUNITY LEVEL}. Although there are no direct benefits to you from participating in this survey, we hope that the survey will help to improve health services and support for people living with HIV and AIDS. Do you have any questions about the survey? Would you be willing to participate in our study? Respondent Title: ……………………………………………………………………… Respondent Gender: …………………………………………………………………. Respondent Age: ……………………………………………………………………….. Respondent number of Years of Experience: ……………………………. Date of Interview: ………………………………………… Questions Answers Observation Counseling and Testing 1. Why is it important for a service provider to do HIV/AIDS Counseling and Testing? 1. Identify HIV positive persons 2. Provide HIV infection prevention 3. Provide Treatment 4. Provide support and care 5. Other…………………….. 2. Why is it important for a service provider to encourage individuals/couples to go for HIV Counseling and Testing? 1. Know their HIV status 2. Supports positive living 3. Supports healthy lifestyles 4. Supports good nutrition 5. Other……………………………… 3. What are the circumstances where a patient may be tested HIV as part of a clinical care and screening? 1. Sexual assault 2. Domestic violence 3. Abandoned babies 4. Response to court order 5. Other…………………… 4. What are the five C’s that all for forms of HCT should be voluntary and adhere? 1. Consent 2. Confidentiality 3. Counseling 4. Correct test results External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 34 Questions Answers Observation 5. Connections to care, treatment and prevention services 5. With regard to the HIV/AIDS Test results, who must receive post counseling? 1. HIV positive persons 2. HIV Negative persons 3. All tested persons 4. None 5. Other 6. What must be included in the post counseling messages? 1. Risks for HIV Transmission 2. Safe sex and use of condoms 3. Contraception and fertility planning 4. HIV testing for partners and children 5. Repeat HIV testing after 3 months for negative persons 6. PMTCT (safe feeding and infant prophylaxis) 7. When do we repeat HIV counseling and testing for pregnant/breastfeeding women? 1. Every 3 months throughout pregnancy 2. At Labor/delivery 3. At the 6 weeks EPI visits 4. Every 3 months throughout breastfeeding 5. Other…………………… 8. When do we repeat HIV counseling and testing for the general population? 1. 6 – 12 months depending on risk 2. Other……………….. 9. When do we repeat HIV counseling and testing for adolescent? 1. 6 – 12 months if sexually active 2. Frequent new sexual partners 3. Not using barrier protection 4. Other…………………… 10. When do we repeat HIV counseling and testing for adult and ado exposed to HIV 1. After 6-12 weeks for window period 2. Other………………………………. 11. When do we repeat HIV counseling and testing for Female Sex workers? 1. Every 6 – 12 months 2. Other……………………….. Antiretroviral Therapy 1. What are the management and care requirements during labor, delivery and postnatal phases? 1. Iron, folate and calcium supplementation 2. Hemoglobin testing 3. Provision of ARV 4. Diagnosis 5. Prevention and management of opportunistic infections 6. Counseling on infant feeding 7. Safer sex 8. Family planning and contraception 9. Other…………… 2. When to start eligibility criteria for ART in pregnant and breastfeeding women? 1. On the same day of positive diagnosis regardless of CD4 count 2. Tested positive during labor 3. Diagnose positive within first year postpartum regardless CD4 count or infant feeding practice 4. Other…………………….. External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 35 Indicate the number of pills that a patient has to take each day for each of these medications at the first line Regimens. Category of patient Name of Antiretroviral medication Number of antiretroviral pills Wake up Breakfast, morning Lunch, Afternoon Supper, Evening, Bed Time Pregnant (ANC 1) and breastfeeding women not on ART Pregnant women (ANC 1) on ART Pregnant women (ANC2) with creatinine ≤85µmol/l and any CD4 cell count Pregnant women (ANC2) with creatinine ≥85µmol/l and TDF Contraindicated Pregnant women with contraindication to EFV Unbooked woman presents in labor and tests HIV positive or with emergency caesarean with no ART Mothers diagnosed with HIV within 1 year post-partum or still breastfeeding beyond 1 year Adolescents with weight <40kg or age <15 years Adolescents with weight ≥40 kg and age ≥15 years External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 36 COLLECTION OF HMIS AND ROADS II TA PROJECT ROUTINE DATA FORM Indicators 2011 2012 2013 2014 2015 Morbidity and Mortality Number of new infection cases Number of HIV/AIDS death Number of infected persons receiving ARV drugs Prevention and care Number of HIV loss of follow up cases Number of people reached with the prevention messages Number of condoms distributed Capacity Building Number of trained health workers Number of trained community and para social workers Number of trained care-givers Number of managers trained Number of counselors trained Number of cluster members trained Coordination Number of active national TWGs committees (Gender TWG, MARPs-Key pop TWG, M&E TWG) Number of quarterly coordination meetings convened by the Executive Secretariat Number of HTC sites established Number of Individuals receiving HTC services and their test results Total number of clients testing positive External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 37 INVENTORY ASSESSMENT FACILITY LEVEL Number of Questionnaire: ……………………………………… Location: …………………………………………………………….. Health Facility: …………………………………………………… Date of Observation: …………………………………………………. Personnel and Service Availability How many care providers worked in the Health Facility? #:_______ How many of these care providers were trained in HIV VCT? # _______ How many of these care providers were trained in HIV Treatment, Management and Care? # _______ Drugs and Materials in L&D Unit (Write NA if not applicable) Available Expired Stock out HIV Screening Test Kit (KHB) Confirmatory Test Kit (Stat Pack) Tie Break Test Kit (Unigold) Nevirapine tablet Nevirapine syrup AZT (Zidovudine) tablet AZT syrup 3 TC tablet (Lamuvidine) Fixed dose combination (AZT/3TC/NVP) Oxytocin IV antibiotics Parenteral anticonvulsants Reagent for disinfection Help baby for breath set / HBB set / Containers for disinfecting instruments Container for blood soaked waste Is the L&D unit out of stock of any medical supplies needed for safe delivery? Gloves Goggles Aprons delivery sets sharp boxes Functional vacuum to do instrumental delivery Are there any instruments needed for neonatal resuscitation that are missing? suction machine ambu-bags Available Expired Stock out External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 38 Heater Lamp TTC eye ointment vitamin K ARV Prophylaxis Dosing Chart is posted Referral slips Additional copy of L&D Register Condition of L&D Room Service Delivery YES NO NA Review records of 2 most recent clients. Did they have counseling and testing? Did they have a partograph done? Did they receive oxytocin? Neonatal care Review records of 2 most recent L&D clients who were HIV-positive Did they receive Family planning and infant feeding counseling, referrals to ART and pediatric PMTCT follow-up? Data Quality Control Check for last month of Data Reported Month of Data: _______ Indicator (Data Source) # Recorded in Register # Reported in Monthly Summary Form Mark X if different # of women who delivered # of HIV-positive women who delivered # of newborns born to HIV+ mothers who received ARV prophylaxis # of infants born to HIV positive mothers that are on OI prophylaxis # of children born to HIV positive mothers and are tested at 18 months External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 39 FOCUS GROUP GUIDE: PLWHIV Number: Location: --------------------------------------------------------------- Group: ……………………………………………………………………………………………………. Hello, I am {NAME} here to discuss the (LOCAL NAME OF PROJECT) supported by USAID/Djibouti. We are here to understand any experience you might have had related to health or community services, particularly HIV testing, treatment and prevention {IF NECESSARY, ADD DETAILS} in this community. We hope to learn more about your experience by listening to you and others people who have taken part in activities or received services from different providers or organizations. You have been selected a group from beneficiaries of the ROADS II Project and we would very much like to hear about your opinions and experiences. Before we get started, I would like to explain to you how the Group discussion works. Informed consent Your participation in the group discussion and in every aspect of the assessment is completely voluntary. If some questions are difficult or make you uncomfortable, we can skip them. You may also ask me to clarify any questions if you do not understand them. You may also decide to stop the group discussion at any time. All of the information that you provide for the study will be kept completely confidential. We record your responses, but the guide will not have your name on it, and your responses to our questions are identified only by a number, never by name. The study will not be used to evaluate this group or to report what you tell us to those in the administration. Please respect the privacy of the others also participating in today’s discussion and keep all information discussed here today confidential. If you have questions or concerns after we are finished, you may contact {CONTACT PERSON AT THE COMMUNITY LEVEL}. Although there are no direct benefits to you from participating in this survey, we hope that the survey will help to improve health services and support for people living with HIV and AIDS. Do you have any questions about the survey? Would you be willing to participate in our study? Number of Participants: …………………. Number of Male participant: ………….. Number of Female participant: …………….. Date and Time: …………………………… I. General knowledge and attitudes about testing Now, if you don’t mind, I would like to ask you some general questions about what you think of HIV testing. External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 40 1. Why is it important for people to know their HIV status? 2. What thoughts are about the practice of offering people HIV testing and counselling at a health care facility, when they came for something else [For PMTCT: when they came for ante-natal care?] 3. What thoughts for the practice of offering people HIV tests in their homes? 4. Is it okay to require certain people to have an HIV test? 5. What types of people should be required to have an HIV test and why? 6. If someone has HIV, what can be done to help them stay healthy and live longer? II. Counseling and Testing Now I would like to ask you some questions about getting tested for HIV 1. In which circumstances a person often request to be tested for HIV? 2. Why is it hard for someone to be tested for the first time? 3. How long for someone in the general population tested negative should renew his HIV test? 4. What are the main reasons to go for an HIV test? 5. Should a person be accompanied at the facility to get tested? 6. What will happen and how he should feel about it? 7. Before taking a blood/oral sample for testing, what should do a health care provider or counsellor? 8. How long after the test do they receive the results? An approximate guess is fine 9. Is it important for the tested to be accompanied to the facility when receiving test results? 10. Any suggestions for how to make it easier for women and men to find out their HIV status? 11. Any suggestions for how to help people who are HIV-positive get the medical care they need to stay healthy and live longer? III. Disclosure to others, HIV-positive (for HIV-positive respondents only) Now if it is okay with you, I would like to ask you some questions about whether and how you have shared your HIV status with people around you. 1. Is it better for tested individuals to keep their HIV status a secret from most people? External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 41 2. Is it good to share the HIV test results with anyone outside the health clinic? 3. With whom should some share his HIV test results? 4. After getting his HIV test results, what type of assistance the health care provider should do to help the individual to share his HIV status with people around him? IV. Experience with stigma and discrimination Now, if you don't mind, I would like to ask you some other questions about your personal experiences. 1. Have anyone of you ever been made to feel bad because of things people did or because of your HIV status? 2. Can you tell us what had happened? 3. In the past twelve months, have anyone of you ever found himself avoiding or isolating himself from friends or family because of your HIV status? 4. Next I would like to ask you some questions about how anyone of you are treated when you go to health care facilities and whether you think you have ever been treated differently than other people because of your HIV status. 5. What have been the most difficult things that have happened in your life since you found out your HIV status? V. Support 1. Since you received your HIV-positive result, have you received any financial assistance, food assistance, or emotional and social assistance from the government or any HIV or AIDS support organization? 2. Do you think that knowing your HIV-positive status has been good for you overall? VI. Access to Treatment and Food security 1. In the past year, have you ever had problems getting your ART on time because you were not able to reach the clinic? 2. During the past year, has the cost of medication or the cost of clinic care ever interfered with your ability to get your ART and take your medication on time? 3. During the past one month, have you ever missed a dose of your ART because you did not have enough food? 4. Now that you are taking ART, how is your health? Would you say it is: 5. Have you experienced any side-effects since you started taking ART? External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 42 FOCUS GROUP GUIDE: SEX WORKERS Number: Location: --------------------------------------------------------------- Group: ……………………………………………………………………………………………………. Hello, I am {NAME} here to discuss the (LOCAL NAME OF PROJECT) supported by USAID/Djibouti. We are here to understand any experience you might have had related to health or community services, particularly HIV testing, treatment and prevention {IF NECESSARY, ADD DETAILS} in this community. We hope to learn more about your experience by listening to you and others people who have taken part in activities or received services from different providers or organizations. You have been selected a group from beneficiaries of the ROADS II Project and we would very much like to hear about your opinions and experiences. Before we get started, I would like to explain to you how the Group discussion works. Informed consent Your participation in the group discussion and in every aspect of the assessment is completely voluntary. If some questions are difficult or make you uncomfortable, we can skip them. You may also ask me to clarify any questions if you do not understand them. You may also decide to stop the group discussion at any time. All of the information that you provide for the study will be kept completely confidential. We record your responses, but the guide will not have your name on it, and your responses to our questions are identified only by a number, never by name. The study will not be used to evaluate this group or to report what you tell us to those in the administration. Please respect the privacy of the others also participating in today’s discussion and keep all information discussed here today confidential. If you have questions or concerns after we are finished, you may contact {CONTACT PERSON AT THE COMMUNITY LEVEL}. Although there are no direct benefits to you from participating in this survey, we hope that the survey will help to improve health services and support for people living with HIV and AIDS. Do you have any questions about the survey? Would you be willing to participate in our study? Number of Participants: …………………. Date and Time: …………………………… I. General knowledge and attitudes about testing Now, if you don’t mind, I would like to ask you some general questions about what you think of HIV testing. 1. Why is it important for people to know their HIV status? External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 43 2. What thoughts are about the practice of offering people HIV testing and counselling at a health care facility, when they came for something else [For PMTCT: when they came for ante-natal care?] 3. What thoughts for the practice of offering people HIV tests in their homes? 4. Is it okay to require certain people to have an HIV test? 5. What types of people should be required to have an HIV test and why? 6. If someone has HIV, what can be done to help them stay healthy and live longer? II. Counseling and Testing Now I would like to ask you some questions about getting tested for HIV 1. In which circumstances a person often request to be tested for HIV? 2. Why is it hard for someone to be tested for the first time? 3. How long for someone in the general population tested negative should renew his HIV test? 4. What are the main reasons to go for an HIV test? 5. Should a person be accompanied at the facility to get tested? 6. What will happen and how he should feel about it? 7. Before taking a blood/oral sample for testing, what should do a health care provider or counsellor? 8. How long after the test do they receive the results? An approximate guess is fine 9. Is it important for the tested to be accompanied to the facility when receiving test results? 10. Any suggestions for how to make it easier for women and men to find out their HIV status? 11. Any suggestions for how to help people who are HIV-positive get the medical care they need to stay healthy and live longer? III. Disclosure to others, HIV-positive (for HIV-positive respondents only) Now if it is okay with you, I would like to ask you some questions about whether and how you have shared your HIV status with people around you. 1. Is it better for tested individuals to keep their HIV status a secret from most people? 2. Is it good to share the HIV test results with anyone outside the health clinic? 3. With whom should some share his HIV test results? External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 44 4. After getting his HIV test results, what type of assistance the health care provider should do to help the individual to share his HIV status with people around him? IV. Experience with stigma and discrimination Now, if you don't mind, I would like to ask you some other questions about your personal experiences. 1. Have anyone of you ever been made to feel bad because of things people did or because of your work? 2. Can you tell us what had happened? 3. In the past twelve months, have anyone of you ever found avoided or isolated from friends or family because being considered as HIV positive? 4. Next I would like to ask you some questions about how anyone of you is treated when you go to health care facilities and whether you think you have ever been treated differently than other people because of your work. 5. What is the most difficult things in the life if someone found out his/her HIV status? V. Support 1. Do you think HIV-positive should receive any financial assistance, food assistance, or emotional and social assistance from the government or any HIV or AIDS support organization? 2. Do you think that knowing the HIV-positive status is good for any individual? VI. Access to Prevention 1. In the past year, have you ever had problems getting your condoms on time because you were not able to reach the clinic? 2. During the past year, has the cost of other medication or the cost of clinic care ever interfered with your ability to get your condoms? 3. During the past one month, have you ever not had sex because you did not have condoms? 4. Have you ever requested HIV testing and you did not receive it. Can you tell us why? External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 45 FOCUS GROUP GUIDE: TRUCK DRIVERS Number: Location: --------------------------------------------------------------- Group: ……………………………………………………………………………………………………. Hello, I am {NAME} here to discuss the (LOCAL NAME OF PROJECT) supported by USAID/Djibouti. We are here to understand any experience you might have had related to health or community services, particularly HIV testing, treatment and prevention {IF NECESSARY, ADD DETAILS} in this community. We hope to learn more about your experience by listening to you and others people who have taken part in activities or received services from different providers or organizations. You have been selected a group from beneficiaries of the ROADS II Project and we would very much like to hear about your opinions and experiences. Before we get started, I would like to explain to you how the Group discussion works. Informed consent Your participation in the group discussion and in every aspect of the assessment is completely voluntary. If some questions are difficult or make you uncomfortable, we can skip them. You may also ask me to clarify any questions if you do not understand them. You may also decide to stop the group discussion at any time. All of the information that you provide for the study will be kept completely confidential. We record your responses, but the guide will not have your name on it, and your responses to our questions are identified only by a number, never by name. The study will not be used to evaluate this group or to report what you tell us to those in the administration. Please respect the privacy of the others also participating in today’s discussion and keep all information discussed here today confidential. If you have questions or concerns after we are finished, you may contact {CONTACT PERSON AT THE COMMUNITY LEVEL}. Although there are no direct benefits to you from participating in this survey, we hope that the survey will help to improve health services and support for people living with HIV and AIDS. Do you have any questions about the survey? Would you be willing to participate in our study? Number of Participants: …………………. Date and Time: …………………………… I. General knowledge and attitudes about testing Now, if you don’t mind, I would like to ask you some general questions about what you think of HIV testing. 1. Why is it important for people to know their HIV status? External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 46 2. What thoughts are about the practice of offering people HIV testing and counselling at a health care facility, when they came for something else [For PMTCT: when they came for ante-natal care?] 3. What thoughts for the practice of offering people HIV tests in their homes? 4. Is it okay to require certain people to have an HIV test? 5. What types of people should be required to have an HIV test and why? 6. If someone has HIV, what can be done to help them stay healthy and live longer? II. Counseling and Testing Now I would like to ask you some questions about getting tested for HIV 1. In which circumstances a person often request to be tested for HIV? 2. Why is it hard for someone to be tested for the first time? 3. How long for someone in the general population tested negative should renew his HIV test? 4. What are the main reasons to go for an HIV test? 5. Should a person be accompanied at the facility to get tested? 6. What will happen and how he should feel about it? 7. Before taking a blood/oral sample for testing, what should do a health care provider or counsellor? 8. How long after the test do they receive the results? An approximate guess is fine 9. Is it important for the tested to be accompanied to the facility when receiving test results? 10. Any suggestions for how to make it easier for women and men to find out their HIV status? 11. Any suggestions for how to help people who are HIV-positive get the medical care they need to stay healthy and live longer? III. Disclosure to others, HIV-positive (for HIV-positive respondents only) Now if it is okay with you, I would like to ask you some questions about whether and how you have shared your HIV status with people around you. 1. Is it better for tested individuals to keep their HIV status a secret from most people? 2. Is it good to share the HIV test results with anyone outside the health clinic? 3. With whom should some share his HIV test results? External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 47 4. After getting his HIV test results, what type of assistance the health care provider should do to help the individual to share his HIV status with people around him? 6. What is the most difficult things in the life if someone found out his/her HIV status? IV. Support 1. Do you think HIV-positive should receive any financial assistance, food assistance, or emotional and social assistance from the government or any HIV or AIDS support organization? 2. Do you think that knowing the HIV-positive status is good for any individual? V. Access to Prevention 1. In the past year, have you ever had problems getting your condoms on time because you were not able to reach the clinic? 2. During the past year, has the cost of other medication or the cost of clinic care ever interfered with your ability to get your condoms? 3. During the past one month, have you ever not had sex because you did not have condoms? 4. Have you ever requested HIV testing and you did not receive it. Can you tell us why? External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 48 FOCUS GROUP GUIDE: YOUTH Number: Location: --------------------------------------------------------------- Group: ……………………………………………………………………………………………………. Hello, I am {NAME} here to discuss the (LOCAL NAME OF PROJECT) supported by USAID/Djibouti. We are here to understand any experience you might have had related to health or community services, particularly HIV testing, treatment and prevention {IF NECESSARY, ADD DETAILS} in this community. We hope to learn more about your experience by listening to you and others people who have taken part in activities or received services from different providers or organizations. You have been selected a group from beneficiaries of the ROADS II Project and we would very much like to hear about your opinions and experiences. Before we get started, I would like to explain to you how the Group discussion works. Informed consent Your participation in the group discussion and in every aspect of the assessment is completely voluntary. If some questions are difficult or make you uncomfortable, we can skip them. You may also ask me to clarify any questions if you do not understand them. You may also decide to stop the group discussion at any time. All of the information that you provide for the study will be kept completely confidential. We record your responses, but the guide will not have your name on it, and your responses to our questions are identified only by a number, never by name. The study will not be used to evaluate this group or to report what you tell us to those in the administration. Please respect the privacy of the others also participating in today’s discussion and keep all information discussed here today confidential. If you have questions or concerns after we are finished, you may contact {CONTACT PERSON AT THE COMMUNITY LEVEL}. Although there are no direct benefits to you from participating in this survey, we hope that the survey will help to improve health services and support for people living with HIV and AIDS. Do you have any questions about the survey? Would you be willing to participate in our study? Number of Participants: …………………. Number of Male participant: ………….. Number of Female participant: …………….. Date and Time: …………………………… I. General knowledge and attitudes about testing Now, if you don’t mind, I would like to ask you some general questions about what you think of HIV testing. External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 49 1. Why is it important for people to know their HIV status? 2. What thoughts are about the practice of offering people HIV testing and counselling at a health care facility, when they came for something else [For PMTCT: when they came for ante-natal care?] 3. What thoughts for the practice of offering people HIV tests in their homes? 4. Is it okay to require certain people to have an HIV test? 5. What types of people should be required to have an HIV test and why? 6. If someone has HIV, what can be done to help them stay healthy and live longer? II. Counseling and Testing Now I would like to ask you some questions about getting tested for HIV 1. In which circumstances a person often request to be tested for HIV? 2. Why is it hard for someone to be tested for the first time? 3. How long for someone in the general population tested negative should renew his HIV test? 4. What are the main reasons to go for an HIV test? 5. Should a person be accompanied at the facility to get tested? 6. What will happen and how he should feel about it? 7. Before taking a blood/oral sample for testing, what should do a health care provider or counsellor? 8. How long after the test do they receive the results? An approximate guess is fine 9. Is it important for the tested to be accompanied to the facility when receiving test results? 10. Any suggestions for how to make it easier for women and men to find out their HIV status? 11. Any suggestions for how to help people who are HIV-positive get the medical care they need to stay healthy and live longer? III. Disclosure to others, HIV-positive (for HIV-positive respondents only) Now if it is okay with you, I would like to ask you some questions about whether and how you have shared your HIV status with people around you. External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 50 PROJET DE FORMULAIRE DE DONNÉES EN ROUTINE DU SIGS ET DU PROJET D'AT DE ROADS II Location : --------------------------------------------------------------- Partner Organization:.............................................................................................. Indicators 2010 2011 2012 2013 2014 Morbidity and Mortality Number of new infection cases Number of HIV/AIDS death Number of infected persons receiving ARV drugs Prevention and care Number of HIV loss of follow up cases Number of people reached with the prevention messages Number of condoms distributed Capacity Building Number of trained health workers Number of trained community and para social workers Number of trained care-givers Number of managers trained Number of counselors trained Number of PLHIV cluster members trained Coordination Number of active national TWGs committees (Gender TWG, MARPs￾Key pop TWG, M&E TWG) Number of quarterly coordination meetings convened by the Executive Secretariat Number of HTC sites established Number of persons provided HCT services and their HIV test results Number of persons tested HIV positive Is it better for tested individuals to keep their HIV status a secret from most people? External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 51 1. Is it good to share the HIV test results with anyone outside the health clinic? 2. With whom should some share his HIV test results? 3. After getting his HIV test results, what type of assistance the health care provider should do to help the individual to share his HIV status with people around him? I. Thoughts about stigma and discrimination Now, if you don't mind, I would like to ask you some other questions about your personal experiences. 1. What should be the behavior of the general population with the individuals infected by the virus of HIV? 2. Can you tell us what had happened? 3. How a person should feel if he is being avoided or isolated from friends or family because of his HIV status? 4. How anyone who is HIV positive should treated at the health care facilities and have you seen a HIV positive individual been treated differently than other people because of your HIV status. 5. Have you any idea what should be the most difficult things that could happen in the life of someone who found his HIV status? II. Support 1. What are the consequences of giving HIV positive individuals financial assistance, food assistance, or emotional and social assistance from the government or any HIV or AIDS support organization? 2. Do you think that knowing the HIV-positive status is good for any person? 52 ORGANIZATIONAL CAPACITY ASSESSMENT TOOL: FACILITATOR’S COPY External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 53 ORGANIZATIONAL CAPACITY ASSESSMENT (OCA) TOOL Goal: The goal of this tool is to assist organizations in assessing the critical elements for effective organizational management, and identifying those areas that need strengthening or further development. The OCA tool assesses technical capacity in seven domains, and each domain has a number of sub￾areas. OCA Domains: 1. Governance 2. Administration 3. Human Resources 4. Financial Management 5. Organizational Management 6. Program Management 7. Project Performance Management Using This Tool: This Organizational Capacity Assessment tool is designed to enable organizational learning, foster team sharing, and encourage reflective self-assessment within organizations. Recognizing that organizational development is a process, the use of the OCA tool results in concrete action plans to provide organizations with a clear organizational development road map. The OCA can be repeated on an annual basis to monitor the effectiveness of previous actions, evaluate progress in capacity improvement, and identify new areas in need of strengthening. External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 54 GOVERNANCE Objective: To assess the organization’s motivation and stability by reviewing its guiding principles, structure and oversight. Vision/Mission Subsection Objective: To review the organization’s vision and/or mission statements, learn what drives the organization, how the statements reflect what it does and how they are communicated and understood by staff. Resources: vision and/or mission statements, anonymous staff and board questionnaires (see Facilitator’s Guide) Vision/Mission 1 2 3 4 The vision and/or mission is • Not a clearly stated description of what the organization aspires to achieve or become The vision and/or mission is • A moderately clear or specific understanding of what the organization aspires to become or achieve • Not widely held • Rarely used to direct actions or to set priorities The vision and/or mission is • A clear, specific statement of what the organization aspires to become or achieve • Well-known to most but not all staff • Sometimes used to direct actions and to set priorities The vision and/or mission is • A clear, specific and forceful understanding of what the organization aspires to become or to achieve • Well-communicated and broadly held within the organization • Consistently used to direct actions and to set priorities Guiding Questions Vision/Mission Subsection Checklist Yes No N/A Comments/Quality Notes Are the statements understood and relevant to the organization’s current purpose? Is the vision or mission statement posted where staff and/or visitors see it regularly? Is the statement(s) used in organizational materials (i.e., staff handbooks, orientation materials)? Does the organization review the vision and mission statements in conjunction with strategic planning? Are the vision and mission used to set priorities? If so, please describe how. External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 55 ORGANIZATIONAL STRUCTURE Subsection Objective: To determine if the organization’s structure—most often depicted in an organogram but also perhaps in a narrative—is in line with its mission, goals and programs and if systems exist to ensure strong coordination among departments or functions. Resources: organizational diagram, organogram or narrative Organizational Structure 1 2 3 4 The organization has The organization has The organization has The organization has • No formal structure • A basic structure, but it is incomplete • A well-designed structure (e.g., • A well-defined structure relevant to its • An unclear description of its and/or undocumented organogram) relevant to its mission/goals and programs departments and their functions • A structure that is not aligned with its mission/goals and programs • Clearly defined and appropriate mission/goals and programs • Identified the functions and functions and responsibilities of • Unclear definitions of department responsibilities of departments departments functions • Clearly defined and appropriate lines of • Clear, appropriate lines of • Somewhat clear lines of responsibility responsibility and communication communication and coordination and communication among departments among departments among departments • A narrative description of the structure if appropriate Guiding Questions Organizational Structure Subsection Checklist Yes No N/A Comments/Quality Notes Is the organizational and reporting structure clearly documented and disseminated? Is there an organogram or similar document outlining supervisory and staff responsibilities? Please describe the functions of the departments and how departments communicate with each other. Please describe how the structure supports the mission. External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 56 SECRETARIAT COMPOSITION AND RESPONSIBILITIES Subsection Objective: To assess the secretariat’s composition, terms of reference (TOR), procedures and oversight to ensure that the secretariat is capable of providing adequate guidance to the organization. Resources: board membership, board TOR, board meeting minutes, anonymous board questionnaire Secretariat Composition and Responsibility 1 2 3 4 The secretariat The secretariat The secretariat The secretariat • Is drawn from a narrow spectrum; • Is drawn from a somewhat broad • Is drawn from a broad spectrum; all • Is drawn from a broad spectrum; all members have little or no relevant spectrum; some members have members have relevant experience members have relevant experience Experience relevant experience • Has term limits that are defined and • Has term limits that are defined and • Has term limits that are not defined • Has term limits that are not defined or reasonable reasonable or are unreasonably long or short are unreasonable • Informally elects officers • Has officers elected/appointed • Has no process for electing officers • Has no process for electing officers • Has well-planned, documented according to board procedures • Has infrequent or poorly attended • Has well-planned meetings at regular meetings held at regular intervals with • Has regular, well-planned, documented and undocumented meetings intervals, but attendance and/or good attendance meetings with good attendance • Does not have TOR or a clear documentation is irregular • Has clear TOR reflecting appropriate • Has clear TOR and a good understanding of its key functions • Has TOR, but they are incomplete separation of roles from the understanding of its functions, all of and/or do not provide appropriate executive management team which are consistently carried out with separation of roles from the executive • Has a good understanding of its appropriate separation from the management team functions as defined in the TOR and executive management team • Has some understanding of its mostly carries them out • Displays willingness and a proven track functions as defined in the TOR, but they are inconsistently carried out • Is involved in strategic planning/policy formulation, but participation is not record to learn about the organization, to participate in strategic planning/policy • Is rarely or not at all involved in strategic planning/policy formulation always consistent formulation and to address organizational issues External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 57 Guiding Questions Secretariat Composition and Responsibility Subsection Checklist Yes No N/A Comments/Quality Notes Discuss board membership (e.g., the number of members and their experience). Are there term limits for secretariat members (e.g., two years)? Is there a system for electing secretariat members? Describe it. Are there regular secretariat meetings? How often? Are the secretariat meetings well-attended? What percentage of members attends each meeting? Are board meetings well-documented? Are minutes circulated and referred to? Does the secretariat have clearly defined TOR that detail key functions? Are secretariat members involved in strategic planning and developing a financing strategy for the organization? Describe their involvement. Are secretariat members involved in decision-making related to strategic direction and policies (including finance/administration and programming)? Do secretariat members review and approve the organization’s annual budget and annual financial statements? Is there separation of secretariat and executive roles? Is this written and adhered to? Is there a succession plan for secretariat members whose term may be interrupted? External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 58 LEGAL STATUS (NGO) Subsection Objective: To assess the organization’s legal standing—and therefore sustainability—by checking legal registration and compliance with local tax and labor laws. Resources: registration, where possible and feasible, local tax laws, local labor laws Legal Status 1 2 3 4 The organization is • Not legally registered, registration has expired, or the organization does not know its legal status and local labor laws • Not aware of its tax status and/or is not paying taxes • Not aware of statutory audit and reporting requirements The organization is • Not currently a legally recognized entity in the country in which it operates but has applied for legal status • Aware of its tax status and local labor laws but is not fully compliant • Aware of statutory audit and reporting requirements but is not fully compliant The organization is • Legally registered and aware of its tax status • Not always compliant with tax obligations and/or labor laws • Not always compliant with statutory audit and reporting requirements The organization is • Legally registered and aware of its tax status • Fully complies with tax obligations and labor laws • Fully complies with statutory audit and reporting requirements Guiding Questions Legal Status Subsection Checklist Yes No N/A Comments/Quality Notes Is documentation of legal registration readily available or posted in the office? Are labor laws adhered to and documented in human resource policies or in the handbook? Does the organization comply with the tax codes for both itself and the staff? Does the organization comply with annual statutory requirements such as audits and other reporting? Does the board review and approve the audit and other statutory reports? External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 59 SUCCESSION PLANNING Subsection Objective: To assess the organization’s ability to continue smooth operations and to manage programs in the event of an absence of, or shift in, leadership. Resources: job descriptions of senior management, succession plan, organizational chart Succession Planning 1 2 3 4 The organization The organization The organization The organization • Is very dependent on the Chief • Is dependent on the CEO/ED • Has limited dependence on CEO/ED; • Is reliant but not dependent on the Executive Officer (CEO)/Executive • Would continue to exist without the s/he does not have sole control of, CEO/ED Director (ED) CEO/ED but most likely in a very for example, finances and planning • Has a clear, documented succession plan • Would cease to exist or function different form, or with significantly • Would continue in a similar way • Has the potential for a smooth transition without the CEO/ED less capability and reduced program without the CEO/ED, but fundraising to a new leader; fundraising and program • Has no plan for how it would quality and/or program quality would suffer quality would not be major problems continue if the CEO/ED left • Has a very basic succession plan significantly • Would handle transition by having a describing how the organization will • Has a documented plan for how it senior management team fill in or one or continue if the CEO/ED leaves would continue should the CEO/ED leave, but no member of management could take on the more members of the management team would take on the CEO/ED role CEO/ED role Guiding Questions Succession Planning Subsection Checklist Yes No N/A Comments/Quality Notes Is there a deputy or other staff/board member who can fulfill the duties of the CEO/ED for short or long periods? Is this role documented in the organizational structure and in the job description of the staff/board member? In what ways is the capacity of the person to take on the role being built? Is there a plan for handling a transition process, including fundraising, operations and program quality? Please describe. External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 60 ADMINISTRATION Objective: To assess the organization’s capacity to develop and apply policies and procedures, the existence and quality of its administrative systems and its staff knowledge of the systems. OPERATIONAL POLICIES, PROCEDURES AND SYSTEMS Subsection Objective: To assess the availability of and adherence to operational policies. Resources: policy and procedures manual, anonymous staff questionnaires, related payment vouchers Operational Policies, Procedures and Systems 1 2 3 4 The organization has The organization has The organization has The organization has • No documented operational policies • Documented some operational • Documented most or all operational • Complete and appropriate operational and procedures policies and procedures, but they are policies and procedures and they are policies and procedures incomplete or not compliant with compliant with national and donor • Policies and procedures that are national and donor regulations regulations known and understood by staff • Policies and procedures that are not • Policies and procedures that are • Policies and procedures that are consistently adhered to known but not consistently adhered consistently adhered to, reviewed and • Not oriented or trained staff in the policies and procedures to • Oriented or trained staff in the policies and procedures updated • No process for regularly reviewing and updating operational policies and procedures External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 61 Guiding Questions Operational Policies and Procedures* Subsection Checklist Yes No N/A Comments/Quality Notes Are there written operational procedures? Are they approved? Do the documented procedures adequately support the operational needs of the organization (i.e., travel, procurement, fixed assets, IT, file management, meeting and workshop planning, security and safety, etc.)? Do the written operational policies/procedures address donor specific rules and regulations (for USG-funded programs this includes: code of ethics/conflict of interest; use and billing of shared equipment; vehicle utilization and maintenance; marking and branding; publications and media releases; etc.)? Are the operational policies and procedures presented in a way that is easy for non-financial staff to understand and apply? Are staff oriented/trained in the procedures? How often? How is the orientation/training documented? Are the operational procedures formally reviewed/updated? How often? What is the process? Is it documented? Are copies of forms/templates incorporated in the manual and/or readily available? Are there systems to ensure compliance with operational procedures? Please describe. Have there been findings in external or internal audits related to noncompliance with operational procedures? * Many of the questions above, and below in the subsequent administrative sections, can be answered in advance by the facilitator per detailed review of the organization’s administrative and/or financial policies and procedures manual. (Get the copy of the manuals if it exists) External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 62 INFORMATION SYSTEMS Subsection Objective: To assess the functionality of the organization’s information systems and its documentation of information system policies and procedures. Resources: information system policies and procedures, staff interviews Information Systems 1 2 3 4 The organization has The organization has The organization has The organization has • No documented information system • Documented some information • Documented most or all information • Complete and appropriate policies and procedures system policies and procedures, but system policies and procedures information system policies and • An insufficient information system to they are incomplete or inappropriate • An information system that adequately procedures manage operations and/or programs • An information system that supports supports operations and programs at • An information system that effectively • No one designated to manage the operations and programs at basic a good level of functionality without and efficiently supports operations and information system levels of functionality major inputs programs at a high level of • No one designated to manage the • A staff member (or outside provider) functionality and maintenance information system designated to manage the information system • A staff member (or outside provider) designated to manage the information system Guiding Questions Information Systems Subsection Checklist Yes No N/A Comments/Quality Notes Is a staff member or outside provider designated to manage the information system? Are information policies and procedures documented, and if yes, where? Do the policies and procedures include management of both manual and electronic information (i.e., acquisition and replacement, acceptable use, supported software, support coverage, security, file management, disaster recovery, equipment borrowing and responsibility, etc.)? Is the existing system sufficient to manage operations/programs (availability and reliability of IT system, internet connectivity, quality and reliability of support)? External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 63 HUMAN RESOURCES MANAGEMENT Objective: To assess the organization’s ability to maintain a satisfied and skilled workforce, to manage operations and staff time and to implement quality programs. JOB DESCRIPTIONS Subsection Objective: To review the systems for developing, disseminating, following and updating job descriptions (JDs) to ensure that staff roles and responsibilities are clearly defined and understood and that they are relevant to the needs of the organization. Resources: sample job descriptions for each position or level (depending on size of organization) Job descriptions 1 2 3 4 The organization has • No JDs for staff, volunteers or interns The organization has • JDs for each staff member, but not all key sections are covered • Staff, volunteers and interns who are not aware of or do not have copies of their JDs The organization has • Clear JDs for each staff member that include all sections • Staff, volunteers and interns with copies or access to copies of their JDs • JDs that are not respected/adhered to, reviewed or The organization has • JDs for each staff member that cover all sections • Staff, volunteers and interns with copies of or access to their JDs • JDs that are respected/adhered to, Guiding Questions Job Descriptions Subsection Checklist Yes No N/A Comments/Quality Notes Are there JDs with relevant details—title, job duties/responsibilities, reporting requirements, supervisory responsibilities (if any), qualifications, and skills required—for all positions in the organization, including those for volunteers and/or interns? Are JDs accessible to staff, volunteers and interns? Are JDs stored in personnel files, and are the files updated? Is there a process for reviewing JDs for adherence and/or revision? External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 64 RECRUITMENT Subsection Objective: To assess the organization’s systems for recruiting staff and consultants including confirming and documenting professional and salary history. Resources: recruitment manual/guidelines or policy, recruitment guidelines, documentation of employment history, personnel manual Recruitment 1 2 3 4 The organization has The organization has The organization has The organization has • Neither guidelines nor a consistent • Basic guidelines for recruitment, but • Clear, transparent recruitment • Clear, transparent recruitment approach to recruiting staff they are not consistently applied or guidelines, but they are neither guidelines that are consistently • No system for verifying employment reviewed consistently applied nor regularly and reviewed history for staff or consultants • No process for verifying staff or reviewed • A process for verifying, updating and consultants’ employment history • Has a process for verifying filing employment history • Not oriented or trained HR staff in employment history but does not • Consistently oriented and regularly applying the guidelines or update the information trained/updated HR staff in applying • Not provided opportunities for career • Not consistently oriented or trained the guidelines advancement HR staff in applying the guidelines • Not provided opportunities for career advancement • Provided opportunities for career advancement Guiding Questions Recruitment Subsection Checklist Yes No N/A Comments/Quality Notes Are there written recruitment guidelines? Are there procedures for verifying employment history and for filing that information? Do the guidelines include announcing/advertising, collecting CVs/short-listing, interviewing candidates, checking references and employment history, making offers and employment agreements, and are they transparent? Has appropriate staff been trained to follow recruitment procedures? Are recruitment procedures always followed? Are there opportunities for career advancement with the organization? External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 65 STAFFING LEVELS Subsection Objective: To assess the organization’s management of staffing—positions available, positions filled, vacancies—for the program and for the organization as a whole and the means for ensuring staffing levels are and remain adequate. Resources: staffing plan and/or organizational diagram, vacancy and turnover data, attendance information, retention policy Staffing Levels 1 2 3 4 The organization has The organization has The organization has The organization has • No formal staffing plan • A formal staffing plan • A formal staffing plan • A formal staffing plan • Positions/vacancies that are not • Documented positions and vacancy • Documented and available vacancy • Documented positions and vacancy Documented data data data • Many key management and technical • Some key positions filled with qualified • Qualified and skilled staff in all key • Qualified and skilled staff in all positions open or filled by staff without the right qualifications or and skilled staff • No system to ensure that positions positions (technical, administrative, finance) positions • Active recruitment to fill gaps Skills are filled quickly • A system to ensure that positions are • A system for rapidly filling new • No system to ensure that positions • High turnover rate or staff attendance filled quickly positions where staff turnover is high are filled quickly problems affecting program • Moderate turnover or minor • Minimal turnover and no attendance • High turnover and severe problems implementation attendance problems problems with staff attendance affecting • Not conducted or documented exit • Conducted and documented exit • Conducted and documented exit program implementation interviews interviews interviews and used the information • No retention procedures External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 66 Guiding Questions Staffing Levels Subsection Checklist Yes No N/A Comments/Quality Notes Has the organization documented positions needed and staffing status? Are all key positions filled or is there active recruitment to fill gaps? What process is there to ensure that staff have the qualification and skills for their positions? Is data kept on current vacancies and staff turnover? Please share. Is attendance data kept? Are there issues with attendance? If so, how are they dealt with? What approaches are used for retaining staff? Are these approaches reviewed and modified as needed? Does the organization conduct and document exit interviews and identify/analyze reasons for staff departures? External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 67 PERSONNEL POLICIES Subsection Objective: To ensure that personnel policies document and verify staff time and that best practices in managing personnel are adhered to. Resources: personnel manual, staff time records, work schedule policies, 2–3 personnel files, payment vouchers Personnel Policies 1 2 3 4 The organization has The organization has The organization has The organization has • No personnel policy manual • Basic personnel policies that include • Comprehensive and donor compliant • Comprehensive and donor compliant either a drug-free workplace policy, nondiscrimination policies (for US personnel policies including a drug￾free workplace policy, discrimination personnel policies • Policies that are adhered to and organizations), or timekeeping policy policies (for US organizations), and correspond to HR practices • Inconsistently applied the polices timekeeping policy, at a minimum • Disseminated policies to all staff and • Not disseminated the policies to all • Polices that are adhered to and required and filed signature staff and/or required signature aligned with HR practices statements statements • Not disseminated the policies to all • Regularly reviewed and updated • No process for updating personnel staff or required signature statements policies, manuals and staff time policies, manuals or staff time records • Not updated personnel policies and manuals or time records records External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 68 Guiding Questions Personnel Policies Subsection Checklist Yes No N/A Comments/Quality Notes Are there documented personnel policies that include guidelines on work schedules, employee compensation (salary) and benefits, leave, performance reviews, grievances and disciplinary procedures, ending employment (resignation/termination), administrative procedures and employee conduct, and an ethics policy and awareness program? Are donor-specific HR regulations incorporated into the policies? How is this done? (For USG-funded projects, this includes a drug-free workplace policy, timekeeping policy, and non-discrimination policy [as required]). Are the time management policies and procedures donor compliant (i.e., USG requires tracking leave and separating time by project, staff and supervisor orientation, staff and supervisor signatures, reconciliation with payroll and labor billings)? Is there an HIV workplace policy as well as other health related workplace policies, such as no alcohol, no smoking, etc.? Does staff sign receipt and acknowledgement for the personnel manual, and are the signatures recorded and filed? Is there a process for validating adherence to the policies and addressing non-compliance issues? How often is the personnel manual reviewed and updated? External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 69 STAFF PERFORMANCE MANAGEMENT Subsection Objective: To review the organization’s systems for managing staff performance including performance appraisals. Resources: samples of completed performance appraisals or a blank form Staff Performance Management 1 2 3 4 The organization has The organization has The organization has The organization has • No process for regularly assessing • A process for assessing staff • A process for assessing staff • A process for assessing staff performance staff performance performance, but it does not include performance that includes setting that includes setting objectives, listing • No probationary period or review setting objectives, listing objectives, listing responsibilities/ responsibilities/ tasks, assessing process for new staff responsibilities/tasks, supervision or tasks, assessing performance on past performance on past activities, • Not updated or filed changes in staff professional development activities, supervision and professional supervision and professional work status, salary and benefits • A three-month probationary period development development for new staff but no formal review • A performance review process for • Regularly conducted appraisals for all • A process that is not participatory new staff that is not timely or staff at least once a year and follows an auditing rather than a consistently done • Regularly reviews new staff performance supportive approach • A participatory process regularly used after the probationary period • Inconsistently filed or updated for performance appraisals • Consistently filed, updated and made changes in staff work status, salary • Conducted appraisals for some, but changes in staff work status, salary and and benefits not all, staff benefits • Consistently filed and updated changes in staff work status, salary and benefits Guiding Questions Staff Performance Management Subsection Checklist Yes No N/A Comments/Quality Notes Is there a documented process for assessing staff performance including setting objectives, listing responsibilities/tasks for the review period, assessing performance on past activities, and reviewing supervision and professional development? Is the performance assessment participatory, allowing discussion by both staff and supervisors? Are performance assessments done for all staff and conducted regularly (at least yearly)? Are performance assessments conducted for new staff at the probationary period conclusion? Are changes in staff status, salary, and benefits documented in the personnel files? External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 70 FINANCIAL MANAGEMENT Objective: To assess the quality of the organization’s financial system and policies and procedures and the staff’s knowledge of the system. FINANCIAL SYSTEMS Subsection Objective: To assess the existence and use of the financial system, especially its ability to respond to management needs and donor requirements. Resources: financial manual, accounting journals, chart of accounts, payment vouchers, staff training plan/curricula, staff interviews Financial Systems 1 2 3 4 The organization has The organization has The organization has The organization has • No formal financial system • A basic financial system, but it is • A good financial system with most or • A complete and appropriate financial • Transactions that are either not incomplete and/or not compliant all required components system recorded or are recorded on an ad accounting standards • A computerized accounting system • A fully operational, computerized hoc basis • Systems that are not consistently that is not fully operational accounting system • A filing system that maintains only adhered to • Systems that are consistently adhered • Systems that are consistently adhered invoices/receipts for all • Not oriented or trained financial staff to to, reviewed and updated and incoming funds on systems • Oriented or trained financial staff on systems • Systems known and understood by trained staff • No financial staff qualified • No process for reviewing and updating the financial system • A narrative description of its financial system in its financial manual • Not included a narrative description of its financial system in its financial manual External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 71 Guiding Questions Financial Systems* Subsection Checklist Yes No N/A Comments/Quality Notes Does the organization have a cash, accrual or modified system? How is it implemented? Is the organization using accounting software? If so, which system? If not, describe the manual system. Is there a chart of accounts (income and expenses, assets and liabilities)? Does it address donor￾specific requirements? (For USG-funded programs, this includes codes for unallowable expenses, tracking sub-grant expenses and advances, tracking advances to individuals.) Does the organization use another set of codes to assign transactions to a specific project/donor? Are all payments and receipts recorded in the organization’s bookkeeping system? How often are they recorded? How are transactions in the accounting system linked to supporting documentation? Are there separate bank accounts per funding sources (if required by donor)? Are the bank accounts reconciled monthly against the bank journals/cash books? Are cash payments made? Are they through petty cash or an alternate system? Please explain. Are field office expenses recorded on an advance and reconciliation basis in the system? Please explain. Is there a system for determining exchange rates? How are gains and losses recorded in the system? * Many of the questions above can be answered in advance by the facilitator per detailed review of the organization’s financial policies and procedures manual. External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 72 FINANCIAL POLICIES AND PROCEDURES Subsection Objective: To assess the existence and use of financial policies and procedures and their ability to respond to management needs and donor requirements. Resources: financial manual, accounting journals, chart of accounts, staff interviews, payment vouchers, staff training plan/curricula Financial Policies and Procedures 1 2 3 4 The organization has The organization has The organization has The organization has • No documented financial policies • Some documented financial policies • Documented most or all financial • Complete and appropriate financial and procedures and procedures, but they are policies and procedures and they policies and procedures • Inappropriate separation of incomplete and/or do not comply compliant • Policies and procedures that are known donor requirements • Policies and procedures that are and understood by staff Responsibilities and tasks (1-2 • Policies and procedures that are consistently adhered to • Policies and procedures that are inconsistently adhered to • Oriented or trained staff in the consistently adhered to, reviewed and People are responsible of all • Not oriented or trained staff in the policies and procedures policies and procedures • No process for regularly reviewing and updating financial policies and procedures updated Guiding Questions Financial Policies and Procedures* Subsection Checklist Yes No N/A Comments/Quality Notes Are there written financial procedures? What is the approval process? Are the policies and procedures comprehensive and responsive to the organizational needs (addressing, at a minimum, chart of accounts, bank/cash management, internal controls, fraud, financial documentation, shared costs/indirect costs, budgeting and reporting, audit and cost share)? Do the written procedures address donor specific requirements (for USG-funded programs, this includes definitions of reasonable, allocable and allowable/unallowable expenses, VAT tracking and reimbursement procedures)? Are staff oriented/trained in the procedures? How and how often? Are financial procedures formally reviewed/updated? How often? What is the process for revisions? Is the process documented? Are there systems to ensure compliance with financial procedures? Please explain. Have there been findings in external or internal audits related to noncompliance with financial procedures? * Many of the questions above can be answered in advance by the facilitator per detailed review of the organization’s financial policies and procedures manual. Financial/administrative procedures should also include fixed assets, procurement, travel and payroll, discussed in the Administration section. External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 73 FINANCIAL DOCUMENTATION Subsection Objective: To assess if record keeping is adequate and if financial files are audit ready. Resources: financial files, finance manual, staff interviews Financial Documentation 1 2 3 4 The organization has The organization has The organization has The organization has • No written financial documentation • Some written financial documentation • Financial documentation procedures • Complete and appropriate financial Procedures procedures, but they are incomplete that are mostly or completely documentation procedures • No filing system, and financial files are and/or inappropriate documented in writing and • Procedures that are known and not readily available • Procedures that are not consistently appropriate understood by staff • No one designated to manage the adhered to and/or are not known to • Procedures that are generally adhered • Procedures that are consistently financial files staff to, known and understood by staff adhered to, reviewed and updated • A basic filing system, but financial files • Financial documentation files that are • Up-to-date financial files in a secure are not complete not regularly updated or secure location • No one designated to manage the • A staff member designated to manage • A staff member designated to manage financial files the financial files the financial files Guiding Questions Financial Documentation Subsection Checklist Yes No N/A Comments/Quality Notes Is there a written process for managing financial documentation? Are there written guidelines describing the documentation required for each type of transaction? Are these guidelines formally reviewed/updated? How often? What is the process for revisions? Is it documented? Does each transaction (and payment voucher) include and/or reference supporting documentation? Is financial documentation up-to-date? Is financial documentation kept in a secure and consistent location? Is/are there a designated person(s) to manage financial files? Is there a policy on which and how long financial documents are kept? Is the policy compliant with local law and donor regulations? External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 74 BUDGETING Subsection Objective: To assess the organization’s financial planning and if there is a system for monitoring budgets and determining additional funding requirements. Resources: organization’s budget, project budgets, budget worksheet, chart of accounts, budget tracking worksheet Budgeting 1 2 3 4 The organization has The organization has The organization has The organization has • No formal master budget • A basic master budgeting process, but • A good master budgeting process that • A complete and appropriate master • No core-cost budget it is incomplete includes most or all required budget • Project budgets, but they are not • A core-cost budget, but it is not components • A core-cost budget that is aligned with clear and/or not aligned with aligned with the strategic plan and/or is • A core-cost budget that is generally the strategic plan and regularly project needs not regularly reviewed to address aligned with the strategic plan, but is reviewed; any shortfalls are addressed • Not included core costs in its shortfalls not regularly reviewed to address • Clear project budgets that are reviewed project budgets • Project budgets, but they are not shortfalls regularly by senior management and always clear and not consistently • Project budgets that are clear, but not adapted to align with project needs and aligned with project needs reviewed regularly nor consistently donor requirements • An inconsistent methodology for aligned with project needs • A consistent methodology for including including core costs in its project • A consistent methodology for core costs in project budgets that is budgets including core costs in project budgets, but the methodology is not documented and does not ensure full cost recovery documented and ensures full cost recovery External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 75 Guiding Questions Budgeting Subsection Checklist Yes No N/A Comments/Quality Notes Does the organization have a master budget that includes the costs of running the organization, or does the organization operate only with project budgets and a separate core-cost (administration/overhead) budget? Are core costs included in budgets in a consistent manner? Is full cost recovery achieved? Does the budget align with the strategic plan and is it approved by the board? Are budgets prepared or reviewed annually? Who approves budgets? Is approval received before the fiscal year starts? Please explain. Are program and financial staff involved in budgeting? Is there a budget holder (named individual) responsible for managing each budget? How are budgets monitored? Are there regular meetings with senior management and relevant program staff to review budget status? How are non-budgeted expenses handled (i.e., approval, budget allocation)? How are funds reallocated between line items as a result? Are there systems to manage cash flow on a monthly basis? Please explain. External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 76 FINANCIAL REPORTING Subsection Objective: To assess whether the organization’s routine financial reporting system allows it to meet statutory and donor requirements and stakeholders’ needs for information. Resources: annual financial statements, financial reports to donors, donor grant agreements, management reports, senior management meeting minutes, board meeting minutes Financial Reporting 1 2 3 4 The organization has The organization has The organization has The organization has • No routine system for financial reporting • A basic system for financial reporting, but reporting requirements and • A good financial reporting system; reporting requirements and deadlines • A complete and appropriate financial reporting system; • No recent financial statements deadlines are not adhered to are generally adhered to requirements and deadlines are • Not yet submitted a financial report • Designated staff to prepare and • Regularly delivered financial reports to consistently adhered to to a donor and/or other stakeholders approve reports and financial stakeholders (donors, budget • Regularly delivered accurate and • No one designated to prepare or approve reports or financial statements statements • Inconsistently delivered financial reports to stakeholders (donor, budget holders, senior management, board members) senior management, board members), but they are not always accurate and/or complete • Sporadic reviews of financial reports by senior staff complete financial reports to stakeholders (donors, budget holders, senior management, board members) • A system for senior staff to review • Irregular reviews of financial reports • Some documented financial reporting months and to use the reports to by senior staff procedures make decisions . • Complete and appropriately documented financial reporting procedures regularly reviewed and updated Guiding Questions Financial Reporting Subsection Checklist Yes No N/A Comments/Quality Notes Are there written procedures for financial reporting? What reports (organizational, management, donor) are prepared and how often? Are the reports accurate and submitted on time? Is there a person designated to prepare financial reports (statements, management and donor reports)? Review the reports? Approve the reports? External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 77 ORGANIZATIONAL MANAGEMENT Objective: To assess the organization’s planning, management of external relations and information and means of identifying and capitalizing on new opportunities. STRATEGIC PLANNING Subsection Objective: To assess the organization’s ability to realize its mission and goals by reviewing its strategic plan. Resources: strategic plan Strategic Planning 1 2 3 4 The organization has The organization has The organization has The organization has • No strategic plan • A basic strategic plan that does not • A comprehensive, written strategic • A comprehensive, written strategic reflect its vision, mission and values plan that reflects its mission, vision and plan that reflects its mission, vision and • A plan that is not based on an analysis values values of strengths and weaknesses, the • Based the plan on a review of strengths • Based the plan on a review of strengths external environment and clients’ and weaknesses, the external and weaknesses, the external needs environment and clients’ needs environment and clients’ needs • A plan that does not include priorities, measurable objectives or clear • Included priorities, measurable objectives and clear strategies • Included priorities and measurable objectives strategies • Not used the plan for management • Referred to the plan for management • Not used the plan for management decisions or operational planning decisions and operational planning decisions or operational planning • No process for regular reviews • Regularly reviewed the plan • No process for regularly reviewing the • Not defined resource needs or does • Clear resource needs and a plan not have a corresponding budget corresponding budget • Not defined its resource needs External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 78 Guiding Questions Strategic Planning Subsection Checklist Yes No N/A Comments/Quality Notes Does the organization have a strategic plan? When was it developed and for what period of time? Did strategic planning include stakeholders? Does the strategic plan outline the organization’s mission, niche, competitors, and partners? Does the strategic plan include priority areas, measurable objectives, and clear strategies? Is the strategic plan used to guide workplanning and staffing decisions? Does the organization have a mechanism for incorporating lessons learned and best practices into the planning process? Does the plan include a process for regular reviews? Does the plan identify resource needs and costs? External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 79 OPERATIONAL PLAN DEVELOPMENT Subsection Objective: To assess the contents, approval and reviews of the annual operational plan. Resources: operational plan Operational Plan 1 2 3 4 The organization has The organization has The organization has The organization has • No operational plan • An annual operational plan • An annual operational plan • An annual operational plan • Included goals, measurable objectives and strategies, but no timelines, responsibilities or indicators • Included goals, measurable objectives, strategies, timelines, responsibilities and indicators • Included goals, measurable objectives, strategies, timelines, responsibilities and indicators • Not linked the operational plan to • Linked the plan to project/program • Linked the plan to program/project project or program workplans and workplans and budgets workplans and budget budgets • Not developed the operational plan • Developed the plan with staff • Not developed the operational plan with staff participation participation with staff participation • No dates for quarterly reviews • Set dates for quarterly reviews • Not set dates for quarterly reviews • Not submitted the plan on time to HQ • Submitted the plan on time to HQ or • Not submitted the plan on time to or donors (if required) donors (if required) HQ or donors (if required) Guiding Questions Operational Plan Subsection Checklist Yes No N/A Comments/Quality Notes Does the organization have an annual operational plan? How are project workplans integrated into the annual operational plan? Does the plan have clearly stated goals and measurable objectives and strategies, clear timelines, responsibilities and indicators? Please describe its contents. Is the plan linked to the annual budget? Was the plan developed with staff participation? Does the organization have an annual planning cycle? External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 80 RESOURCE MOBILIZATION Subsection Objective: To assess the organization’s ability to identify and capitalize on new business opportunities through grants and partnerships. Resources: business development plan, resource development plan, funding strategy Resource Mobilization 1 2 3 4 The organization has The organization has The organization has The organization has • No business plan or funding strategy • A business plan and has taken • A business plan based on an analysis • A business plan based on an analysis of its • Not estimated its future resource preliminary steps to estimate future of its programs and resource needs programs and resource needs and the Needs resource needs based on an analysis and the activities in its strategic plan activities in its strategic plan • Taken no steps to identify additional of its programs and/or its strategic • Identified resource providers • Identified resource providers local, national or international plan • Created a communication strategy • Created a communication strategy for resources or opportunities to • Identified additional resource for resource mobilization resource mobilization support its programs and activities, either directly or through providers or opportunities and their interests and potential for support • Received support from at least one source or has a clear plan for • Successfully bid for resources from one or more sources Partnerships • Not created a communication fundraising or proposal writing • Sufficient funds to support its activities • Not created a communication strategy for resource mobilization strategy to attract resources • Insufficient funds to support its activities External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 81 Guiding Questions Resource Mobilization Subsection Checklist Yes No N/A Comments/Quality Notes Has the organization designated a person to carry out the resource mobilization activities? Does the person have the required skills and qualifications for this task? Does the organization have a business plan or funding strategy? Is it in line with its strategic plan? Does the organization know the resources it needs based on an analysis of its programs or through reviewing strategic planning resource needs? Does the organization have sufficient funds to support activities for the next year? Three years? Does the organization receive support from more than one donor? Who are the donors? Have potential resource providers (sources) been identified? Is there a development plan (fundraising/proposal writing) for obtaining additional resources? Is there a communication and networking strategy to attract resources? What is the minimum amount of money that will attract the organization to compete for a proposal? How many proposals has the organization submitted in the past year? External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 82 COMMUNICATION STRATEGY (SECRETARIAT) Subsection Objective: To assess the comprehensive, completeness and effectiveness of the organization’s communication strategy. Resources: communication strategy, sample USAID-funded and non-USAID-funded publications Communication Strategy 1 2 3 4 The organization has The organization has The organization has The organization has • No strategy for identifying • An incomplete strategy, lacking • A complete communication • A comprehensive audiences, channels, materials, and dissemination for promotion of objectives, responsibility, timelines and dissemination strategy, • Tasked staff member(s) with communication strategy • Tasked staff member(s) with technical/best practice innovation mechanisms communication strategy communication strategy and overall achievements • Assigned responsibility for management including management, including • No one assigned responsibility for communication strategy documentation oversight documentation development developing/overseeing development • A process for testing oversight communication strategy and • No process/tools for testing materials/messages and • A process for testing and products (written, oral and/or materials/messages based on test results revising materials/messages online) • Developed branding/marking • Developed its own branding based on test results • No process/tools for testing the policies for projects as required policy (including appropriate • Developed its own branding materials/messages by USAID but does not have an USAID branding/marking policy (including appropriate • No branding/marking policies or procedures for documents or equipment organizational branding/marking policy requirements) and oriented staff, but it is inconsistently adhered to • Created templates for documents and a style guide USAID branding/marking requirements), oriented staff, and instituted a system to monitor compliance • Created templates and a style guide and trained staff h i External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 83 Guiding Questions Communication Strategy Subsection Checklist Yes No N/A Comments/Quality Notes Does the organization have a communication strategy? Is it comprehensive (i.e., including objectives, responsibility, audience, channels, resources, testing, dissemination, timeframe and monitoring)? Does the organization have its own brand/logo/tagline? Does the organization have a documented branding/marking policy? Does it comply with USAID requirements (See http://transition.usaid.gov/branding/gsm.htmlandhttp://www.usaid.gov/branding for guidance.)? Are staff trained on it? Is there a process to review compliance? Is/are qualified staff member(s) tasked with communication strategy management and documentation (oral, written and online) oversight? Does the organization pre-test materials/messages as part of documentation development and are revisions made based on the test results? Does the organization have templates and a style guide and have staff been trained on the guidance? External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 84 CHANGE MANAGEMENT Subsection Objective: To assess the organization’s sustainability and relevance by reviewing its systems and processes for responding to internal or external emerging situations, reviewing programs and analyzing needs. Resources: policy review plan or timeline Change Management 1 2 3 4 The organization has The organization has The organization has The organization has • No process for responding to • Basic processes for reviewing internal • Established processes for reviewing • Established effective and consistent internal changes (staffing, leadership changes, such as policy reviews or the internal change routines for planning and reviewing and and budget issues) funding environment • Processes for planning for and responding to internal and external • No process for planning for or • No process for planning for or responding to external change change responding to external changes responding to external changes, such • Consistently involved staff in • Consistently involved staff in reviewing (government policies or donor as regular reviews of the operational reviewing the effectiveness of the effectiveness of new/revised priorities/funding) plan and budget monitoring new/revised management systems and management systems and policies • Inconsistently involved staff in policies, processes, programs • Systems for monitoring whether changes reviewing the effectiveness of • Few delays or major problems are implemented and lead to new/revised management systems and policies • Significant delays or problems encountered in response to change encountered in response to change improvements • Ways to gauge staff comfort with the way change is introduced and addressed Guiding Questions Change Management Subsection Checklist Yes No N/A Comments/Quality Notes Does the organization have a process for responding to internal and/or external changes? Please describe. Does the organization have an established routine for involving staff in reviewing policies and procedures or plans? Please explain. Does the organization have a process for reviewing and updating the strategic or operational plans to address or prepare for government, donor or funding changes? Does the organization have a process for monitoring whether revisions are implemented and lead to improvements? Does the organization have a means for gauging staff comfort with how change is addressed? External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 85 KNOWLEDGE MANAGEMENT Subsection Objective: To assess the organization’s ability to link with other organizations (government, national, international, community, technical, academic) and its system for sharing knowledge, experiences, technical expertise and best practices with staff. Resources: listing of association memberships and linkages with external organizations, staff reports on meetings attended, organizational newsletters Knowledge Management 1 2 3 4 The organization has The organization has The organization has The organization has • No technical linkages with other • Basic technical linkages with other • Essential and appropriate links with • Active links with appropriate organizations to share best practices organizations to share best practices other organizations to share best organizations to share best practices or or program experiences or program experiences practices or program experiences program experiences • No process for ensuring staff are • Staff who are updated on best • A process for routine staff sharing of • A process for routinely sharing technical continuously updated on best practices, but not regularly best practices and lessons learned expertise and experiences with staff and Practices • No process for ensuring learning is • Not applied new knowledge or best stakeholders applied to the program or shared practices to ongoing programs or • Applied best practices to its program and with stakeholders shared them with stakeholders • Has no process for reviewing/integrating new/current knowledge and best practices in annual planning shares information with stakeholders and appropriate staff • Annual planning that includes reviews and integration of new/current knowledge and best practices Guiding Questions Knowledge Management Subsection Checklist Yes No N/A Comments/Quality Notes Does the organization have a relationship or collaborate with appropriate civil society organizations, government partners, and/or relevant international policy or academic bodies? Does the organization have a process for reviewing and adapting new systems or best practices? Do government and other organizations know what the organization does? Do government and other organizations view and seek the organization out as a technical resource? Does the organization publicize and/or disseminate information about itself and its work to the public and/or other organizations, stakeholders and/or beneficiaries? External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 86 STAKEHOLDER INVOLVEMENT Subsection Objective: To assess the organization’s ability to coordinate programs and to involve stakeholders. Resources: list of key stakeholders, stakeholder report Stakeholder Involvement 1 2 3 4 The organization has • No information about key stakeholders and service providers in the same geographic and/or technical areas in which it operates The organization has • Some information about stakeholders and service providers in the same geographic and/or technical areas in which it operates • Information that is incomplete and out of date The organization has • Current information about stakeholders working in the same geographic and technical areas • Identified where stakeholders are, what they do, their expectations and how/if they can collaborate • No regular meetings with stakeholders The organization has • Complete and up-to-date information about all stakeholders working in the same geographic and technical areas and, if appropriate, collaborative agreements with them • Regular (at least annually) meetings with stakeholders to review relevant activities and their impact on the organization’s area of operations Guiding Questions Stakeholder Involvement Subsection Checklist Yes No N/A Comments/Quality Notes Does the organization have an appropriate and broad definition of stakeholders? Does the organization have complete and up-to-date information about all stakeholders working in the same geographical and technical areas? Are there collaborative agreements where appropriate? Are regular meetings held with stakeholders to share information, review relevant activities and impact and explore ways to collaborate? How often are these meetings held? External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 87 INTERNAL COMMUNICATION (IF NO TIME IT CAN BE DELETED) Subsection Objective: To review the organization’s approach to internal communication. Resources: staff questionnaires (Facilitator’s Guide) Internal Communication 1 2 3 4 The organization has The organization has The organization has The organization has • Limited communication between • Limited communication between and • Open communication between and • Open communication between and and among management and staff among management and staff among management and staff among management and staff • Few structured opportunities to • Opportunities for discussions • Regular opportunities for discussing • Regular opportunities for exchanging exchange ideas or to discuss management, program or between and among management and staff, but they are rarely used management, program or technical areas ideas or discussing management, program or issues • Sometimes encouraged staff ideas and • Encouraged staff ideas and input • Consistently encouraged and • Not encouraged staff ideas or input input • Staff who are comfortable raising incorporated staff ideas and input • Staff who feel uncomfortable raising • Staff who feel uncomfortable raising issues but find it more difficult to • Staff who feel comfortable initiating issues issues challenging ones discussions, contributing ideas and raising issues Guiding Questions Internal Communication Subsection Checklist Yes No N/A Comments/Quality Notes Are management and staff accepting of different communication styles and flows (formal, informal, face-to-face, and confidential)? How often are meetings held for all staff? What other mechanisms are there for assisting internal communication (e.g., internal newsletters, memos, social events)? Does management encourage and incorporate staff ideas and input? Are staff comfortable raising challenging issues using the existing communication mechanisms? External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 88 DECISION-MAKING Subsection Objective: To assess how the organization makes decisions, who is involved, and how decisions are communicated. Resources: staff questionnaires (Facilitator’s Guide) Decision-Making 1 2 3 4 The organization has • Not included staff in the decision￾making process • Not communicated or explained decisions that affect the organization • Staff who feel excluded The organization has • An unclear process for seeking and including staff ideas in the decision￾making process • Inconsistently communicated or explained decisions to staff • Staff who feel they play a minor role in making decisions The organization has • Encouraged staff ideas but seldom incorporated them into decisions • Communicated and explained decisions to staff • Not fully included staff participation in making decisions The organization has • Sought, respected and incorporated staff ideas into decision-making • Communicated and explained decisions to staff • Staff who feel a sense of responsibility, accountability and ownership of decision-making Guiding Questions Decision-Making Subsection Checklist Yes No N/A Comments/Quality Notes Does the organization encourage staff to participate in decision-making forums (staff meetings, strategic planning, visioning)? Are multiple staff members (technical, administrative, financial) involved in relevant decision-making processes? Are multiple methods used in making decisions? What methods are used? Does management communicate and explain decisions affecting the organization? Do forums exist for staff to voice concerns and competing ideas should a conflict arise about a decision(s)? External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 89 PROGRAM MANAGEMENT Objective: To assess the organization’s ability to implement comprehensive programs that respond to local needs and priorities by reviewing compliance with donor requirements, management of sub-grants with partners, technical reporting and whether its comprehensive health services meet the needs of specific target populations. DONOR COMPLIANCE Objective: To assess the organization’s capability to respond to USG donor requirements; thereby ensuring the effective implementation of its USG-funded programs. Resources: copy of the USAID A-122 Cost Principles, staff interviews (Facilitator’s Guide) Donor Compliance 1 2 3 4 The organization • Is not familiar with the terms of the cooperative agreement, A-122 Cost Principles (i.e., reasonable, allocable, and allowable) or Standard Provisions • Has not listed and assigned responsibility for all donor requirements The organization • Is knowledgeable of the terms of the cooperative agreement, A-122 Cost Principles and Standard Provisions • Is aware of donor requirements, has assigned responsibility, but does not have systems in place to ensure The organization • Is knowledgeable of the terms of the cooperative agreement, A-122 Cost Principles and Standard Provisions • Has systems in place to ensure compliance with donor requirements • Does not comply consistently The organization • Is knowledgeable of the terms of the cooperative agreement, A-122 Cost Principles, and Standard Provisions • Has systems in place to ensure compliance with donor requirements • Complies consistently External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 90 Guiding Questions Donor Compliance Subsection Checklist Yes No N/A Comments/Quality Notes Does the organization have a copy of all donor agreements readily available? Does the organization have copies of all the modifications that have been made to the donor agreements? Are copies of the A-122 Cost Principles and Standard Provisions readily available for USG-funded agreements? Is the organization aware of all USG requirements which may be applicable, including, but not limited to, submission of workplans, marking and branding plans, environmental compliance plans, financial reports, semiannual and annual technical reports, inventory reports, VAT reports, audit reports, DEC submissions, procurement approvals, sub-grant approvals, sub-grant certifications, etc.? Has responsibility for each of the above requirements been assigned? Are requirements fulfilled correctly and on-time? Is a system in place to ensure compliance with the requirements? Describe the system. Have there been findings in external or internal audits related to non-compliance with donor regulations? External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 91 TECHNICAL REPORTING Subsection Objective: To review the organization’s ability to document technical activities and results for donors, program planning and program development. Resources: most recent technical report, workplan Technical Reporting 1 2 3 4 The organization The organization The organization The organization • Does not document quantitative or • Documents qualitative progress on its • Documents both qualitative and • Documents both quantitative and qualitative progress on its workplan workplan, including objectives and quantitative workplan progress and qualitative workplan progress, and or its objectives and strategies, facilitating factors or barriers strategies, facilitating factors and barriers reviews objectives and strategies, facilitating factors and barriers reviews objectives and strategies, facilitating factors and barriers • Does not identify lessons learned • Does not identify lessons learned or • Documents lessons learned and best • Documents lessons learned and best and/or best practices best practices practices practices • Does not report on donor, government or other program • Does not report on government, donor or other program indicators • Reports on donor, government or other program indicators • Reports on donor, government and other program indicators indicators • Does not use information to • Does not use information to • Uses information to review/revise • Does not use information to review/revise strategies with staff or review/revise strategies with staff and strategies with staff and stakeholders review/revise its strategy with staff and stakeholders stakeholders stakeholders Guiding Questions Technical Reporting Subsection Checklist Yes No N/A Comments/Quality Notes Are there systems for regularly reviewing and documenting progress on the workplan (program, donor or national indicators, objectives and strategies)? Please explain. Are systems in place to identify facilitating factors and barriers to progress? Are there systems for identifying lessons learned or best practices? Are lessons learned, gaps or shortfalls and best practices documented? Does the organization review findings and revise strategies with staff and stakeholders based on the findings? External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 92 REFERRAL (NGO – SAFETSTOP CENTERS) Subsection Objective: To assess the organization’s systems and processes for directing clients to other providers, ensuring those providers offer quality services and monitoring clients’ access to services. Resources: referral plan, memoranda of understanding with referral sites, referral reports or data Referral 1 2 3 4 The organization has The organization has The organization has The organization has • Not mapped referral sites • Mapped referral sites • A clear referral process with • A clear referral process system and • Not established links for referring clients for HIV and AIDS treatment • No agreements with government, private or NGO health or social government, private or NGO health or social service providers to ensure strong linkages with government, private or NGO health or social service or other health/support services service providers to ensure that that clients requiring HIV and AIDS providers to ensure that clients requiring requiring HIV and AIDS treatment or treatment or other health or HIV and AIDS treatment or other health other health or support services have support services have access to or support services have access to them access to them them • A process for following up clients and monitoring quality of care • Clients who are not always appropriately referred or who encounter problems at referral sites • A process for following up clients and monitoring quality of care • Clients who are consistently referred to appropriate locations and who do not encounter problems at referral sites Guiding Questions Referral Subsection Checklist Yes No N/A Comments/Quality Notes Does the organization have partners to whom they refer clients? Are the referral sites documented and mapped for easy reference? Does the organization have a list of alternative referral sites should the usual agency not be able to help? Are clients consistently referred to the right location? Does the organization develop memoranda of understanding with referral organizations? Does the organization follow up with clients to ensure satisfaction with the referral? Does the organization follow up with referral agencies to monitor receipt of services? Does the organization monitor quality of care provided at the referral sites? External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 93 COMMUNITY INVOLVEMENT (NGO) Subsection Objective: To ensure the organization’s programs respond to and address community needs by reviewing how they involve community members in planning and decision-making. Resources: community participation and/ or mobilization plan; if not documented, discuss approach with appropriate staff Community Involvement 1 2 3 4 The organization • Orients communities on its programs, but does not actively include them • Does not involve affected families and communities in planning and decision-making The organization • Orients communities on its program and discusses its approach with community leaders • Inconsistently involves affected families and communities in planning and decision-making The organization • Orients communities and leaders on its program and actively engages themin the activities • Involves affected families and communities in planning and decision- making and sometimes integrates their ideas into program design and revision The organization • Orients communities and leaders on its program and actively engages them in activities and service provision • Involves affected families and communities in planning and decision-making and consistently integrates their views into program design and revision Guiding Questions Community Involvement Subsection Checklist Yes No N/A Comments/Quality Notes Does the organization involve the community in program activities? How? Does the organization identify and involve community leaders in program orientation, design and review? Please explain. Does the organization solicit feedback and information from target audiences? Are target audiences involved in making decisions? Does the organization regularly interact with target audiences? Does the organization use feedback and information collected from target audiences to inform program activities? Does the organization share program results with relevant community leaders or associations for planning and troubleshooting? External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 94 CULTURE, GENDER AND HANDICAP Subsection Objective: To evaluate the organization’s systems for assessing culture and gender issues among the populations it serves and for integrating cultural and gender concerns into its programs. Resources: community or client assessments, program plans Culture and Gender 1 2 3 4 The organization does The organization does The organization does The organization does • Not consider local cultural or • Consider local cultural or gender • Consider local cultural or gender • Consider local culture or gender gender issues in programming issues in its programming concerns in its programming concerns in its programming • Not have tools for assessing local • Not have tools for assessing local • Have tools for assessing cultural and • View culture and gender as integral to cultural or gender issues cultural or gender issues relevant to gender issues program success • Not discuss the role of local culture programs • Have guidelines for culturally relevant • Have tools for assessing cultural and and gender norms in program • Discuss the role of local culture and and gender based approaches and gender issues design with staff gender norms in program design with programming • Have guidelines for culturally relevant staff • Not train staff on how to use the tools or findings and gender-based approaches and programming • Train staff on the tools, interpreting findings and incorporating elements of culture and gender in program design Guiding Questions Culture and Gender Subsection Checklist Yes No N/A Comments/Quality Notes Does the organization consider local culture and gender in programming? Please describe how. Does the organization have clearly documented guidelines for culturally relevant and/or gender-based approaches and programming? Does the organization provide training in gender and/or cultural issues and survey tools? Are findings from culture and/or gender assessments used in program development and implementation? Are monitoring tools used to continue assessing local issues? External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 95 PROJECT PERFORMANCE MANAGEMENT Objective: To assess the organization’s systems for overseeing field activities, for setting standards and monitoring actual performance against them and for setting indicators and monitoring progress toward achieving outcomes. FIELD OVERSIGHT ACTIVITIES Subsection Objective: To assess the organization’s systems for overseeing field activities. Resources: field oversight policies and procedures, trip reports, management meeting minutes Field Oversight 1 2 3 4 The organization The organization The organization The organization • Has no formal procedures and • Has some documented field oversight • Has most or all documented • Has documented and comprehensive processes for overseeing field policies, but they are incomplete oversight policies and procedures field oversight policies and procedures administrative and programmatic • Reviews annual workplans and • Approves annual workplans on a • Approves workplans and provides operations progress reports, but irregularly regular basis feedback • Monitors compliance with program • Monitors compliance with program • Reviews data and progress reports and and donor requirements and donor requirements provides feedback • Reviews and approves field-level HR and finance manuals • Monitors compliance with program and donor requirements • Reviews quarterly project M&E data and progress reports • Reviews and approves field-level HR and finance manuals • Provides technical and administrative guidance • Provides technical and administrative guidance • Makes irregular supervision visits • Makes at least semi-annual supervisory visits, and results are discussed with management and technical staff External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 96 Guiding Questions Field Oversight Subsection Checklist Yes No N/A Comments/Quality Notes Does the organization have documented field oversight policies and procedures? Do field offices have standardized field-level HR and finance manuals that are reviewed and approved by HQ? Does the organization monitor compliance (e.g., staff qualifications, timesheets)? How? Is HQ involved in reviewing field office annual workplans/budgets? Who has approval authority? Is there a system for regular reporting from the field office to headquarters (e.g., annually)? Does the organization review M&E data? How often? Does the organization conduct supervisory visits at least twice a year? Who conducts these visits and how are results discussed with management and staff? Is there a system to regularly monitor programs and their effectiveness? External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 97 STANDARDS Subsection Objective: To assess the organization’s application of recognized standards in service delivery. Resources: standards/guidelines used, monitoring reports Standards 1 2 3 4 The organization has • No standards for service delivery The organization has • Minimal standards for service delivery • Not made staff aware of the standards • Not applied the standards appropriately The organization has • A good system for using standards for service delivery • Made staff aware of the standards • Appropriately trained staff to apply and monitor the standards • A process for monitoring standards, but it is not applied comprehensively The organization has • Solid standards for service delivery • Made staff aware of the standards and has trained staff to apply them • A process for monitoring adherence to standards that is consistently adhered to • A process for improving adherence to standards Guiding Questions Standards Subsection Checklist Yes No N/A Comments/Quality Notes Is there a document that outlines service delivery standards? Do relevant staff have copies and/or are they accessible? Is there training for staff and is adherence to standards monitored? Is there a process for improving adherence to standards? External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 98 QUALITY ASSURANCE Subsection Objective: To assess the organization’s ability to identify and address gaps in meeting performance standards. Resources: quality monitoring tools (could be part of M&E tools) Quality Assurance 1 2 3 4 The organization has The organization has The organization has The organization has • Unclear performance expectations • Performance expectations, but no • Performance expectations and a • Performance expectations and a system • No process for monitoring the process to assess performance process that assesses performance that assesses performance against quality of services it provides, against standards against standards standards through program evaluations, • Taken client satisfaction into • Taken client satisfaction into consideration monitoring or supervision consideration • Included an analysis of gaps or weaknesses • Analyzed gaps or weaknesses to identify root causes • Identified a plan to address root causes • Not developed an improvement plan • An improvement plan to address gaps or weaknesses • Studied and incorporated the results into the program Guiding Questions Quality Assurance Subsection Checklist Yes No N/A Comments/Quality Notes Is a process for quality assurance documented? Are technical and program performance expectations outlined? Is QA used to assess performance against standards and determine root causes? Is client satisfaction data collected and utilized to improve the program/results? Does the organization determine how to address the root causes and test the process? Are results assessed and used to improve implementation? Does the organization try to scale up effective practices? External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 99 SUPERVISION Subsection Objective: To assess the organization’s systems for supportive review of and feedback on staff performance and program activities. Resources: supervision plan or guidelines, supervisors’ reports Supervision 1 2 3 4 The organization has The organization has The organization has: The organization has • Not developed a supervision plan or • A supervision plan but no approach • A clear supervision plan with a • A detailed supervision plan with a approach • Detailed supervisory responsibilities, supportive approach supportive approach • Not clarified supervisory but they are not followed • Detailed supervisory responsibilities • Detailed supervisory responsibilities that responsibilities • Not trained supervisors or provided that are followed are followed • Not trained supervisors or provided tools • Trained supervisors and provided • Trained supervisors and provided them tools • An unclear process for supervision them with tools with tools • No process for carrying out • No process for reviewing findings • Logistical and program barriers to • A mechanism for carrying out visits supervision with staff and management providing regular supervision according to the timeline • No process for documenting or discussing findings with staff and management • A process for documenting and discussing findings with staff and management • A process for following up and addressing issues Guiding Questions Supervision Subsection Checklist Yes No N/A Comments/Quality Notes Does the organization have a supervision plan? Does the organization use a supportive supervisory approach? Is supportive supervision training conducted regularly? What supervisory tools does the organization use? Are findings documented and discussed with management and staff? Is there a process for following up and addressing findings? External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 100 MONITORING AND EVALUATION (M&E) Subsection Objective: To assess how the organization collects and uses data to plan, monitor and evaluate its programs. Resources: M&E plan, M&E tools, M&E reports Monitoring and Evaluation 1 2 3 4 The organization has The organization has The organization has The organization has • No M&E plan • A basic M&E plan • A well-defined M&E plan • A well-defined M&E plan • No process for monitoring • Identified outcome indicators • Process and outcome indicators • Process and outcome indicators implementation • Developed data collection tools • Trained staff to collect data, and • A process for using data for follow-up • Not identified indicators to • Trained staff in M&E collection is consistently done monitoring, program adjustments, • No system for data processing: tools, trained data collectors, data quality review or a plan • No system for regularly collecting, analyzing or reporting data • A process for consistently using data/findings for follow-up monitoring, support or planning and determining progress towards achieving targets • A process for data quality review and using information • No process for reporting against targets • A strategy for reporting on progress against targets • No process for sharing results with field and against targets and involving staff and data • A strategy for regularly sharing information with stakeholders, including the community Guiding Questions Monitoring and Evaluation Subsection Checklist Yes No N/A Comments/Quality Notes Is there a documented M&E plan that includes process (output) and outcome indicators, data collection tools and schedule, quality review and methods for Has the M&E plan been approved (if appropriate)? By whom? Is M&E training offered to relevant staff? External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 101 Is M&E data collected by trained staff using standardized tools on a regular basis? Is someone responsible for data quality review? Are M&E findings reported on and shared with staff and appropriate stakeholders, including the community? Are M&E results used to improve the program? External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 102 ANNEX B: ORGANIZATIONAL CAPACITY TOOL COMPONENTS The organizational capacity components chosen for this evaluation are listed below along with specific indicators and focal areas. Some examples of components and their indicators: • Governance and Leadership o Succession Plan o Organizational Structure • Finance and Administration o Information Systems o Operational Policies, procedures and systems • Human Resources Management o Personnel Performances Management o Staff Competences • Financial Management o Accounting Systems o Financial Reporting • Organizational Management o Strategic Planning o Change Management • Program Management o Donor Compliance o Community Commitment • Project Performance Management o Field Activities Monitoring o Quality Insurance All indicators were assessed by area and organization using a 4-point grading system outlined below. • The first level is from zero to 25%. This component is under development and/or exists but is inappropriate for the current situation. • The second level is from 25% to 50%. This component is developed but not regularly applied • The third level is from 50% to 75%. This component is developed and has the support of the stakeholders, but funding and sustainability components are missing. • The fourth level is from 75% to 100%. This component is developed and applied regularly with the support of the stakeholders and/or financing or other elements are in place to ensure sustainability. External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 103 ANNEX C: OBSERVATION AND EVALUATION BY ORGANIZATION All indicators assessed by areas and organization relied on a point system as follows: The assessment tool graded each indicator using a 4-point scale, according to indicator definitions. • Level One: Zero to 25%. This component is under development and/or exists but is inappropriate or inadequate for intended levels of performance. • Level Two: 26% to 50%. This component is present but not consistently available. • Level Three: 51% to 75%. This component is present and has the support of the stakeholders, but the funding and sustainability elements / characteristics are missing. • Level Four: 76% to 100%. This component is present and well developed, and applied regularly with the support of the stakeholders and/or financing. Elements to support sustainability are in place. What follows is an example of scoring an indicator for one of the component: Human Resource Management Indicator: Job Descriptions • 1 point (0-25%): The organization does not have job descriptions • 2 points (26% – 50%): Job descriptions exist but are not complete; personnel do not know their job descriptions or don’t have a copy of their job descriptions. • 3 points (51% – 75%): Job descriptions are complete, staff have access to copies of job descriptions, but the descriptions are not followed, reviewed or updated • 4 points (76% – 100%): Job descriptions are complete, staff have access to copies of their job descriptions and the descriptions are adhered to and updated as needed. FHI 360 GOVERNANCE FHI 360 was able to strengthen the organizational capacities of NGOS and government structures. They reached their sustainability goals in the sense that they equipped every institution with basic documents for organizational operation. However, they did not set up a succession plan. External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 104 ADMINISTRATION FHI 360 was able to develop procedures manuals (standard operating procedures manuals) but did not set up documentation of policies and procedures for the information system. HUMAN RESOURCES MANAGEMENT FHI 360 developed along with organizations, job descriptions, transparent and coherent recruitment directives, data on vacant positions, and trained the staff in counseling, financial management and M&E. However, given that institutions’ staff management policies emanate from the government, they could not progress fully. As long as this situation exists, it would be impossible to achieve human resources management sustainability. 100% 100% 50% 75% 50% Vision / Mission Organizational Structure Composition and Responsibilities Legal Status (ONG) Succession Planning Component 1 - Governance 100% 25% Operational policies, Procedures and Systems Information Systems Component 2 - Administration External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 105 FINANCIAL MANAGEMENT ORGANIZATIONAL MANAGEMENT FHI 360 trained staff on best practices of organizational planning, but there is no process to ensure new knowledge. FHI 360 has identified the expectations of stakeholders and determined how they can work together. There are opportunities to strengthen the discussions between and among managers and staff. Note that these indicators, most of which achieved only 50%, are those that ensure the sustainability of the organization (resource mobilization, management of change) and are critical to sustainability. 75% 100% 100% 50% 25% 100% Job Descriptions Recruitment Staff Levels and Retention Personnel Policies HR Performance Management Staff Skills Development Component 3 - HR Management A complete transfer of skills took place in finance. However the institutions do not have an audit system in place, which is strongly recommended especially if the organization will receive funds from donors. The audit system will increase the credibility of the organization. 100% 100% 100% 100% 100% 25% 100% Accounting System Internal Controls Financial Documentation Budgeting Financial Reporting Audits Policies and Financial Procedures Component 4 - Financial Management External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 106 PROGRAM MANAGEMENT There are systems in place to ensure compliance with the donor requirements. FHI 360 submits reports to donors and the organizations they support on indicators. Reference sites are determined, there are no agreements with government partners, private sector and civil society to ensure that customers require treatment or other health or support services have access. A mobilization plan is in place but does not systematically engage communities and leaders in planning and decision making. Respect of gender and culture is put into practice. PROJECT PERFORMANCE MANAGEMENT The operations manuals exist as well as monitoring and evaluation plans. FHI 360 seems to follow the supervision from a distance but have developed materials that enable organizations to ensure supportive supervision. Integrated supervision could be encouraged. 75% 75% 50% 50% 50% 50% 75% 50% 50% Strategic Planning Operational Planning Resource Mobilization Communication Strategy Change Management Knowledge Management Stakeholders Involvement Internal Communication Decision Making Component 5 - Organizational Management 100% 100% 50% 75% 100% Donor Compliance Technical Reports Referral (NGO and SafeTStop Center) Community commitment (NGO) Culture, Gender and Disability Component 6 - Program Management External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 107 EVALUATION OF SECRÉTARIAT EXÉCUTIF GOVERNANCE The Secretariat has existed since 2009. When the Global Fund stopped, PEPFAR stepped in and has helped since then. A succession plan does not exist, but the decision to have a succession plan comes from the government and not a ministry. Each position has well described responsibilities. There is a technical committee that does multi-sectoral coordination. The SE is reformed and it will depend on the Prime Minister and he will be called National Agency Coordination (ANC). The office will coordinate 3 Diseases. TORs are developed. The mission and vision are established. The staff is not meeting as the ES is in transition ADMINISTRATION The SE has policies and complete and detailed operational procedures. But the procedures are not yet being applied. 75% 100% 50% 50% 100% Field Activities Monitoring Standards Quality Assurance Supervision Monitoring & Evaluation Component 7 - Project Performance Management 100% 100% 100% 100% 25% Vision / Mission Organizational Structure Composition and Responsibilities Legal Status (ONG) Succession Planning Component 1 - Governance External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 108 HUMAN RESOURCE MANAGEMENT Each position is well described. Recruitment is done by the Ministry of Health. In terms of staff retention, the SE has no strategy, but each position is doubled (two people). As it is a permanent staff, the staff motivation comes from the passion of their work. The staff is proud to be part of a great cause and the SE team is cohesive. Staff evaluation is not done in all institutions of government. The practice existed after independence and it was every two years with salary review but for now, performance evaluations are not done. Note that the lowest scored indicators are those that influence sustainability. FINANCIAL MANAGEMENT The SE has software for accounting. They have two control levels for signature but they do not have a system of internal audit, which would be attractive to donors for direct funding of the organization. In this context, financial reporting would also have to be improved. 100% 100% Operational policies, Procedures and Systems Information Systems Component 2 - Administration 100% 100% 75% 75% 50% 50% Job Descriptions Recruitment Staff Levels and Retention Personnel Policies HR Performance Management Staff Skills Development Component 3 - HR Management External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 109 ORGANIZATIONAL MANAGEMENT The SE worked with FHI 360 on strategic and operational plans. The SE is waiting for the ministerial bill that will change its name and function. The SE needs an international consultant to execute the plan developed with FHI 360. It would be better to not interrupt services. PROGRAM MANAGEMENT FHI 360 was able to equip the SE with office appliances and computer equipment. The SE will need an international consultant to review organizational policies when they implement ANC. The World Bank built the building where the offices are located. 100% 100% 100% 100% 75% 25% 75% Accounting System Internal Controls Financial Documentation Budgeting Financial Reporting Audits Policies and Financial Procedures Component 4 - Financial Management 100% 100% 75% 100% 75% 75% 75% 100% 100% Strategic Planning Operational Planning Resource Mobilization Communication Strategy Change Management Knowledge Management Stakeholders Involvement Internal Communication Decision Making Component 5 - Organizational Management External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 110 PROJECT PERFORMANCE MANAGEMENT The SE has a complete package on supervision and M & E developed by FHI 360. Supervision of staff is not well reinforced by government institutions; instead the supervision of activities exists but could be strengthened further. PROGRAMME DE LUTTE CONTRE LE VIH/SIDA (PLSS) GOVERNANCE The mission of PLSS is very clear for all staff of the program and they have a strong understanding of the latter. It is a well-defined structure that reflects its mission, goals and programs. The responsibilities of the various units that make up the program are clearly defined. PLS has three units: Coordination, Technical Assistance and Prevention. They hold weekly meetings. For the past two years, the unit has no longer received support from FHI 360 and it is affecting some their tasks. 75% 100% 75% 100% 100% Donor Compliance Technical Reports Referral (NGO and SafeTStop Center) Community commitment (NGO) Culture, Gender and Disability Component 6 - Program Management 100% 100% 75% 50% 75% Field Activities Monitoring Standards Quality Assurance Supervision Monitoring & Evaluation Component 7 - Project Performance Management External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 111 ADMINISTRATION PLSS has the capacity to develop and implement policies and procedures through a procedures manual which was revised in 2015 with support from the Global Fund. An information system does not exist, yet the procedures to find out who is in what unit and what he does are there through the tasks of each other. HUMAN RESOURCE MANAGEMENT Program staff have clear and comprehensive job descriptions, and these are well understood. With Global Fund help, the program has in place a benefits package for bonuses based on the performance of each to retain its staff. PLSS personnel are all civil servants and are affected by the ministry of health. The program does not have a formal plan for the management of its workforce. There is no approach to the succession plan. PLS is supporting on-the-ground providers. They are training on counseling and screening. The staff wanted to see a career development plan in place or an exchange plan to see the experiences of other countries. The staff at the centers is not stable (transfer, illness, newly hired) that makes it difficult to keep trained staff on site. Within PLSS, the staff is stable and motivated because they have bonuses but no competency development plan is in place. 100% 100% 25% 100% 50% Vision / Mission Organizational Structure Composition and Responsibilities Legal Status (ONG) Succession Planning Component 1 - Governance 100% 50% Operational policies, Procedures and Systems Information Systems Component 2 - Administration External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 112 FINANCIAL MANAGEMENT Financial system, program policies and procedures exist. PLSS receives no cash from FHI 360, but the staff has been trained on the FHI 360 financial procedures. There is no internal audit. If a government to government system is put in place, a control system should be installed. ORGANIZATIONAL MANAGEMENT The program has a strategic plan and an operational plan reviewed with FHI in 2015. They have weekly meetings between the different units. They do not have the ability to write proposals to win contracts. The program still has to improve in managing change, as soon as FHI began to give support to the Executive Secretariat they could not get by. 100% 100% 75% 25% 25% 25% Job Descriptions Recruitment Staff Levels and Retention Personnel Policies HR Performance Management Staff Skills Development Component 3 - HR Management 100% 100% 100% 100% 100% 25% 100% Accounting System Internal Controls Financial Documentation Budgeting Financial Reporting Audits Policies and Financial Procedures Component 4 - Financial Management External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 113 PROGRAM MANAGEMENT The program has regulations that compliant to donors. They produce technical reports per unit before the M & E unit compiles the general quarterly report interim and annual. With the knowledge received from FHI, the program leads community activities. PROJECT PERFORMANCE MANAGEMENT A monitoring and evaluation plan is available with the support of experts from FHI. Supervisory activities are conducted with the Global Fund. Quality monitoring tools have been developed with assistance from FHI and are available at the program level 100% 100% 25% 50% 25% 75% 25% 50% 75% Strategic Planning Operational Planning Resource Mobilization Communication Strategy Change Management Knowledge Management Stakeholders Involvement Internal Communication Decision Making Component 5 - Organizational Management 75% 75% 50% 100% 100% Donor Compliance Technical Reports Referral (NGO and SafeTStop Center) Community commitment (NGO) Culture, Gender and Disability Component 6 - Program Management External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 114 UNION NATIONAL DES FEMMES DJIBOUTIENNE (UNFD) GOVERNANCE The National Union of Djiboutian Women (UNFD) is a non-profit NGO (Law 1901) chaired by the first lady of the country, His Excellency Mohamoud Kadra Haid. Established on 30 April 1977. The vision and UNFD missions are very clear to everybody. UNFD has a chart that was reviewed with international experts from FHI. UNHCR supports a refugee program by UNFD channel. ADMINISTRATION The UNDF has the ability to develop and implement policies and procedures, and has the quality of administrative systems. A procedure manual was developed with support from FHI. 100% 100% 75% 75% 100% Field Activities Monitoring Standards Quality Assurance Supervision Monitoring & Evaluation Component 7 - Project Performance Management 100% 100% 100% 100% 75% Vision / Mission Organizational Structure Composition and Responsibilities Legal Status (ONG) Succession Planning Component 1 - Governance External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 115 HUMAN RESOURCE MANAGEMENT Job Descriptions are available and approved by the committee before recruitment. These are reviewed according to the new missions assigned to each. Bonuses based on responsibilities exist. The staff appraisal system does not exist FINANCIAL MANAGEMENT Accounting personnel were all trained on financial procedures by FHI, even the staff who are not on the FHI specific program. UNFD praised the FHI financial system that allowed them to set up a system they use with all other donors. There is no internal audit. They call for consultants to prepare the budget. 100% 25% Operational policies, Procedures and Systems Information Systems Component 2 - Administration 75% 100% 75% 50% 25% 75% Job Descriptions Recruitment Staff Levels and Retention Personnel Policies HR Performance Management Staff Skills Development Component 3 - HR Management External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 116 ORGANIZATIONAL MANAGEMENT Work on the strategic and operational plan of the organization began with experts from FHI, but it is not yet finalized. Weekly meetings are held with the organization coordinator. Quarterly reports are available. It would also be better to increase the participation of stakeholders. PROGRAM MANAGEMENT Technical reports are developed by different project managers. 100% 100% 75% 75% 100% 25% 75% Accounting System Internal Controls Financial Documentation Budgeting Financial Reporting Audits Policies and Financial Procedures Component 4 - Financial Management 50% 100% 75% 75% 75% 75% 25% 75% 75% Strategic Planning Operational Planning Resource Mobilization Communication Strategy Change Management Knowledge Management Stakeholders Involvement Internal Communication Decision Making Component 5 - Organizational Management External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 117 UNFD has a capacity of social mobilization with the community, and a solid experience in the field of small projects generating revenues of activities. Lack of regulatory compliance with donor funding could be a blockage. PROJECT PERFORMANCE MANAGEMENT UNFD organizes training supervision in the field. They have monitoring and evaluation of performance against standards and against indicators. They monitor progress for results. MASSABA GOVERNANCE The association was created in 1992 to help children exposed to AIDS. They have a status, vision and an organizational chart. The association works with road and a health center. This is a community center and government. Their mission is to fight AIDS and poverty. The association trains young, men, women, Imams and Ethiopian. 25% 100% 100% 100% 100% Donor Compliance Technical Reports Referral (NGO and SafeTStop Center) Community commitment (NGO) Culture, Gender and Disability Component 6 - Program Management 75% 100% 75% 75% 100% Field Activities Monitoring Standards Quality Assurance Supervision Monitoring & Evaluation Component 7 - Project Performance Management External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 118 ADMINISTRATION Sustainability is established even if aid stopped, the association will continue. According to FHI, the Board should be reinforced. HUMAN RESOURCE MANAGEMENT They have an organogram but they don’t take notes of meetings. The President would like to be trained on how to take notes. They were able to reduce stigma in the community. They teach about prevention and how the virus is transmitted. They noticed a decline in teenage pregnancies in the community. Among 10 people detected only 3 come back to get the results according to the president. 100% 100% 100% 75% 50% Vision / Mission Organizational Structure Composition and Responsibilities Legal Status (ONG) Succession Planning Component 1 - Governance 75% 75% Operational policies, Procedures and Systems Information Systems Component 2 - Administration External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 119 FINANCIAL MANAGEMENT They have a system of three signatories implemented by FHI. They monitor each other even if there is no audit as such. To develop a budget, they need help. ORGANIZATIONAL MANAGEMENT They cannot take notes during meetings. They say they talk a lot and no one writes. 75% 100% 75% 75% 50% 75% Job Descriptions Recruitment Staff Levels and Retention Personnel Policies HR Performance Management Staff Skills Development Component 3 - HR Management 100% 100% 100% 75% 100% 50% 75% Accounting System Internal Controls Financial Documentation Budgeting Financial Reporting Audits Policies and Financial Procedures Component 4 - Financial Management External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 120 PROGRAM MANAGEMENT They have a strong community involvement. The reports are prepared by their collators and not them. PROJECT PERFORMANCE MANAGEMENT They have a monitoring plan developed by FHI. This association still needs technical support to implement supportive supervision and monitoring of field activities. 75% 75% 75% 75% 75% 75% 75% 100% 100% Strategic Planning Operational Planning Resource Mobilization Communication Strategy Change Management Knowledge Management Stakeholders Involvement Internal Communication Decision Making Component 5 - Organizational Management 75% 50% 100% 100% 100% Donor Compliance Technical Reports Referral (NGO and SafeTStop Center) Community commitment (NGO) Culture, Gender and Disability Component 6 - Program Management External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 121 RESEAU DES PERSONNES VIVANT AVEC LE VIH (RNDP+) GOVERNANCE The RNDP + is an NGO governed by the law and the text of the law of 1 July 1901 and the Decree of 16 August 1901 on freedom of association in the Republic of Djibouti. The network has received support from international experts of FHI in order to have stability and guidelines. The association has an organogram and a board of directors. FHI helped to establish its missions. They are the only MENA to have a law. ADMINISTRATION The network received the capacity to develop and implement policies and clear procedures for all members. The network is made of 6 associations. They will attend New York general assembly of people living with HIV. 50% 75% 75% 50% 100% Field Activities Monitoring Standards Quality Assurance Supervision Monitoring & Evaluation Component 7 - Project Performance Management 100% 100% 100% 100% 25% Vision / Mission Organizational Structure Composition and Responsibilities Legal Status (ONG) Succession Planning Component 1 - Governance External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 122 HUMAN RESOURCE MANAGEMENT The office staff have well defined and clear Job Descriptions. There are 500 members. The members are people living with HIV / AIDS. Members make a monthly contribution. The members love the association because it is an opportunity to meet other people living with HIV. Personnel policies do not exist. Other indicators such as performance management and skills development will prevent the sustainability. FINANCIAL MANAGEMENT FHI supported the network to establish an accounting system. Financial report is done at the end of each activity and they are validated by the financial department of FHI. The association has an ability to meet the needs of management and the requirements of donors. With the support of experts from FHI the association has developed proposals and they are solicited by other donors including the Japanese government and an international association of PLWHA. Without internal controls, a good financial reporting and auditing, it would be difficult to manage funding. 75% 75% Operational policies, Procedures and Systems Information Systems Component 2 - Administration 75% 100% 100% 25% 50% 75% Job Descriptions Recruitment Staff Levels and Retention Personnel Policies HR Performance Management Staff Skills Development Component 3 - HR Management External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 123 ORGANIZATIONAL MANAGEMENT They have a strategic plan and an action plan. They have a board of directors. They received the technical support from FHI 360 for the financial mobilization. There is an internal communication, they organize many meetings. And the decision is made by the Board and then by the committee. An improvement in the area of organizational management will contribute to make them sustainable. PROGRAM MANAGEMENT The network has the ability to implement comprehensive programs that meet local needs and priorities by examining compliance with the requirements of donors, management of sub-grants with partners, technical reports and whether its services comprehensive health meet the needs of target populations. Nevertheless they must be supported more to the network 100% survive. 100% 25% 75% 100% 50% 25% 75% Accounting System Internal Controls Financial Documentation Budgeting Financial Reporting Audits Policies and Financial Procedures Component 4 - Financial Management 75% 100% 75% 50% 50% 75% 75% 100% 100% Strategic Planning Operational Planning Resource Mobilization Communication Strategy Change Management Knowledge Management Stakeholders Involvement Internal Communication Decision Making Component 5 - Organizational Management External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 124 PROJECT PERFORMANCE MANAGEMENT Formative supervisions are conducted by technical coordinators. The data collection tools were developed with technical assistance from FHI. They have a monitoring and evaluation plan developed by FHI 360 but they do not have a staff responsible for M & E. ASO GOVERNANCE In 2014, three associations (women, religious and Young) met for designer ASO as implementing actor of HIV prevention activities and manager of resources and background from FHI 360. A tender was launched for the recruitment of a qualified coordinator for the supervision and implementation of HIV prevention activities. The Office of the Association was elected at a general meeting with all members. The mandate of the Bureau is renewable annually. The vision of the association is to promote youth awareness and populations on HIV prevention through sport and cultural activities. However the President has not yet shared the status of the ASO that contains the vision and mission. 75% 50% 100% 100% 100% Donor Compliance Technical Reports Referral (NGO and SafeTStop Center) Community commitment (NGO) Culture, Gender and Disability Component 6 - Program Management 75% 100% 75% 50% 100% Field Activities Monitoring Standards Quality Assurance Supervision Monitoring & Evaluation Component 7 - Project Performance Management External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 125 ADMINISTRATION It is composed of a President and Vice President, Secretary General and his deputy, General Treasurer and his deputy controller of accounts. And 80 peer educators (young people (men and women, religious) and 10 sex workers were trained to raise awareness about HIV prevention. The activities that have been carried both to the community and truck drivers of the Ethiopia-Djibouti corridor are: chats sessions and film screening HUMAN RESOURCE MANAGEMENT The association has only paid staff (the coordinator) and other members are unpaid volunteers for various activities. The description of the coordinator’s tasks is well defined and as well as the role of the Bureau of ASO. The performance of players in the ASO is set to achieve the targets for awareness. 100% 100% 100% 50% Vision / Mission Organizational Structure Composition and Responsibilities Legal Status (ONG) Component 1 - Governance 75% 75% Operational policies, Procedures and Systems Information Systems Component 2 - Administration External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 126 FINANCIAL MANAGEMENT The association has a bank account and check books kept by the coordinator. The signatories are the President and the General Treasurer after verification of compliance queries planned activities. Manuals, procedures, financial and accounting reports are available and checking expenditure and revenue is provided by the controller and reports are shared during business meetings and general assembly. The financial resources of the association come from donations and membership dues. ORGANIZATIONAL MANAGEMENT Business planning is done with the support of FHI 360 and both parties agree on the budget for the implementation of field activities. The coordinator ensures communication with peer educators and members of the office of the ASO. 100% 100% 75% 75% 50% 75% Job Descriptions Recruitment Staff Levels and Retention Personnel Policies HR Performance Management Staff Skills Development Component 3 - HR Management 100% 100% 100% 75% 100% 50% 75% Accounting System Internal Controls Financial Documentation Budgeting Financial Reporting Audits Policies and Financial Procedures Component 4 - Financial Management External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 127 PROGRAM MANAGEMENT The use of funds is done according to procedures established by FHI 360. The staff has been trained on finance management procedures and resources. Technical activity reports are available. The peer educators are members of the community and are active in raising public awareness about the risks of HIV transmission and prevention methods. PROJECT PERFORMANCE MANAGEMENT Data collection tools exist and supervision reports are available. Data reports are integrated in the supervision reports. However a quality assurance tool is not available. The coordinator ensures the monitoring activities of condom distribution and outreach. The coordinator was also trained in logistics management of inventories of condoms. 75% 75% 75% 75% 75% 75% 75% 100% 100% Strategic Planning Operational Planning Resource Mobilization Communication Strategy Change Management Knowledge Management Stakeholders Involvement Internal Communication Decision Making Component 5 - Organizational Management 75% 75% 75% 100% 100% Field Activities Monitoring Standards Quality Assurance Supervision Monitoring & Evaluation Component 6 - Program Management External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 128 75% 75% 75% 100% 100% Field Activities Monitoring Standards Quality Assurance Supervision Monitoring & Evaluation Component 7 - Project Performance Management External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 129 ANNEX D: SUMMARY OF KEY FINDINGS BY EVALUATION QUESTION Questions Data Sources Methods/Sampling Data Analysis Key Findings 1) To what extent has ROADS II’s Technical Assistance (TA) to NGOs and system in managing the NGO network, resulted in increased capacity to carry out their mandate to deliver HIV/AIDS Services targeted at vulnerable populations? • Project Annual and Quarterly Reports • Strategic Operation Plans •National Strategic Plan •Management Assessment Questionnaire •Document Review • Interviews •All managers of supported NGOs and MOH and other HIV/AIDS partners will be interviewed •Descriptive (frequencies, Cross tabs) •Desk review • Governance capacity has significantly improved related to vision/mission, organizational structure, composition and responsibilities and legal status; • On administration, NGOs show strength in operational policies, procedures and systems; • Regarding financial management and skills, organizations have improved in accounting systems, policies and financial procedures, budgeting, financial documentation and internal controls; • Acceptable standards have been met in all organizational management components but improvement is needed regarding resource mobilization; • In program management, organizations improved their ability to promote community commitment and to consider culture, gender and disability in the service provision; • Performance management systems enable supported organizations to improve their ability to monitor and evaluate their activities and to comply on the standards. 2) To what extent has ROADS II’s technical assistance (TA) to the Executive Secretariat and to the National HIV/AIDS control program increased their capacity to manage the nation’s sustainable HIV/AIDS prevention, care and support interventions for key and other vulnerable populations? • Inventory Assessment Questionnaire •Management Assessment Questionnaire •Observation • Interviews • Systematic Random sampling of supported Health facilities •All health providers of selected Health facilities •Descriptive (frequencies, Cross tabs) • The establishment of 3 Technical Working Groups as platforms to discuss and exchange the project achievement progress and to improve intervention strategies and approaches; • An increase from 1 in 2013 to 4 in 2014 of the number of the coordination meetings among implementers led by the Government AIDS, TB and Malaria Executive Secretariat to discuss the progress and challenges in the implementation of the activities and to propose solutions to better organize and coordinate efforts; • Capacity of the PLS Sante to train and supervise External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 130 Questions Data Sources Methods/Sampling Data Analysis Key Findings the health providers and community and social workers on HCT and treatment services 3) To what extent has the capacity strengthening resulted in reduced risk of HIV/AIDS among key and other vulnerable populations? •HMIS Routine data •Health Facility Assessment questionnaire; • Focus group discussion •Document Review • Interviews of health providers and beneficiaries • Systematic Random sampling of supported Health facilities •Descriptive (frequencies, Cross tabs) •Desk review • Secondary data analysis • An increase from 6 in 2013 to 9 in 2014 of the implementation of HCT sites to improve the quality of services and also increase the awareness of HIV prevention and accessibility of services; • An understanding of the importance of conducting (HCT): identifying HIV positive persons and prioritizing the prevention of new infections were stated by 81% and 85.1% of health providers, respectively; • An increase from 96,280 in 2013 to 142,066 in 2014 and to 178,780 in 2015 of the condoms distributed by the project to promote safe sex. 4) How successful has the program been in reducing HIV/AIDS transmission and improving health of the targeted beneficiaries? • Project Annual and quarterly reports •HMIS Routine Data • Focus group Assessment questionnaire •Document Review • Focus Group •The association of PLHIV and a convenient sample of civil society associations supporting HIV/AIDS services • Secondary Data Analysis •Desk review • An understanding of the importance of conducting (HCT): identifying HIV positive persons and prioritizing the prevention of new infections were stated by 81% and 85.1% of health providers, respectively; • An awareness of adolescents, truck drivers and sex workers of the importance of knowing their HIV status and the use of condoms to prevent future infections and the need to support HIV positive persons; • An increase of the acceptability of PLWHIV with HIV and their compliance to ARV medication for a healthy and long life style; • An increase in the number of people accepting and receiving the HCT services (from 20% in 2013 to 46% in 2015); • A decrease in new HIV infections from 36% in 2013 to 30.6 % in 2015; • An increase in the detected HIV positive cases from among those tested (23% to 41.9% from 2013 to 2014); • An increase in the provision of ARV to HIV patients since 2011.from 1,082 to 1,945 in 2015 5) Are the original project assumptions still valid and will they provide sufficient guidance for appropriate programmatic and technical assistance decisions? • Project Annual and Quarterly Reports •Document Review •Desk Review • Building capacity of civil society, NGOs and government institutions (Executive Secretariat, PLS) to manage and implement HIV interventions has resulted in: o Increased knowledge and capacity of health External Evaluation and Performance Monitoring Services for ROADS II, Final Report AID-603-TO-16-00001 131 Questions Data Sources Methods/Sampling Data Analysis Key Findings providers and community workers to deliver and promote HCT, treatment and care services; o Increased awareness of communities on the importance of HCT services and the prevention of HIV o Increased organizational capacity to manage HIV programs and services o Increased use of HCT services o Increased numbers of people on ARV 6) What are the strengths and weaknesses of the ROADS II management, coordination and communication processes? • Project Annual and quarterly reports •Management Assessment questionnaire •Document Review • Interviews • 2/3 managers of supported NGOs/MOH/ partners will be interviewed •Descriptive (frequencies, Cross tabs) •Desk review Strengths • Increase of civil society, NGOs and government institutions (Executive Secretariat and PLS) to manage, coordinate and implement HIV/AIDS services; • Established 3 technical working groups to exchange, discuss and review HIV intervention progress and strategies; • Increased from 1 to 4 coordination meeting led by the Executive Secretariat to discuss the progress and the challenges of HIV interventions. Weaknesses • Need to build stronger relations between the community implementing partners with health facilities to better support and supervise the community workers and to integrate their data reports into the HMIS; • Additional resources are needed. Lack of review reported data regarding why HIV positive cases have decreased to ensure positive cases have healthy and long life style and prevent transmission; • Need to create bulletins and newsletters to disseminate HIV intervention results to the general public.