This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Southern Hemisphere. PERFORMANCE EVALUATION Of the USAID Botswana HIV and AIDS civil society strengthening project–Maatla i USAID BOTSWANA PROJECTS PERFORMANCE EVALUATION– MAATLA CIVIL SOCIETY STRENGTHENING PROJECT TO SUPPORT HIV AND AIDS AND RELATED SERVICE DELIVERY IN BOTSWANA November 24, 2015 Evaluation Mechanism Number: RFTOP number SOL-674-15-000007 Prime Partner: Southern Hemisphere Team Leader, Project Manager Maatla: Dena Lomofsky DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. ii Acknowledgements Southern Hemisphere would like to acknowledge the support and guidance of both USAID in Pretoria and Gaborone. FHI360 was extremely helpful in providing documents and supporting the fieldwork. The teams from NCONGO, FHI360, BOCAIP and BONEPWA, especially the coordinators and centre supervisors who facilitated the field work by directly helping with the mobilisation of participants. We would also like to thank all the key respondents, including those from the Government of Botswana, BOCONGO, BONASO, WUSC, FHI360, BOCAIP and BONEPWA+ as well as the community based volunteers who took part in the evaluation for making time to partake in this exercise. Southern Hemisphere would also like to thank our evaluation and project management team for ensuring the evaluation was completed to the desired quality, ethical standards and in time. We would also like to thank our expert for reviewing the instruments and the draft report. Executive Summary The HIV/AIDS epidemic in Botswana has been reported as the most critical development challenge in the country’s history and has been declared a national emergency since the first reported case in 1985 (Government of Botswana – GoB, 2010). Botswana has the third highest global adult prevalence rate (Central Intelligence Agency, 2015) of 19.03%, which increases to 50% for women between the ages of 35 and 39 (GoB, 2013). The GoB has committed itself to achieving zero new infections by 2016. Despite the important role that civil society organisations play in strengthening and complementing government programs, critical capacity gaps were identified which led to a Civil Society Capacity Building Strategic Framework 2010 – 2016. The strategy was developed by the Botswana Network of AIDS service organisations (BONASO) and the National Coordinating Council for AIDS (NACA), supported by USAID and PCI. It is this strategy that informed the design and relevance of the Maatla project, the implementation of which was managed by FHI360 through a cooperative agreement with USAID which was signed in 2011. RATIONALE OF THE PROJECT Maatla’s theory of change assumes that by building the capacity of CSOs and local government coordination structures, the structures will be more capable of supporting the HIV/AIDS response and broader health needs in Botswana (USAID, 2015). The critical assumptions linked to Maatla’s goal, include: 1. Viable government entities at the district level for adequate CSO coordination 2. Resources available from other sources to sustain the civil society sector 3. Willingness and interest to assist BONASO by all stakeholders Maatla has three main results areas as reflected in the effectiveness section below. The project was implemented by eight different organisations divided into Core sub-partners (who provided organisational and technical capacity building and model development) and sub-partners (who were recipients of these efforts and were intended to deliver services and cascade capacity building to their members). The various key roles for the partners were: Core sub-partners: FHI360 – overall management, technical support; WUSC – organisational development support; NASTAD – district level strengthening and NCONGO – district based coalition model. Sub-partners: BOCONGO – organisational capacity building; BONASO – Technical capacity building; BONEPWA+ - HIV/AIDS service delivery; BOCAIP – HIV/AIDS service delivery. EVALUATION PURPOSE The evaluation’s purpose is to assess: (1) how well the activities are being implemented so far, whether according to plan (2) to inform program design for the next round of USAID supported programs, and (3) to contribute to evidence of what works or doesn’t work in improving health outcomes among beneficiaries and communities. Evaluation questions 1. How likely is each activity to achieve its desired end results, as defined in each activity’s results framework? 2. What modifications, if any, to the strategic approach and theory of change have potential to substantially increase program effectiveness? i 3. To what extent has each activity been implemented according to activity design? Where there have been deviations or shortfalls, what are the causes? 4. To what extent has USAID fulfilled its role in the activities by providing adequate funding, responsive managerial oversight and technical support, and assistance in solving problems in implementation as they arise? 5. What actionable, prioritized lessons have been learned for the design of potential follow-on activities, i.e., what should continue and what should change from the current activities in a potential future activity? EVALUATION METHOD The evaluation took place between May and November 2015 in Gaborone, and in the Okavango and Ngami districts. The Maatla project is also conducted in Chobe and Ghanzi districts. The evaluation triangulated data from document review, 51 semi-structured qualitative key informant interviews (KIIs) with key project stakeholders, five focus group discussions, organizational and technical capacity assessments (11) and data from the Maatla monitoring systems. The qualitative data was analysed using NVIVO 10 and coding reports were produced for thematic analysis. The capacity assessments were analysed in excel. This final report incorporates feedback received from USAID and FHI360. FINDINGS - EFFECTIVENESS Evaluation questions: • How likely is each activity to achieve its desired end results, as defined in each activity’s results framework? 3. To what extent has each activity been implemented according to activity design? Where there have been deviations or shortfalls, what are the causes? Result 1: Strengthened technical competence and organizational capacity of NGOs and networks to support field offices, affiliates and member organizations with expanded and strengthened delivery of strategic and high quality HIV/AIDS services A needs driven process for capacity assessment was developed, which built on previous capacity building programs for CSOs in Botswana, such as AED. The organizational needs were identified through the Organizational Capacity Assessment Tool (OCA and OCA-Lite)1 and the Technical Capacity Assessment Tool (TCA), which were developed collaboratively with the sub-partners concerned and piloted for relevance. The OCA process was led by WUSC, and the TCA by FHI360. Organisational capacity assessment In terms of improvements, all organisations have shown improvements at a National level, with BOCAIP, BOCONGO and NCONGO exceeding the threshold of 80%. BONASO remains the most challenged organisation. Both BONASO and BONEPWA+ had very low baselines as they were in unstable positions in the early phase of the project. Technical capacity assessment The radar graph that follows displays comparative scores for all TCA domains by organization from the data collected for this evaluation. The radar graph shows that all organizations have good scores of 80 percent and above and have a good balance between domains. 1 The OCA-Lite was developed for branches of national organizations and smaller organizations in the districts that did not require the full spectrum of criteria. ii TCA Ratings by Domain—National—BONEPWA+ and BOCAIP-External National organization branches had only a few partial criteria ratings as their TCA ratings were just short of perfect. Capacity building strategies A range of different capacity building strategies were used to support the organizations, namely mentoring, short- and long-term training, organizational development advisors, consultants, provision of policies and procedures, and placing of interns. The OD capacity building program was responsive and based on needs. The Governance Training was very well structured and was the most successful component, it had cascading built into it through a case study approach and included both classroom and practical components. Besides the capacity building provided, the biggest enabler was actually the funding that the organisations received to hire staff which enabled them to do their work and design the necessary organisational systems (e.g. finance manager, communications officer, program officer). A few key factors affected the successful attainment of result 1 and 3 (three was essentially the graduation component for the sub-partners). • The CSO sector in Botswana is in a very difficult funding environment, especially for network organisation. The underlying assumption that the sub-partners would have other funding did not materialise. This affected their ability to hire staff and implement projects. • Progress was hampered by the small staff compliments of the recipient organisations. The same people had to attend training, participate in partner meetings and implement programs. • There was an over-supply of capacity building for these small organistions, especially since FHI360 and WUSC were both trying to build technical and organisational capacity respectively and simultaneously, and coordination between them was not optimal. • Capacity building progress was not measured around key milestones, and transfer of learning into organisational practice was not always ensured for each domain. • The cascade model did not work well with the other domains (besides governance). • At times, the organisational development advisors were asked by the sub-partners to carry out actual organisational functions, and were not able to provide the OD support that was envisaged. “We should have been allowed to recruit more people to be capacitated from our side—there were five people from their side to capacitate only one or two people on our side. There was no balance.” (KII sub-partner). This view was corroborated in KIIS with core sub￾partners. iii • Limits on recruitment also limited effectiveness. There was a tension between whether the project would allow the organisations to hire staff, or whether FHI360 would provide technical assistance. • Maatla intended to take organisational development beyond basic systems and structures to encourage more enterprise thinking in organisations focused on resource mobilisation, income generation and enhanced membership value and to build an enabling environment for resource mobilisation. However, the loss of BOCCIM as a mentoring and private sector partner was a blow to the program, and inhibited the attainment of this result. This affected the financial sustainability of the sub-partners. Technical Capacity Strengthening The main technical contribution of the Maatla project at national level has been to help translate government policies and guidelines into HIV/AIDS programs and to develop models around these that were being implemented in the districts. This is a positive contribution to the health system as long as the implementing organizations are able to continue implementing them. Methodologies A number of new methods were tested or implemented under this objective, namely: - Community based TB Care (the government had been promoting clinic based TB DOT services) - Integrated family centered services through enhanced referrals and linkages among CSOs and government departments - Positive Health Dignity and Prevention (PHDP) minimum package offered by BONEPWA+ - Community volunteers working with MoH to strengthen adherence to treatment for people on ARVs (learning from community based TB adherence models) - Gender and HIV mainstreaming (assessment and training) - Combination Prevention Implementation Model (CATCH) The technical capacity building component was constrained because BONASO, the key sub-partner that was supposed to receive technical capacity support and cascade it down the districts, did not have the technical staff nor the commitment to do so. However, the BOCAIP and BONEPWA+ were responsive and FHI360 worked closely with them to develop and disseminate guidelines, manuals and training for service provision. The cascade model worked better with the technical component because BONEPWA+ and BONASO had offices in the districts, but it required a lot of follow up support from core sub-partners and the provision of TAs to ensure that it happened. Result 2: Developing and implementing a model to strengthen district level systems (CSOs and local government) to provide and coordinate HIV/AIDS services in hard to reach areas The intention of Result 2 was to develop and implement a model to strengthen district level systems to provide and coordinate HIV/AIDS services in hard-to-reach areas. Although no integrated model was developed, the program operated according to four models of which only one was documented. Organizational and technical capacity assessments were carried out of individual CSOs and organizational development plans were developed. Based on these assessments and plans, capacity building was implemented, strategic plans and constitutions developed, and roles and responsibilities between boards and management clarified. As a result organizations have been audited which have given them a better possibility for successful resource mobilization. Technical M&E support was provided and as a result good quality data was collected. One of the enabling factors was that the FHI 360 M&E officer was placed in the district. However, it is doubtful whether iv the M&E efforts by the Maatla program has equipped the district organizations sufficiently so that they can produce their own M&E plans and continue collecting quality MIS data. It is a concern that FHI 360 was unable to establish procedures for accountability purpose so all organizations could manage their own funding. Furthermore despite efforts by NCONGO in facilitating proposal writing, it is doubtful if organizations are able to financially sustain themselves. As part of the Maatla program, NCONGO strengthened its membership engagement for Ngami and Okavango CSOs and thereby increased its authority as a network organization. It furthermore established and supported district coalitions in the Chobe and Ghanzi districts with the support of WUSC ODAs placed in those districts. However, concerns were raised as to how sustainable those structures would be once the project is closing, despite being registered and having the support of NCONGO. Capacity assessments of the DMSACs took place and district level government coordination and collaboration strengthening plans were developed and implemented. Roles and responsibilities of DMSAC members were also clarified. Reporting by the CSOs to DMSAC improved through the consolidated report produced by NCONGO (and other coalitions) on behalf of their members. Tools, manuals and standard operating procedures were developed and applied. District Service inventories and referral tools facilitated better referrals and follow up. As a result, government officials and CSOs are collaborating. The mid-term assessments conducted revealed that all 4 districts have reached a minimum threshold or target set for government and CSO coordination and collaboration (70%) and on average districts improved by 30%. Overall, results of the Maatla program are that there are decreased default rate on treatments (both TB and ART treatments) and there is increased awareness on HIV and AIDS prevention in the communities. Recruiting volunteers from the communities who are trained and intensively supervised by organisations like BONEPWA+ and BOCAIP and using tools for CSOs such as standard operating procedures and communication guide on community education have all been enabler s for the success of the program. However, the sustainability of this program is questionable as the existence of some of the coalitions, CSOs and their volunteers has been depending on Maatla funding. Despite efforts, a proper sustainability plan has not been implemented and with the collapse of the PPP initiative it is unlikely that the success of the program will be sustained. Result 3: Strengthened Institution(s) Able to Provide Sustainable Capacity Building to Civil Society The intention of this sub-objective was to begin transferring capacity and responsibility for technical and organizational capacity building to BONASO and BOCONGO, respectively. Overall, the findings from KRA 3 suggest that while BOCONGO made good strides and took full advantage of their participation in Maatla, they are not yet at the point where they are strong enough technically or financially to carry out this role—at least not in the HIV and AIDS sector. Funding that they have received is for other sectors (e.g. climate change). This raises the question about whether BOCONGO was a good partner for a PEPFAR project in this first place, since their mandate is much broader than HIV/AIDS. Their participation could, however, help with mainstreaming HIV/AIDS and gender into other sectors. BONASO is now in a position to partner with NACA as a funded organization, which is positive, but they are not able to fulfil the technical lead as intended by Maatla. The re-pivoting of PEPFAR towards service delivery and away from organisational strengthening also affected the progress of results 1 and 3 especially for BOCONGO and BONASO. The building blocks that were put in place with BONASO and BOCONGO for them to carry the project objectives out were not developed further because this aspect of the project was closed out in FY14. This change also affected the sustainability of the capacity strengthening intervention. Further, the initial project design saw sub-partners graduating to direct USAID assistance, but this opportunity fell away when USAID/PEPFAR changed project direction. v Design The project was well conceptualised but the challenges for design related to the complicated structure of the project team, there were too many core-sub partners trying to work with sub￾partners with limited capacity to absorb their efforts, as described above. The other main design challenge was that FHI360 had not imagined that the project would have had such a large service delivery component, and their team was geared for capacity building. How this misunderstanding came about is not clear, but USAID picked it up in FY2 and FHI360 managed to re-orientate Maatla. Relevance The Maatla project was relevant to both the GoB’s National Strategic Framework for HIV/AIDS II (NSFII), and it’s strategy for strengthening of CSO AIDS Service Organizations. While the project clearly boosted the efforts of the GoB to address HIV/AIDS, the challenge is that USAID and the GoB tend to work in parallel. This threatens the sustainability of the intervention because government systems may not be able to absorb the outputs of the project once it is over. Role of USAID Evaluation question: 4. To what extent has USAID fulfilled its role in the activities by providing adequate funding, responsive managerial oversight and technical support, and assistance in solving problems in implementation as they arise? USAID played a good oversight role and picked up a number of challenges with implementation, such as the lack of implementation of the service delivery component during FY2012, and that Maatla had an insufficient youth focus. The insistence on including a service delivery component did strengthen the program as, in reality, organizations such as BOCAIP and BONEPWA+ may not have had sufficient resources to implement programs, and as such the core sub-partners would have been providing capacity building in a vacuum. However, this did pose a challenge for FHI360, which had allocated resources and hired a team to implement an OD program. The wisdom of USAIDs directive to close-out the OD component of the program has also been questioned in this evaluation. It may have been an expedient decision with regard to how USAID Botswana responds to its corporate needs, but threated the effectiveness and sustainability of the original project objectives. Another challenge has been the extent to which USAID brings the GoB into the center of its programming rather than offering parallel programs. This is also raised elsewhere in the report in more detail. Both of these issues are discussed in detail in the relevance section. The program seemed to have been reasonably funded, although how this was costed is not clear to the evaluators as we were not privy to the bidding process. The main challenge was that implementing organizations still seem to have been short-staffed, the over-reliance on volunteers to deliver program objectives is inefficient in the long-run due to their high turn-over rate, and inadequate consideration of what it meant for costing to work in hard-to reach areas in terms of travel time and resources available in these areas. vi LESSONS LEARNED Evaluation question: 5. What actionable, prioritized lessons have been learned for the design of potential follow on activities, i.e., what should continue and what should change from the current activities in a potential future activity? The following lesson learned can be extracted from this evaluation Result 1 • The capacity building components are more effective when delivered in a structured way as demonstrated by the Governance training. E.g. short term training should always be followed by on-site mentoring with a practical component, and cascading can be built in as well. • Long term capacity building requires both shared and embedded ODAs, depending on the domain and the person receiving support. • Do not overload capacity building recipients as they will not be able to absorb and implement what they are learning. • OD activities need to be incentivised and contained in work plans to ensure progress, otherwise people will always prioritise the demands of their workload. • Ensure that organisations are able to hire enough staff to implement the program as designed. • implementable programs and services. These are described under result 2. Result 2 • Capacity building addressing the gaps identified in the capacity assessment and followed up with extensive mentoring on site is a useful model to ensure that organizations are truly developed to provide service delivery. • Financial, technical and organizational support of organizations, coupled with standard operating procedures, guidelines, manuals and forms/tools, are enablers for efficient service delivery. • Having an FHI TA M&E officer based in the district is an enabler for ensuring that data collected by CSOs is validated and that the reports are of good quality. • An established and capacitated coalition such as NCONGO can, with technical support from FHI360 and WUSC ODAs, facilitate the establishment of coalitions in other districts. However, it is crucial to have ODAs and TAs based there. • Having a coalition such as NCONGO consolidating the CSO report for DMSAC ensures a better quality report and as a consequence improved respect and recognition of the importance of the CSO sector by government. • A multi-sectoral approach to addressing HIV and AIDS is useful. Having a capacitated platform such as the DMSAC, where stakeholders from government and CSOs have a clear understanding of roles and responsibilities, is an enabling factor. It is, however, important that government representation on DMSAC consists of senior decision makers. • A district service inventory is a useful tool for facilitating referrals. Also developing a referral form together and providing training in its application ensures greater understanding of it. It is crucial to have follow-up as part of the referral system. Likewise it is important that the health facilities provide feedback to the CSOs on the referral. vii • Using capacitated volunteers from the same communities as the patients and supervised by capacitated CSOs is a useful model to extend health services to hard-to-reach areas. This improves the linkage between communities and health facilities. It is important to facilitate the trust building between health facilities and the volunteers. Result 3 • The main lesson learned was that it is not realistic to take an organization in crisis, such as BONASO, and expect to take leadership in a sector. This needs the full cooperation of the organization’s leadership to buy-in to the role envisaged for them in the project and to manoeuvre strategically within this. Unfortunately, the leadership of BONASO and FHI360 did not manage to build a cooperative relationship in this regard. • The other lesson was that the process for building organizational capacity to take over the project should have been more clearly defined, and time bound. Design OD and TA should not have been separate processes, and should have been more closely linked to improved benefits in the health system; there should have been more defined health system outcomes from the start. Sustainability Alignment to government objectives is a good practice but to really ensure continuity and lasting benefits in the government health system it is important to try to bring government into the project as a core sub-partner from the start. USAID should identify how to achieve this within their regulations, possibly through their bi-lateral relationships with the GoB. CONCLUSION The overall conclusion is that Maatla has helped to strengthen the HIV/AIDS response at national and district level by contributing to the organizational strengthening of the CSOs responsible for implementing the civil society response. The main OD achievements for Result 1 were the strengthening of the governance of the participating organizations, and creating a group of governance trainers who can continue to strengthen this aspect of civil society in Botswana. A number of respondents indicate that challenges with governance have been the root cause of much of the funding problems of CSOs in Botswana. Three out of the four implementing organizations took full advantage of their participation in Maatla (even though their advancement was hamstrung by their small size), but a strategic opportunity was missed with BONASO to deliver on their mandate as per the strategic framework for Botswana HIV and AIDS service organizations. The project also helped to strengthen service delivery and strengthen health systems in the identified hard to reach districts, particularly though the referral system and the implementation of MoH guidelines and the development of models such as the PDHP, CATCH and the referral system. Many good practices can be carried forward into the next round of PEPFAR and since USAID will be working with FHI360 and some of the same sub-partners, we are sure what has worked well in Maatla will be transferred into PEPFAR 3. However, the sustainability of what has been achieved in the districts where USAID will not be working is questionable as the sub-partners do not have the required funds to continue with the services at the same level. Further, BOCONGO and BONASO are not likely to carry out the organisational capacity building and technical support continuity role that was envisaged in the initial project design under result 3. We didn’t have milestones established for what should have been done by the end of each year. The implicit assumption was that it would happen naturally, but we should have been more conscious of transitioning roles earlier in the project. We could have been more deliberate in the project about the milestones. (KII core sub-partner) viii RECOMMENDATIONS Evaluation question 2 • What modifications, if any, to the strategic approach and theory of change have potential to substantially increase program effectiveness? Result 1 A few key recommendations have been highlighted for the executive summary, more detail can be found in the main report. FHI360 1. When engaging in a capacity building program make sure that it follows a well-designed structured process, with milestones that are time bound, that progress on the OD components is incentivised, and where the various capacity building tools are better structured (the Governance training “Governance Matters) is a good example). 2. The tool for measuring the organisational capacity strengthening (in this case the OCA), needs to be more nuanced to capture whether the systems designed are actually being implemented and whether the organisations are able to learn and innovate. 3. It is recommended that in future FHI360 make the extension of sub-agreements performance based. Result 2 For FHI360 4. It is recommended that extensive consultation and facilitation take place to ensure local buy￾in into new programming direction. This will ensure that the needed working relations exist between the various sub-partners such as BONASO, BOCCIM, NCONGO. 5. Although sustainability was part of the concept and addressed in the beginning of the program, efforts were not maintained, particularly with the collapse of the BOCCIM cooperation. It is recommended that sustainability is built into the program and is constantly addressed throughout the program cycle. 6. Having insufficient personnel kept being a recurrent issue raised by respondents. It is recommended that a sufficient number of supervisors be recruited in the service delivery organizations in order to supervise the volunteers in hard-to-reach areas. 7. It is important to provide incentives for the volunteers both in monetary terms and through recognition. Particularly an organization like BONEPWA+, who provides services to people living with HIV by volunteers who are themselves infected or affected by HIV, should invest in income-generating activities for their volunteers. It is also recommended that volunteers help social workers identify OVC as they are in the community. 8. Due to the link between HIV and GBV it is recommended that continued awareness creation should take place in the communities regarding gender and GBV. This should be targeted specifically at men and involve men as mobilizers. It is recommended that community dialogues should be facilitated. 9. It is recommended to continue supporting refresher courses in the Maatla target districts to sustain the knowledge gained. Particularly more M&E capacity building is needed. For USAID 10. The program should be rolled out into more hard-to-reach areas to ensure extended coverage of the program. This is particularly the case with the mobile HIV testing and ix counseling services. Also, strategies for reaching men should be developed to ensure that they go for testing. 11. As the CSOs are assisting government in providing services in hard-to-reach areas, it is recommended that government fund the CSOs for this. CSOs should adopt a business-like costing approach, not just charge government for transport and food, but should cost personnel and overheads to work towards sustainability. Result 3 For sub-partners 12. BOCONGO must enter into an agreement with the NGO council regarding its role and functions relative to the council—in particular it’s capacity building function for civil society. 13. BONASO should fulfil the role provided for it in the civil society capacity building strategy, and enhance its monitoring and evaluation function significantly with the financial support received from NACA. 14. The graduation strategy for sub-partners should have been followed through, before they were exited from the project. Design General 15. Every project should have proper planning documents behind it, including a log frame or some other detailed results framework and an operational plan that covers the full length of the project lifespan. While annual planning is necessary, it needs to be done within the framework of an overall project plan covering the lifecycle of the project, with key milestones and critical pathways for a detailed theory of change. 16. More realistic targets should be set or a different way of calculating targets needs to be implemented, based on the delivery capacity of the organizations that are implementing the project and the funding provided by USAID – organisations are over-stretched by the targets required by PEPFAR which also pushes services up to levels that are not sustainable. Relevance & Sustainability for USAID 17. USAID must ensure that it considers the sustainability of interventions that have already been funded before closing-out or changing direction. The requirement to be able to re￾pivot all USAID programming on six months’ notice is not developmental, and results in inefficiencies, gaps in service delivery and collapsed organizations. The findings show that government tends to work in parallel with USAID and hence better mechanisms need to be found to ensure sustainability and continuity of the benefits of USAID programs,. The key assumption in the USAID Botswana results framework that Government will be ready to absorb systems and structures built up through USAID projects needs to be central to the design of the activities so that it can be influenced to occur, and needs to be monitored closely. Project Management, M&E General 18. Do not create projects with too many core sub-partners who are trying to work with sub￾partners. Perhaps one core sub-partner and the GoB is sufficient. Make sure that sub￾partners mandate is well aligned to PEPFAR. 19. Projects of this scale should not forego baseline and mid-term reviews as occurred for Maatla. x 20. A conceptual framework needed to have underpinned the design of the M&E system. There are some good ideas in the document produced by Measure in 2003 regarding the monitoring and evaluation of health systems projects. Designing the M&E system around health system outcomes would have made it easier to report on changes that had occurred and how the project contributed to these changes. 21. Because M&E is a fairly specialized skill, it is necessary to provide training, on-going mentoring and repetition to enhance quality compliance. Ideally, organizations should have specialized monitoring capacity or be provided with sufficient technical assistance to support these tasks. The use of data for project management and improvement must be emphasised and demonstrated. 22. It would be beneficial if government officials were more involved in piloting the monitoring tools, e.g. the referral system, so that they are able to help roll them out to areas that are not part of the Maatla project. xi Contents 1 INTRODUCTION AND BACKGROUND............................................................................................... 1 1.1 Context of the Program .................................................................................................................. 1 1.2 Rationale of Program........................................................................................................................ 2 1.3 Evaluation Purpose and Evaluation Questions............................................................................ 3 2 EVALUATION DESIGN AND DATA COLLECTION METHODOLOGY......................................... 5 2.1 Evaluation Methods and Process.................................................................................................... 5 3 FINDINGS – EFFECTIVENESS................................................................................................................. 7 3.1 Result 1: Strengthened technical competence and organizational capacity of NGOs and networks to support field offices, affiliates and member organizations with expanded and strengthened delivery of strategic and high quality HIV/AIDS services......................... 8 3.1.1 Capacity Assessments......................................................................................................... 8 3.1.2 Going Beyond Infrastructure ............................................................................................17 3.1.3 Managing the Organization as an Enterprise ....................................................................20 3.1.4 Technical Capacity Strengthening .....................................................................................21 3.1.5 Outcomes Result 1 ...........................................................................................................23 3.1.6 Lessons Learned................................................................................................................24 3.1.7 Conclusions.......................................................................................................................25 3.2 Result 2: A Model Developed and Implemented to Strengthen District Level Systems to Provide and Coordinate HIV/AIDS Services in Hard-to-Reach Areas................................ 26 3.2.1 Introduction ......................................................................................................................26 3.2.2 Capacity Assessment at District Level..............................................................................27 3.2.3 Strengthening the CSO Response at District Level to Deliver Services in Line with their Objectives and Targets.................................................................................................................34 3.2.4 Capacity Building of Local Government...........................................................................35 3.2.5 Strengthen CSO Organizational Leadership.....................................................................36 3.2.6 Defining/Redefining Roles to Make CS and Government More Effective Sub-partners through Relationship Building .......................................................................................................38 3.2.7 Coverage of Program Beneficiaries ..................................................................................39 3.2.8 Public-Private Partnerships to Strengthen Services and Sustainability .............................44 3.2.9 Enabling Environment for Policy Making and Service Delivery ........................................45 3.2.10 ICT Platform for Improved Service Delivery and Systems Functioning.......................46 3.2.11 Outcomes .....................................................................................................................46 3.2.12 Conclusions...................................................................................................................48 3.2.13 Lessons Learned............................................................................................................48 3.3 Result 3: Strengthened Institution(s) Able to Provide Sustainable Capacity Building to Civil Society ...................................................................................................................................... 48 xii 3.3.1 BONASO as Technical Lead for ASOS ............................................................................49 3.3.2 BOCONGO’s Broader Membership Value......................................................................50 3.3.3 Organizational Capacity Building ......................................................................................51 3.3.4 Negotiating Public-Private Partnerships............................................................................51 3.3.5 Advocacy and Communication for External Audiences...................................................51 3.3.6 Transition Planning from Phase 1 and Phase 2 ................................................................. 51 3.3.7 Conclusions.......................................................................................................................52 3.3.8 Lessons Learned................................................................................................................52 4 DESIGN ......................................................................................................................................................53 4.1 Findings .............................................................................................................................................. 53 4.2 Conclusions....................................................................................................................................... 55 4.3 Lessons Learned .............................................................................................................................. 55 5 RELEVANCE..............................................................................................................................................56 5.1 Findings .............................................................................................................................................. 56 5.2 Conclusions....................................................................................................................................... 57 5.3 Lessons Learned .............................................................................................................................. 57 6 SUSTAINABILITY ....................................................................................................................................58 6.1 Lessons Learned .............................................................................................................................. 60 7 PROJECT MANAGEMENT......................................................................................................................60 7.1 The Role of the Core Sub-Partners and USAID...................................................................... 60 7.1.1 Strengths...........................................................................................................................62 7.1.2 Challenges.........................................................................................................................63 7.2 Monitoring and Evaluation............................................................................................................. 64 7.2.1 Strengths...........................................................................................................................65 7.2.2 Challenges.........................................................................................................................65 7.3 Conclusions....................................................................................................................................... 66 7.4 Lessons Learned .............................................................................................................................. 67 8 RECOMMENDATIONS............................................................................................................................67 8.1 Result 1 .............................................................................................................................................. 67 8.2 Result 2 .............................................................................................................................................. 68 8.3 Result 3 .............................................................................................................................................. 68 8.4 Design................................................................................................................................................. 69 8.5 Relevance & Sustainability.............................................................................................................. 69 8.6 Project Management, M&E ............................................................................................................ 70 9 APPENDICES ............................................................................................................................................71 xiii Table of Figures Figure 1 Districts of Botswana with Maatla Presence Labeled with a Black Star................................... 2 Figure 3 Maatla Project Core sub-partners and sub-partners............................................................... 3 Figure 4 Overview of Capacity in the Health Sector, Measure Evaluation, 2003 ................................... 5 Figure 5 BOCONGO Internal and External Domain Ratings Compared ................................................10 Figure 6 TCA Overall Ratings – National Organizations (External Evaluation)......................................12 Figure 7 TCA Ratings by Domain—National—BONEPWA+ and BOCAIP-External ................................12 Figure 8 Internal vs External National TCA BONEPWA+ Ratings (2015)...............................................13 Figure 9 Result 2: Interlink between Activities (based on FHI360 2012)..............................................27 Figure 10 OCA-Lite Overall Ratings.......................................................................................................29 Figure 11 OCA-Lite Ratings by Domain ................................................................................................. 30 Figure 12 External Overall Scores TCA for District Offices of BOCAIP, BONEPWA+ and Sekgele.........31 Figure 13 External TCA Ratings for District Offices of BOCAIP, BONEPWA+ and Sekgele ....................31 Figure 14 Types and Locations of Services Provided by the Two Partners in the First Three Years of the Program ..........................................................................................................................................39 Figure 15 Draft Maatla Model for Family-centered CSO-government HIV and AIDS Service Integration in the Fourth and Fifth Year of the Program.........................................................................................40 List of tables Table 1 Summary Table of Final Sample ................................................................................................. 6 Table 2 Maatla core indicators as per the evaluation terms of reference ...........................................10 Table 3 Individual Domain Scores by Organization for TCA (External) – 2015 .....................................13 Table 4 Maatla core indicators as per the evaluation terms of reference ...........................................13 Table 5 Maatla core quantitative indicators........................................................................................32 Table 6 Key PEPFAR indicators..............................................................................................................42 Table 7 Key indicators for Maatla Result 3 ..........................................................................................49 xiv List of Acronyms AGM Annual General Meeting AIDS Acquired Immune Deficiency Syndrome ART Anti-Retroviral Therapy ARV Antiretroviral ASOs AIDS Service Organization(s) BAIS III Botswana AIDS Impact Survey III BOFWA Botswana Family Welfare Association BOCAIP Botswana Christian AIDS Intervention Program BOCCIM Botswana Confederation of Commerce, Industry and Manpower BOCODOL Botswana College of Open Distance Learning BOCONGO Botswana Council of Non-Governmental Organizations BONASO Botswana Network of AIDS Service Organizations BONEPWA+ Botswana Network of People Living with HIV and AIDS BHRIMS Botswana HIV and AIDS Response Information Management System CA Cooperative Agreement CATCH Combination Prevention Implementation Model CBOs Community Based Organization(s) CBTBC Community Based TB Care CS Civil Society CSOs Civil Society Organization(s) CSR Corporate Social Responsibility DAC District AIDS Coordinator DHIS District Health Information System DHMT District Health Management Team DMSACs District Multi-Sectoral AIDS Committee(s) DoH Department of Health DSCG District Support Group EBP Evidence Based Plan FBO Faith-Based Organization GBV Gender-Based Violence GoB Government of Botswana HCT HIV Counseling and Testing HR Human Resources ICIP Inveneo Certified ICT Partner ICT Information and Communication Technology xv IT Information Technology KII Key Informant Interview M&E Monitoring and Evaluation MER Monitoring Evaluation and Reporting MIS Management Information System MLG Ministry of Local Government MOH Ministry of Health NACA National AIDS Coordinating Agency NASTAD National Alliance of State and Territorial AIDS Directors NCONGO Ngamiland Council of Non-Governmental Organizations NGI Next Generation Indicators NGO Non-Governmental Organization NOP National Operational Plan NSF National Strategic Framework OD Organizational Development ODA Organizational Development Advisors OVC Orphans and Vulnerable Children PEPFAR President’s Emergency Plan for AIDS Relief PHDP Positive Health, Dignity and Prevention PMT Project Implementation Committee PLHIV People Living with HIV and AIDS PMP Performance Monitoring Plan PwP Prevention with PLHIV QA/QI Quality Assurance and Quality Improvement RFA Request for Applicants RM Resource Mobilization SAC Strategic Advisory Committee SADC Southern African Development Community SHOPS Supporting Health Outcomes through the Private Sector SOP Standard Operating Procedure SOW Statement of Work SSI Stepping Stones International SSI Semi Structured Interview SMT Senior Management Team S&CD Social and Community Development TA Technical Assistance TB Tuberculosis xvi TCA Technical Capacity Assessment TWG Technical Working Group USAID United States Agency for International Development WAR Women Against Rape WUSC World University Service of Canada 1 1 Introduction and Background The overall purpose of this study is to conduct a performance evaluation of the Maatla Civil Society Strengthening Activity, a five year project in Botswana, using a community based approach aimed at involving and mobilizing communities in their response to HIV/AIDS. USAID’s investment in the Maatla project, for the project lifespan of 2011 through to 2016, was at the Agreement Value Range of $15-$20 million. The project’s goal is to significantly and sustainably strengthen the capacity of the civil society sector in Botswana to support HIV and AIDS and related health service delivery. Maatla aims to achieve this goal by assisting national and district level local civil society organizations to acquire the skills and resources to implement high quality programs for the delivery of a wide range of HIV/AIDS treatment, prevention and care services. The evaluation is primarily intended for USAID to inform decisions and improve on the USAID project. The secondary audience is the core sub-partner and sub-partners, and the Government of Botswana (GoB).2 1.1 Context of the Program The HIV/AIDS epidemic in Botswana has been reported as the most critical development challenge in the country’s history and has been declared a national emergency since the first reported case in 1985 (GoB, 2010). Despite asserted nation-wide response efforts with certain successes over the decades, Botswana still grapples with the epidemic and currently has the third highest global adult prevalence rate (Central Intelligence Agency, 2015). In addition, the prevalence rate has shown an increase in recent national surveys. The 2013 Botswana HIV and AIDS Impact Survey (BAIS) IV (Statistics Botswana, 2013) estimated the HIV national prevalence rate at 19.03 percent3, which indicates an increase of 1.43 percent from the previous 2008 survey BAIS III. Prevalence is highest between the ages of 35 and 39 with a peak of 43.7 percent. The prevalence for women in this age group reaches nearly 50 percent. The Botswana HIV/AIDS response policies and strategies are guided by the HIV/AIDS policy framework, the second national HIV/AIDS strategic framework (NSF II), and include the tenth National Development Plan (NDP 10) and HIV/AIDS-related goals in the Nation’s development vision, Vision 2016. The second national strategic framework (NSF II) outlines plans for 2010 through to 2016, but was revised in 2014 and extended to 2017. The NSF II focuses on the following four priority areas: preventing new infections through encouraging behavioral change; strengthening systems through which the national response could be intensified; strategically managing information for monitoring, learning and re-programming; and scaling up of treatment, care and support (NACA, 2015). The GoB has committed itself to achieving zero new infections by 2016. Meeting this goal will rely on the coordinated and combined efforts of all relevant stakeholders, including civil society and civil society networking organizations (BONASO and NACA, 2010). Civil society organizations (CSO)s are a key partner for government in the national HIV/AIDS response since they successfully provide comprehensive services and enhance advocacy (GoB, 2010). The increasingly important role CSOs have played in the expansion of programs, through outreach and targeting marginalized populations, is acknowledged in the NSF II (NACA, 2010). CSOs in Botswana have demonstrated the ability to “mobilize communities and to act as intermediaries for a wide variety of target groups” (BONASO and NACA, 2010) through programs that are well-received, partly because of their civic nature. Despite the important role CSOs play in strengthening and complementing government programs, critical capacity gaps exist that limit their effectiveness. In response to this problem, BONASO and 2 The project description document refers to core sub-partners (FHI360, WUSC, BOCCIM, NASTAD and NCONGO) and sub-partners (BOCONGO, BOCAIP, BONEPWA+ and BONASO). However, due to conflicts with USAID terminology, the following terms have been agreed to: Core sub-partner replaces core partners and sub-partner refers to implementing partners as per the project description. 3 National statistic for the general population; aged between 18 months and 64 years. 2 NACA developed the CSO Capacity Building Strategic Framework 2010-2016 (supported by Project Concern Internatioanl (PCI) and USAID) proposes approaches for supporting comprehensive technical assistance and organizational development for CSOs responding to HIV/AIDS. Alongside the Strategic Plan, an accompanying operational plan was also developed and released in 2011. These are the two key documents that informed the design of Maatla. Figure 1 Districts of Botswana with Maatla Presence Labeled with a Black Star 1.2 Rationale of Program In order for CSOs to fulfil their important function of complementing national HIV/AIDS response efforts, there is a need to strengthen their institutional and technical capacity. The Maatla project aims to address this need. Maatla’s theory of change assumes that by building the capacity of CSOs and local government coordination structures, the structures will be more capable of supporting the HIV/AIDS response and broader health needs in Botswana (USAID, 2015). The critical assumptions linked to Maatla’s goal, include: 1. Viable government entities at the district level for adequate CSO coordination 2. Resources available from other sources to sustain the civil society sector 3. Willingness and interest to assist BONASO by all stakeholders Maatla result areas in Botswana are: Result 1: Strengthening technical competence and organizational capacity of national NGOs and networks to support field offices, affiliates and member organizations to expand and strengthen delivery of strategic and high quality HIV/AIDS services Result 2: Developing and implementing a model to strengthen district level systems (CSOs and local government) to provide and coordinate HIV/AIDS services in hard to reach areas Result 3: Strengthening an institution to provide sustainable capacity building to the civil society sector 3 Involved in Maatla activities are both core sub-partners and sub-partners. Core sub-partners include: FHI360 with the overall management role, accountability to USAID, technical capacity building lead and project reporting and monitoring and evaluation (M&E); World University Service of Canada (WUSC) whose role is to ensure sustained strategic organizational development support to national and district networks; National Alliance of State and Territorial AIDS Directors (NASTAD) for district level strengthening of government systems and structures; and Ngamiland Council of Non￾Governmental Organizations (NCONGO), who serves as a district-based model and learning site for civil society partnership with the main purpose of coordinating efforts for organizations. NCONGO also has non-HIV service providers whose main services include capacity building, resources mobilizations and advocacy for Orphans and Vulnerable Children, Community Based Resource Management, Youth Empowerment and Environmental Conservation. Sub-partners were Botswana Council of NGOs (BOCONGO) and Botswana Network for AIDS Service Organizations (BONASO), both membership organizations who were initially intended to implement Organizational Capacity Assessments (OCAs) and Technical Capacity Assessments (TCAs) respectively, in a cascade model of capacity development; and Botswana Network of People Living with HIV and AIDS (BONEPWA+) and Botswana Christian AIDS Intervention Program (BOCAIP), whose concentrate on HIV/AIDS service delivery. Figure 2 Maatla Project Core sub-partners and sub-partners To the best of the evaluators’ knowledge there were no critical competitors to the project that functioned at the same time and in the project’s environment. 1.3 Evaluation Purpose and Evaluation Questions EVALUATION PURPOSE The evaluation’s purpose is to assess: (1) how well the activities are being implemented so far, whether according to plan (2) to inform program design for the next round of USAID supported programs, and (3) to contribute to evidence of what works or doesn’t work in improving health outcomes among beneficiaries and communities. Core sub￾partners FHI360 (CA holder) - Overall management, Technical support WUSC - Organisational development support NASTAD - District level strengthening NCONGO - District baseed coaltion model Sub￾partners BOCONGO - Organisational capacity building BONASO - Technical capacity building BONEPWA - HIV/AIDS service delivergy BOCAIP - HIV/AIDS service delivery 4 EVALUATION QUESTIONS 1. How likely is each activity to achieve its desired end results, as defined in each activity’s results framework? 2. What modifications, if any, to the strategic approach and theory of change have potential to substantially increase program effectiveness? 3. To what extent has each activity been implemented according to activity design? Where there have been deviations or shortfalls, what are the causes? 4. To what extent has USAID fulfilled its role in the activities by providing adequate funding, responsive managerial oversight and technical support, and assistance in solving problems in implementation as they arise? 5. What actionable, prioritized lessons have been learned for the design of potential follow-on activities, i.e., what should continue and what should change from the current activities in a potential future activity? It was agreed at the planning meeting that since no impact evaluation of Maatla will be conducted as the capacity building project was closing out at the time of the evaluation, question 6 was no longer relevant 6. Has implementation so far been successful enough to warrant an impact evaluation of each activity before it closes? If not, why not? 5 2 Evaluation Design and Data Collection Methodology 2.1 Evaluation Methods and Process A mixed-method evaluation was conducted based on a document review, semi-structured qualitative interviews (KIIs) with key project stakeholders, focus groups, and organizational and technical capacity assessments. The data was triangulated through document review, interviews with core sub￾partners and sub-partners, as well as stakeholders from government and civil society at both national and district levels. A detailed description of the method and sample is contained in Appendix V including an evaluation matrix which maps each tool to the evaluation questions. Since the start of the project, Maatla has been conducting organizational capacity assessments using the Organizational Capacity Assessment Tool (OCA) and the OCA-lite (a simpler version that was deemed to be more appropriate for organizations in the districts. Maatla also conducted technical capacity assessments (TCA). The initial intention of the evaluation was to conduct OCA/OCA-lite and TCA assessments with a sample of organizations to validate the scores that had been given by the Maatla project team. However, due to time lags in the data collection periods (between the Maatla data and the timing of the evaluation), this no longer made sense. Hence, the assessments were conducted rather to compare the results and see whether the organizations had improved over time. The levels of enquiry in the evaluation were conceptually informed by the Measure Evaluation document “A Guide to Monitoring and Evaluation of Capacity Building Interventions in the Health Sector in Developing countries.” (Measure Evaluation, USAID, March 2003) Figure 3 Overview of Capacity in the Health Sector, Measure Evaluation, 2003 6 Process Consultations were held with USAID and FHI360 at the start of the evaluation through an evaluation planning meeting followed up by a telephonic engagement with the FHI360 team. This was followed by an inception report that included the proposed sample, list of instruments and work-plan. The instruments were designed based on a review of documents received from FHI360. The instruments were reviewed by experts in the HIV/AIDS sector from Botswana, and by USAID and FHI360 and then finalized. Ethics approval was obtained from the Ministry of Health in Botswana and all ethical requirements were followed. No children under the age of 18 were interviewed. Informed consent was obtained by the fieldworkers and the human rights of respondents have been protected. The fieldwork took place between 18 August and 4 September, and was preceded by an in-country planning session on 17 August. A field report was submitted with top line findings shortly thereafter, and telephonic feedback was received and discussion based on the report was held with the FHI360 and USAID teams. The draft report was also reviewed by an expert on HIV programs from Botswana hired by Southern Hemisphere. The data was then analyzed and coding reports were produced for thematic analysis. The report was presented in writing and through a verbal feedback session to USAID and FHI360 in Botswana. This final report incorporates this feedback. Geographical location and sample The Maatla fieldwork took place in Gaborone, and in the Okavango and Ngami districts. The districts were selected following discussion with USAID and FHI360. The Maatla project is also conducted in Chobe and Ghanzi districts. A total of 51 semi-structured interviews and five focus groups were conducted. The capacity assessments were completed with 15 organizations. Semi-structured interviews were conducted with relevant stakeholders from national and district government, core sub-partners and sub-partners, and CSOs who are members of the national networks. Focus groups were conducted with support group members who were also volunteers in the districts. Some respondents participated in both a semi-structured interview (KII) and an organizational capacity assessment (OCA) or a technical capacity assessment (TCA). Table 1 Summary Table of Final Sample Semi-structured interviews Focus groups OCA/TCA National 23 3 District 28 5 8 TOTAL 51 5 11 Data analysis and report writing The qualitative data was analyzed using NVIVO 10. Coding reports were drawn up based on the conceptual framework and the report structure. The report structure was shared with USAID and FHI360 for input, and the USAID guideline for report writing was followed. The OCA/OCA-Lite4 and TCA data were captured in excel and a separate report was produced on these results. This full report is attached as Appendix VIII.The OCA/OCA-lite and TCA scoring system, which uses a simple rating scale, tallying and averaging scoring system, was used. FHI360 provided a basic guide to the OCA approach. 4 The OCA-Lite was developed for the CSOs in the districts as not all the fields in the full OCA were applicable to them. 7 Limitations • Some semi-structured interviews were not conducted either because the respondents were no longer available, or because it was not going to add any value once the TCA or OCA-Lite had been conducted. This meant that some aspects of the evaluation relied entirely on document review and perspectives of those not directly involved. This was particularly true for NASTAD, for example, who has already closed their office in Botswana at the time of the evaluation. • We had intended to conduct the OCA/OCA-Lite and TCA assessments to validate the previous round of assessments, but since most assessments had been done at least six months previously, and conditions had changed since then with the close-out of the capacity development component, this was no longer feasible. Instead, we have used the results to compare the current status of the organizations with the status at the time when Maatla did the assessments. This gives us a sense of the current state of the organization, although the reasons for the differences are not always apparent because the tool did not capture this data. Differences in assessments could be because of the subjectivity of the assessor, or because of changes that occurred in the periods between assessments, and not because of any measurement bias. Since the OCA/OCA-Lite and TCA were only going to be validation exercises, we decided to do only a sample of these organizations. In retrospect, it would have been better to do OCA for all of the sub-partners as then we would have had an independent external verification of their stage of development. • Getting copies of documents to verify the OCA/TCA assertions has been difficult at the district and national level. Those documents that we were not able to take with us were perused on site where possible. Otherwise, only their existence and not their content can be verified. • The Organizational Capacity Assessment (OCA) was not completed with BONASO due to time constraints during the field visit to Botswana, and follow-up appointments were not kept. This meant we were not able to get a more up-to-date assessment of BONASO’s capacity. • We applied the technical capacity assessment to both direct recipients of Maatla technical assistance (e.g. BONEPWA) and to other CSOs in the district who should have benefited from Maatla through the cascade model or by participating in some of the Maatla capacity building workshops. The technical capacity assessment (TCA) was only applicable to those organizations that received direct support and are implementing major areas of focus related to Maatla, and did not apply well to the member organizations (such as Sekgele Training). 3 Findings – Effectiveness In this section the findings, conclusions and lessons learned related to each of the three result areas are presented. The evaluation questions guiding each section for effectiveness are: Evaluation questions: • To what extent has each activity been implemented according to activity design? Where there have been deviations or shortfalls, what are the causes? • How likely is each activity to achieve its desired end results, as defined in each activity’s results framework? • What lessons learned have been identified for the design of potential follow-on activities? 8 3.1 Result 1: Strengthened technical competence and organizational capacity of NGOs and networks to support field offices, affiliates and member organizations with expanded and strengthened delivery of strategic and high quality HIV/AIDS services Result 1 concentrates on building the organizational and technical capacity of the Maatla sub-partners at a national level. This was based on organizational and technical capacity assessments, which identified areas for organizational strengthening. Under the sub-objective “moving beyond infrastructure”, the focus was on helping the organizations to build their organizational development capacity through a range of capacity building methods such as training, mentoring, coaching, and the provision of policies and guidelines. The sub-objective “managing the organization as an enterprise” covered resource mobilization and strengthening the network aspects of the organizations. The final sub-objective, technical capacity strengthening, focused on helping the national organizations to improve the HIV and AIDS prevention, care and support services that their member organizations or branches offered in the districts. 3.1.1 Capacity Assessments This section describes the process that Maatla followed to develop and implement the capacity assessments, both organizational and technical. The information in the first section is from the annual reports from Y1 and Y2. OCA results from internal and external (conducted by the evaluators) assessments are then discussed. Organizational capacity assessment: A needs driven process for capacity assessment was developed, which built on previous capacity building programs for CSOs in Botswana, such as AED. The organizational needs were identified through the Organizational Capacity Assessment Tool (OCA and OCA-Lite)5 and the Technical Capacity Assessment Tool (TCA), which were developed collaboratively with the NGOs concerned and piloted for relevance. The OCA process was led by WUSC, and the TCA by FHI360. In year one, consultations were also held with key government departments such as NACA, Ministry of Health (MoH) and Ministry of Local Government and Rural Development (MLG) to agree on national NGO standards as part of developing the organizational capacity assessment tool. WUSC worked with FHI360 to integrate gender and mainstreaming standards into the organizational development capacity assessment tools. The first pilot assessments took place in 2012, and the first round of capacity building strategies was developed following a prioritization exercise. BOCONGO, BONEPWA+ and BOCAIP OCAs and prioritization exercises were successfully completed in Q3 and BONASO in Q4 (FY12). Three core sub-partners were involved at this stage—WUSC, FHI360 and BOCCIM. 5 The OCA-Lite was developed for branches of national organizations and smaller organizations in the districts that did not require the full spectrum of criteria. Result 1 Organizational capacity assessments Beyond infrastructure Managing the organization as an enterprise Technical capacity strengthening 9 Technical capacity assessment: The TCA was developed following a review of existing national guidelines for Maatla intervention areas, and the establishment of technical working groups (TWGs) for each of the five Maatla service delivery areas: HIV prevention, HIV Counseling and Testing (HCT), palliative care, OVC, and community TB care. Representatives from the TWG were drawn from MoH, MLG, the sub-partners, other CSOs, the UN, and other key stakeholders. Technical capacity assessments were conducted with BOCAIP and BONEPWA+ during Q4 of Y1. Technical capacity assessment tools were reviewed to incorporate gender issues in Y3, and were then used for a second round assessment of BOCAIP. Due to BONEPWA+’s collapse in FY13, second round assessments did not take place. TCAs were planned for the other sub-partners in Y3 but actually only took place in July 2015 (Y4). Service delivery standards were finalized in February 2013 in consultation with the sub-partners. The Annual Report does not mention the involvement of government stakeholders in developing these standards at this stage as was intended in the work￾plan. The process of transferring capacity to BOCONGO and BONASO to take over the organizational capacity assessments began in Y4 (FY 2015). From the Annual Report is seems that only BOCONGO was trained in OCA and OCA-Lite tools, capacity assessments and capacity building processes, and partner relations. BOCONGO was then mentored by WUSC and FHI360 to conduct the third round of organizational capacity facilitated self-assessments for the other three national NGOs (BONASO, BOCAIP and BONEPWA+), which were completed in Q2 of Y4. 3.1.1.1 OCA Results and Process The OCA was done through a self-assessment process, followed by participatory verification by WUSC and led to the development of an OD improvement plan. Progress against this was reviewed six monthly in a partnership meeting held between FHI360 and the relevant organization. The OCA assessment tool has been refined over time during the history of its implementation. From the vantage point of this evaluation it seems as though OCA has been revised at least annually, having received historical data collected with varyingly different instruments in 2012, 2014 and 2015. Overall results The evaluation also conducted an OCA with BOCONGO. This was the sole organization assessed by the evaluators (external assessment) using the OCA tool in August 2015. The results are compared to the most recent internal assessment, which was conducted in February 2015. The purpose of this comparison is to see whether BOCONGO has retained their score in the six months prior to the evaluation. Unfortunately the assessment with BONASO was not completed (see challenges under section 2). The overall scores between internal (88 percent) and external (82 percent) evaluations are similar. A score sitting between 80 percent and 89 percent qualifies BOCONGO as an “expanding” organization, which is “able to rapidly respond to change and sustain itself due to its credible systems, adequate resources and viable programs,” which is generally positive (Organizational Capacity Assessment, FHI360). However, notable differences exist between internal and external evaluation domain ratings for BOCONGO. These differences are illustrated in the radar graph that follows. Each point of the octagon in the radar graph represents one of OCA’s domains with scores at their lowest at the center of the graph (50 percent) and at their highest in the octagon’s corners. The three strongest domain discrepancies between internal and external OCA assessments, in order from largest, are in strategic leadership (81 percent internal vs 55 percent external), inter￾organizational linkages (95 percent internal vs 69 percent external) and process management (90 percent internal vs 69 percent external). Of all notable discrepancies the external evaluation’s rating was lower in all cases except in the case of human resources, where the external evaluation rated BOCONGO more highly on human resource competencies (73 percent internal vs 89 percent external). 10 These differences could be due to the time lag between the assesments, since the organisations were no longer receiving funds from Maatla at the time of the evaluation. This will be discussed in more detail in the subsections to follow. Figure 4 BOCONGO Internal and External Domain Ratings Compared The full report on each of these domains can be found in Appendix VIII In sum, BOCONGO appears to be on the whole a solidly performing organization. Differences in assessments are difficult to account for, especially with only two observations of a single organization. However, the main reason for the external assessment being lower for strategic leadership was resource mobilization. The evaluator felt that while they had secured some future funding, more needed to be done in this regard. Table 2 Maatla core indicators as per the evaluation terms of reference Indicator Latest Result as per Date of Data Source Data Source Date of Data Source % CSO meeting minimum standards for leadership, governance, management of finances, human resources, inter￾organizational linkages, infrastructure, etc. 60% (BOCAIP, BOCONGO and NCONGO scored above 80% in Round 3 OCAs) Annual Report FY15 Mean improvement in Organizational Capacity Assessment (OCA) scores per domain. Strategic Leadership Round 3: 68% Round 1: 52% Mean improvement: 16% Organisational Structure Maatla OCA Scores Year IV 11 Round 3: 87% Round 1: 71% Mean improvement: 16% Human Resource Management Round 3: 69% Round 1: 54% Mean improvement: 15% Financial Management Round 3: 92% Round 1: 75% Mean improvement: 36% Infrastructure Round 3: 79% Round 1: 56% Mean improvement: 23% Program and Service Management Round 3: 85% Round 1: 51% Mean improvement: 34% Process Management Round 3: 66% Round 1: 41% Mean improvement: 25% Inter-organisational Linkages Round 3: 80% Round 1: 59% Mean improvement: 21% 3.1.1.2 Technical Capacity Assessment Tool Results Discussion Compared to the OCA, the TCA tool is a shorter, simpler and more targeted instrument, focused more on output and delivery of organizations concentrating on HIV and AIDS prevention and testing, tuberculosis treatment, Positive Health, Dignity and Prevention (PHDP), Orphans and Vulnerable Children (OVC), and Gender and Gender-based Violence (GBV). TCA was used in this evaluation to evaluate BONEPWA+ and BOCAIP at regional level as well at their head offices. The most recent results from TCAs conducted with the head offices of BONEPWA+ and BOCAIP have been used to compare the results from the external assessment (conducted in August 2015). The most recent TCA for BOCAIP was conducted in September 2014, and for BONEPWA+ in July 2015. The overall TCA results by organization and office are compared in the graph that follows. Both of BONEPWA+ and BOCAIP’s national offices rate very highly at 97 percent. 12 Figure 5 TCA Overall Ratings – National Organizations (External Evaluation) Domains TCA’s four domain areas are technical planning and design (technical planning), which focuses on HIV and AIDS service provision, staff capacity, linkages and networks, and quality improvement and assurance (quality assurance). As there are only four domains, they will be discussed together in the sections that follow, comparing organization and the data that was collected as part of this evaluation with historical data. 2015 External Evaluation The radar graph that follows displays comparative scores for all TCA domains by organization from the data collected for this evaluation. The radar graph shows that all organizations have good scores of 80 percent and above and have a good balance between domains. Figure 6 TCA Ratings by Domain—National—BONEPWA+ and BOCAIP-External National organization branches had only a few partial criteria ratings as their TCA ratings were just short of perfect. 13 See individual domain scores by organization from this evaluation in the following table for more clarity on individual ratings. Table 3 Individual Domain Scores by Organization for TCA (External) – 2015 Organization Technical planning and design Staff capacity Linkages and networks Quality improvement and assurance BONEPWA+ National 98% 100% 100% 83% BOCAIP National 100% 97% 86% 94% TCA Validation Validation has been conducted on the most recent round of BONEPWA+s TCA assessment, but there was no data for BOCAIP for 2015. Having a recent internal measurement for comparison can be a useful validation exercise but in this case comparison raises some questions of validity. An internal review point of BONEPWA+’s head office occurred in July 2015, which was about a month prior to this evaluation. The internal and external ratings for each domain have been plotted on the radar graph that follows and a fairly different picture emerges. The two are only in alignment in terms of quality improvement and assurance but for other domains, there is a disparity where the external evaluator’s rating is higher by about 20 percent per domain. This comparison does not offer any corroboration to either assessment. Figure 7 Internal vs External National TCA BONEPWA+ Ratings (2015) Table 4 Maatla core indicators as per the evaluation terms of reference Indicator Latest Result as per Date of Data Source Date of 14 Data Source Data Source % CSOs implementing HIV/AIDS prevention, care, and treatment programs according to minimum standards. 100% (Both BOCAIP and Bonepwa+ scored higher than 70% for their technical assessment). Technical Assessment Tool for National NGOs and Networks, Bonepwa+ Technical Assessment Took for National NGOs and Networks, Bonpwa+ Technical Assessment Tool for National NGOs and Networks, BOCAIP Technical Assessment Tool for National NGOs and Networks, BOCAIP 27 September 2012 8 July 2015 26 September 2012 29 September 2014 Mean improvement in Technical Capacity Assessment (TCA) scores per domain. Bonepwa+ Technical Planning and Design 83.87% (FY14) 59.26% (FY12) Improvement: 24.61% Staff Capacity 75% (FY14) 60% (FY12) Improvement: 15% Linkages and networks 78.57% (FY14) 55% (FY12) Improvement: 23.57% Quality assurance quality improvement 83.33% (FY14) 5% (FY12) Improvement: 78.33% BOCAIP Technical Planning and Design 81.94% (FY14) 37.04% (FY12) Improvement: 44.9% Staff Capacity 84.09% (FY14) 50% (FY12) Improvement: 34.09% Linkages and networks Technical Assessment Tool for National NGOs and Networks, Bonepwa+ 27 Technical Assessment Took for National NGOs and Networks, Bonepwa+ Technical Assessment Tool for National NGOs and Networks, BOCAIP Technical Assessment Tool for National NGOs and Networks, BOCAIP September 2012 8 July 2015 26 September 2012 29 September 2014 15 85.71% (FY14) 75% (FY12) Improvement: 10.71% Quality assurance quality improvement 88.89% (FY14) 30% (FY12) Improvement: 58.89% 3.1.1.3 Strengths/Enablers for Capacity Assessment What was strengthened in key domains? In the qualitative interviews conducted for this evaluation, respondents were asked to comment on the most significant contribution of Maatla to the organization in each of the OCA domains. A few examples from these responses are presented below: Strategic leadership: Review of strategic plan, proposal writing Governance: Improved understanding by board and management of their roles, increased visibility Human Resources: Mainstreaming gender into HR policies, improved HR record keeping, supervision and management of people, performance appraisal tools, salaries for key staff Finance management: Financial policies and procedures, financial reporting to donors, salaries for finance personnel, understanding accountability and limitations of authority, conducted audits Program and service management: Developing organization-wide integrated work-plans, consulting the districts in development of project plans, volunteer management Process management: Improvement of data collection tools, training of staff in M&E, data quality, understanding qualitative data such as lessons learned Inter-organizational linkages: Improved relationships with district, collaboration with government, improved relationship between BOCAIP and BONEPWA+, ability to negotiate with partners Infrastructure and IT: Computers and printers, renovated properties for income generating potential, website, internet access, IT interns. All this has allowed the organizations to become more efficient, and communicate better with their members or branches. The following quote illustrates the benefits of OCA for one of the implementing organizations: It [the OCA] helped to identify BOCAIP’s weaknesses and making us aware of areas where we did not pay much attention—these are the areas to work on and pay attention to. It helped to improve systems— our organizational systems were not coordinated in the past. With the OCAT—it helped to align all our work and developed tools to build the org into one that delivers services efficiently and effectively. (Semi￾structured interview (KII) Implementing organization). Process issues The process of developing the OCA and the TCA tools was a strength as they were developed to be context specific by looking at the national guidelines. They were built on previous OD processes in Botswana, which ensured continuity, and a consultative process was followed with a high degree of government involvement. The OCA tool for Maatla was also revised to include assessment of how national NGO networks engage with or support their members and affiliates. Organizations found the assessment process empowering. This was enabling as it encouraged their participation and ownership of the process. Of the four national organizations, three said they would continue to use them in the future for themselves and their members/branches. 16 The coupling of the assessments with the development and six monthly review of implementation plans was a strength of the process. As one implementing organization respondent commented: “We have greatly improved our systems and processes because of the OCAs that were conducted. We developed our OD plans and followed through with the implementation of the plans and we are stronger because of this.” The OCA tool has a fair mix of questions to assess whether the systems in place are being implemented. For example, it checks whether there is a resource mobilization strategy, and whether the organization has funding from more than one donor. It is up to the assessor to validate much of the responses by checking into the actual documents to see whether they are up to date (for example, the finances, asset register or personnel records). The TCA was a focused yet comprehensive tool that covered the key domains for technical capacity. WUSC described the approach they used in their OD as inclusive and involving, and suggested that this should have demonstrated a leadership style that was inclusive to the NGOs, which they could translate in their own work as capacity builders. 3.1.1.4 Challenges/Barriers for Organizational Capacity Assessment Gaps in key domains (OCA) Strategic leadership: Leadership processes, such as delegation and internal communication, media engagement and communication with members Governance: Organizations received very high scores for governance Resource mobilisation: Unrestricted funds, domestic sources, diversified sources, capacity statements, readiness Human Resources: Performance appraisals at all levels, salary structure, assigning roles into job descriptions, implementation Process management: Data analysis and data usage Program and service management: Member engagement; strategic and implementation planning Financial management: Debt management, implementation of key policies and procedures (FHI360 continued to manage the funds of BONASO and BONEPWA+ throughout the life of the project so the organizations did not develop internal capacity in this regard) Process issues The technical capacity assessments were not conducted as planned in year 2. Hence the development of technical capacity building plans and the resultant intended technical capacity building did not take place that year either. There were a number of reasons for this. Firstly, BONASO and BONEPWA+ had both experienced governance and financial difficulties and had basically collapsed in the early years of the project. As a result much of year 1 and 2 was spent helping them to rebuild themselves, hire staff and put basic systems in place. Secondly, BONASO was also supposed to be the lead on the TCA process, but they were resistant to taking on this role, as they wanted to concentrate on strengthening their ability to coordinate civil society AIDS service organizations (ASOs), and to fulfil the capacity building role that was being fulfilled by BOCONGO in Maatla. Despite numerous bi-lateral and all-partner discussions and working groups facilitated by FHI360, no compromise on the role of BONASO was reached. Two respondents from different organizations mentioned that it is possible to score high on the OCA but remain weak in implementation, because the bulk of the scoring goes to having policies, procedures or documents in place, but not the extent to which they are utilized. This was confirmed by interviews with core sub-partners: “A weakness is that some parts are just a checklist but you do not capture the implementation part, e.g. ‘do you have an HR policy’? And not ‘do you use it’?” 17 Another related challenge was that the organization could score high on a domain but low in one area that was a critical success factor, e.g. they may have all their finance policies in place but no finance manager. Although the OCA was adapted throughout the project, which may have made it a better tool, it created problems for comparative or longitudinal data analysis. Further, it was suggested by one respondent from a sub-partner that the tool required a level of OD sophistication unnecessary even for the national organizations. There was not always a good fit between the Canadian volunteers and the CSOs. Some sub-partners felt that some Organizational Development Advisors (ODAs) were too junior and did not have much to share. “They should bring in the right level of people to build our capacity” (KII, sub￾partner). 3.1.2 Going Beyond Infrastructure A range of different capacity building strategies were used to support the organizations, namely mentoring, short- and long-term training, organizational development advisors, consultants, provision of policies and procedures, and placing of interns. The WUSC strategy for organizational development assistance was initially to have embedded ODAs in each of the implementing organizations, but this changed in FY14 WUSC to having shared ODAs who specialized in certain skills, e.g. HR. Each of these strategies had benefits and challenges, which are highlighted below. A cascade model was used for training, whereby the sub-partners were the main recipients of capacity building interventions, and were expected to cascade these skills and resources down to their members. The annual reports show a clear progression of the capacity building program from building relationships and identifying needs to developing organizational systems and processes. In FY11, relationships were being established between the core sub-partners and sub-partners, and needs established. In FY12, the focus was on recruiting OD advisors through WUSC, mentors through BOCCIM and WUSC, and identifying training providers and consultants in Botswana. While some short-term training and mentoring activities did take place in FY12, the mentoring program of BOCCIM did not succeed in FY 2013 and BOCCIM exited the program in FY14. A number of mentoring groups were also established, such as for resource mobilization and gender mainstreaming. The OD support provided through WUSC and TA by FHI360 continued through FY13 and 14. In FY14 the focus was helping organizations to develop policies and procedures in line with their needs identified in the OCA processes and by the ODAs. The governance training, which was the most structured and beneficial training program, was only identified in 2014. The most successful story has been the governance training. This will leave a legacy. They were not just trained, they did proper accredited training and it was very practical. They have reviewed their strategies, trained their boards, and they have trained their officers who will continue to train and guide their own organizations as they move forward with issues of governance. Governance has been such a big problem in this country. (KII, sub-partner) The domains where the organizations have advanced the most have been in Governance and Financial Management (particularly since Maatla was prepared to pay for their financial audits to take place) and Human Resources. All these areas put the organizations in good standing for their resource mobilization efforts. However, the caveat is that FHI360 continued to directly control the funds of both BONEPWA+ and BONASO, and so their financial systems were not fully implemented during the course of the project. The OD component also resulted in the production of a number of toolkits and guidelines for use in the CSO sector in Botswana, such as a financial management toolkit for small CBOs (see Appendix X for a list of these). See Appendix IX for a list of training conducted. 18 3.1.2.1 Strengths/Enablers The OD training and mentoring program was responsive to emerging needs through the emerging needs tool. The skills of the trainers and the provision of guidelines and tools were identified as strengths by a number of respondents. The following quotes highlight this: The strengths were that the people who trained us were open-minded and were skilled and they gave us tools to use, and we used these when they were gone. (KII sub-partner) The provision of policies and guidelines was very sustainable because they provided us with a good basis to continue the work of the organization if new staff come in. (KII sub-partner) A relationship with the National Internship Program was an enabler as it provided the implementing organizations and their members who participated with additional human resources. The embedded OD advisors were able to get to know the organizations they worked with very well and were able to identify intervention points for OD capacity building, and also learn the personalities of the key individuals, which affects their mentoring relationships. The structure of the governance training was a major enabler. It was long-term, highly structured, practical and accredited. The trainers become licensed governance trainers, and as part of this each trainee is required to cascade governance support to four other organizations. This meant that governance training was cascaded down to 16 organizations. This is the main example where training was cascaded down. Another key strength of the governance aspect was that the WUSC also worked directly with board members to help them understand their roles better. One of the key enablers for high scores in the organizational capacity assessments was the financial support that Maatla provided to these organizations to carry out their business, such as hiring staff, holding AGMs and doing financial audits. The capacity building ensured that they did these things correctly. We couldn’t even run an AGM or hold board meetings because we didn’t have funding, but Maatla gave us funding for this—we also couldn’t do an audit before. (KII sub-partner) We now know there is a difference between management and the board, between operational and strategic leadership. (KII sub-partner) The mentoring groups were identified as a good way for the partners to learn from one another, but they added another time-consuming dimension to an already over-stretched staff compliment. This challenge is raised below. 3.1.2.2 Challenges/Barriers A key contextual challenge was the funding crisis for NGOs in Botswana that took hold at the start of the project. The sub-partners were operating with a scaled down staff component and there were too few people to receive capacity strengthening activities. The following quote from the FY14 Annual Report captures this: “This has been a challenge for capacity building efforts as our partners tend to operate with one program officer – individuals who are frequently out of the office, and have limited time to participate in capacity strengthening activities.” Often it was the same person, such as a program manager, who had to attend all the training that was being offered by the various partners (often close together), and then still deliver on the organization’s programs. This resulted in an over￾supply of capacity building for a relatively small NGO sector. Sub-partners argued that they should have been allowed to recruit more people, and this would have solved many problems. The following quote illustrates this point: “We should have been allowed to recruit more people to be capacitated from our side—there were five people from their side to capacitate only one or two people on our side. There was no balance.” (KII sub-partner). This view was corroborated in SSIQs with core sub-partners. 19 A key assumption behind the Maatla project design was that the organizations would have other funding to hire staff and implement programs, but this was not the reality for many. BONASO had collapsed at the start of the project and BONEPWA+ collapsed in FY12. Helping to resuscitate BONASO became the focus of their scope of work in the first few years and this delayed their OD progress. While Maatla did provide financial assistance for hiring staff, for some organizations this was adequate (such as BOCAIP head office), but for others, such as BONASO, it was not. The implementation of what has been learned on short courses is a challenge, as the most courses were not accompanied by practical projects and mentoring. Participants thus become busy with their other deliverables and meeting their targets, and the OD components, which did not form part of their performance agreements, did not get the required attention. WUSC tried to tie the participation in training to key milestones, but were not able to follow up on whether this took place. An example provided by participants was that of the M&E training to help organizations develop M&E plans. The training itself was beneficial and rough M&E plans were developed in the session, but there was no follow-on support to help the organizations finalize and operationalize them. The result is that the organization concerned became very good at M&E for Maatla specific reporting, but did not extend this in any systematic way to their other projects. Embedded ODAs tended to be used as another pair of hands in over-stretched organizations, and hence had limited impact on the organizations’ longer-term capacity strengthening. Their performance agreements were also not linked to key milestones. The cascade model of training was a barrier, particularly for the OD component. There were a number of reasons for this. Implementing organizations said that they were expected to cascade capacity down to their members but they did not have any funding to do so, as they were expected to do it at no or low cost. One of the primary challenges in this regard was that the OD component was wound down in FY15, just at the point when the organizations could have been in a position to start cascading in a more structured way. This view was confirmed by an KII with a sub-partner Now people feel that they have much capacity and knowledge to share, but we did not build an enabling environment for them to do this sharing … We did help them to raise funds, but if we had the remaining year we could have helped to generate some of the opportunities for them to do this. (KII sub-partner) The loss of BOCCIM as a mentoring and private sector partner was a blow to the program, and significantly reduced the mentoring capacity in key OD areas, particularly resource mobilization (RM). It was also one of the reasons that that RM efforts only began relatively late in FY14. Further, the RM TA capacity was weakened because it was USAID’s view that no additional staff should be recruited by FHI360 to replace the two BOCCIM resource mobilization staff members (Maatla Annual Report 2014 and KII sub-partner). The mentoring groups were a good idea but did not work well due to poor attendance. Even though participants supported the idea of the group, they placed additional demands on already over￾stretched individuals, again linked to the over-supply issue, as it tended to be the same individuals who participated in these groups. The development of annual work-plans and sub-agreements was an annual challenge that caused delays in program implementation. It is not clear exactly what caused these lengthy processes, but the number of sub-partners (four) and core sub-partners (four) was a major contributing factor. Ensuring alignment between partners and with the COP and avoiding duplication through a consultative process was a time consuming challenge. A gap that has been identified is insufficient correlation between time spent by ODAs on TA, and the OCA scores and priority OD capacity building needs. To address this gap, FHI360 worked with WUSC to better target and intensify the level of TA for partners scoring poorly in certain domains, and included new performance benchmarks for ODAs at district level to ensure sufficient time is spent in site-level mentoring of district-based CSOs. Another challenge was that, beside the partner meetings to review the OD plans, there was no incentive for the organizations to adhere to their plans. The activities and milestones were not 20 included in annual work-plans. A suggestion from one of the ODs was that they should have had to commit to a certain number of hours in mentorship or to conducting a certain amount of OD activities. The main strength of the governance program was also its biggest challenge: it was very time consuming and demanded a real commitment from the participants over a long period of time. “I had to go through 20 weeks of training—I had to sit in the office every Sunday for 8-10 hours.” (Governance trainee, KII). It is not likely that people would be able to participate in more than one of these courses at a time. The governance program was also expensive as the course and the trainer came from Canada, but now there is a team of trained governance trainers in Botswana, and BOCONGO is an accredited training organization, so it was a worthwhile investment. The key people involved in the project did not have the correct organizational development experience to know what is best when accompanying an organization in a long-term capacity building process. This included both WUSC and FHI360. 3.1.3 Managing the Organization as an Enterprise The first focus of this objective was to review existing strategic and operational plans from a business perspective, and to help the organizations adapt accordingly. The three main strategies in this objective were to encourage resource mobilization, income generation and enhanced membership value. Due to delays in signing sub-agreements in FY12, the strategic plans were reviewed by BOCCIM in FY13. The strategic plan for BONASO was never completed, even though they had a mentor from BOCCIM. By FY2014 this was still limping along, and the organizational strategies for BOCONGO and BOCAIP were not yet costed. A key intention of this objective was also to build an enabling environment for resource mobilization by bringing donors, business and civil society closer together. The enabling environment activities never took place in FY12 (tapping into regional and international resources to identify opportunities for twinning; convening donor forum), and there are none reported for the consecutive years, probably due to BOCCIM’s absence from the project. In FY15 FHI convened a meeting with the Botswana Stock Exchange to discuss fundraising for CSOs; discussions were still on-going at the time of the evaluation. The emphasis in FY15 was also on helping organizations to do two main things. On the one hand, to think about how they could operate and remain functional on a very low core budget and, on the other, to plan for income from their assets and services. For example, BOCONGO set up a private business arm to sell training services and rented out some office space, and BONEPWA+ generated income from a tuck-shop (Maatla Annual Report, FY12-FY14; semi￾Annual Report FY15). FHI360 started helping organizations to write proposals in FY14, and at the time of the evaluation all of the organizations had managed to receive some funding (either from donors or from the GoB). However, the amount of funding received was nothing close to what the organizations were receiving through Maatla, which suggested that they would have to scale back their services. At the time of the evaluation, BOCONGO had already lost four people over the last year because of slashed funding. Further, funding received is not necessarily for the HIV/AIDS area (e.g. BOCONGO secured funding from the Global Environment Fund (GEF)), or in the Maatla districts. This threatened the sustainability of these interventions. The second focus area was building the NGO network service delivery component, and district visits took place in FY12 and FY13. (This seems to have been used mainly for capacity assessment purposes.) Nothing was reported in this line for FY14 or FY15. Despite FHI360’s efforts to rescue this objective of the project, the exit of BOCCIM left a huge gap at crucial years of the project, which were critical for the organizations to establish themselves with an enterprise mind-set and be ready with sufficient additional funding when Maatla closed-out. 21 3.1.3.1 Strengths/Enablers Proposals for additional funding were sent out in FY2014 and FY2015 and all four implementing organizations had received funding at the time of the evaluation. The organizations had begun to think about how to generate income from properties and BOCONGO had begun to realize the vision of having an income generating training program. 3.1.3.2 Challenges/Barriers The primary barrier to the success of this objective was the exit of BOCCIM in FY14, and while FHI360 stepped in to fulfil this role thereafter, it means that little was done in terms of resource mobilization in the early years of the project. This challenged the critical assumption of the project design that the organizations would have other sources of funding besides Maatla. Some did, but this funding could not necessarily be used for the same purposes as the Maatla project. There were a number of reasons offered for the failure of BOCCIM to play a meaningful role, including that their leadership never bought into the project, that they were not equipped for the role as they had no experience in mentoring, that they took too long to develop the mentoring manual, and that organizations did not have sufficient time for mentoring due to other program commitments by the time they were ready to engage. Another barrier is the lack of legislation in Botswana to encourage private sector giving, such as tax incentives. This is something that should be explored by the GoB in the future, and BOCONGO and BONASO should advocate for this in their interactions with the state and their members. 3.1.4 Technical Capacity Strengthening The purpose of this objective was to ensure that organizations received technical capacity support to help them achieve their particular mandate. It was supposed to be evidence based in that the support interventions were to be informed by the results of the TCA. An integrated approach was adopted that encompassed prevention, HIV Counseling and testing, Community TB Care, Adult and paediatric palliative care and OVC. Cooperation with and alignment to the GoB According to the Annual Report for FY2012, a number of key activities took place that concentrated on the dissemination of national MoH guidelines to the districts, and alignment of sub-partners’ services to an integrated approach to HIV prevention and care. The activities reported in FY2012 suggest good cooperation with FHI360 and the Ministry of Health at national and district level. For example, FHI360 participated in the MoH Technical Working Group (TWG) to review the HCT strategy, and together they facilitated training for BOCAIP and BONEPWA+ for their service providers in the districts to ensure diffusion of training curricular on HIV prevention, care and treatment. This good cooperation in the first two years was confirmed by the interviews, but respondents indicated that communication and involvement of the MoH at a national level faded out after that. The program, however, continued to be aligned to MoH guidelines. For example, in FY14 FHI360 supported CSOs to understand and ensure adherence to the MoH algorithm for HIV testing and counseling (HCT), and training in MoH CBTBC tools for district level CBOs (FY15 Semi-Annual Report). The relationship with NACA continued to be engaging throughout, NACA’s participation in the Strategic Advisory Committee was consistent, and Maatla supported NACA with the National Prevention Pitso in FY13 and FY14 in developing and disseminating a combination prevention implementation model (CATCH) in FY14. Maatla helped with the development and dissemination of national guidelines and strategies, such as the dissemination of national documents on care for OVCs, the SADC minimum package for care of OVCs and youth, a National Strategy on Palliative Care, HCT guidelines and protocols, and Community based TB Care (CBTBC) guidelines. It also helped to develop the combination prevention implementation model (CATCH) (annual reports FY12-FY14). Civil society 22 FHI360 worked closely with the national NGOs and network organizations to build their technical capacity so that they could cascade this down to their members at a district level. The first round of technical assessments which would inform capacity building efforts in FY14 took place for BOCAIP and BONEPWA+ in FY13. The technical tool for BONASO and BOCONGO was only to be developed in 2014. The progress on this was not reported in the FY15 Semi-Annual Report. BONASO was expected to take the lead on this technical capacity strengthening component and FHI360 was going to help them develop an approach and framework for the purpose. According to a number of respondents, FHI360 and BONASO were not able to come to an agreement on their role in the project, despite many discussions and small group sessions in all-partner meetings. BONASO believed its mandate was capacity building of network members as well as technical capacity strengthening, but Maatla had assigned the lead OD capacity building role to BOCONGO. Thus FHI360 did not really have a national civil society counterpart to transfer capacity to, as WUSC had in BOCONGO for the OD component. FHI360 also worked closely with BOCAIP and BONEPWA+ at national level and in the districts to develop and disseminate guidelines, manuals and training for service provision. They also cooperated with government in the districts to expand services in hard-to-reach areas. How the program supported cooperation with government and CSOs is discussed in more detail under result 2. BOCAIP focuses on HCT, HIV prevention and OVC, while BONEPWA+ concentrates on HIV prevention among PLHIV, treatment and care for PLHIV. The two referred to one another at district level. Implementation of the technical program in the districts is evaluated under Result 2. The gender dimension really gained momentum in FY15. In 2014 BOCONGO was assisted to develop a gender mainstreaming toolkit, which was to be implemented in FY2015. Two new partners were brought in to assist with the gender component in the districts, namely, Women Against Rape (WAR) and Stepping Stones International (SSI). They were supported to strengthen coordination and provision of services in their respective districts (Semi-Annual Report FY15). Methodologies A number of new methods were tested or implemented under this objective, namely: - Community based TB Care (the government had been promoting clinic based TB DOT services) - Integrated family centered services through enhanced referrals and linkages among CSOs and government departments - Positive Health Dignity and Prevention (PHDP) minimum package offered by BONEPWA+ - Community volunteers working with MoH to strengthen adherence to treatment for people on ARVs (learning from community based TB adherence models) - Gender and HIV mainstreaming (assessment and training) - Combination Prevention Implementation Model (CATCH) The effectiveness of the technical intervention is discussed under result 2, as it was implemented in the districts. 3.1.4.1 Strengths/Enablers A strength was the flexibility in the cooperative agreement that allowed FHI360 to change its work￾plan to respond to emerging needs in Botswana, for example, to support NACA to develop the combination prevention strategy. The development of guidelines, manuals, materials and directories covering the technical implementation areas is a strength. SIMS was an important catalyst for this because much of the work had been undocumented before the SIMS assessment took place. FHI360 responded very well to the challenges identified by SIMS, and in collaboration with partners, and brought in FHI360 23 expertise from other parts of the world. and developed what appears to be an impressive set of documents6 (see Appendix X for a list of technical guidelines and toolkits developed through the project). The documents were appreciated in the districts, as one respondent commented: “We were able to develop directories at the sites, which are the key critical tools for effective running of the referral system” (KII government respondent). A member of the technical advisory committee mentioned that feedback on the manuals and guidelines had been positive. The cascade model worked well for the most part but follow up support was needed from FHI360 and WUSC to support implementation in the district. The integration of HIV and TB has also been identified as project implementers, USAID and core sub-partners as a strength, as the following quote illustrates: “We have moved to more comprehensive integrated tools. For example HCT tools have incorporated TB screening questions, and staff can talk about all the program areas—HIV, TB, GBV” (TCA assessment, sub-partner). 3.1.4.2 Challenges/Barriers Allocation of staff was a challenge. Firstly, FHI360 has staffed its team to deliver a capacity building project and did thus not have the right team to implement a service delivery project. Secondly, there was tension about where to place people—at FHI as TAs or at the member organization as staff members. “They had more technical staff than us and we were implementing the program so we should have had this technical staff” (KII sub-partner). BONASO did not have any technical staff paid for by Maatla to implement the technical capacity aspects, even though this was their key role in the project. The main challenge identified in the interviews was with implementation at a district level, because the documents are mainly prepared in English and the volunteers cannot use them. The training, however, was done in English and Setswana. The cascading of the capacity building also did not extend beyond Maatla sites in most cases, due to a lack of resources on the part of the sub-partners. BONASO, in particular, reported that many of its members were facing closure or had to scale back operations. This meant that in order to implement the services and reach the targets envisaged by Maatla, the program also had to pay for people to do the program delivery. 3.1.5 Outcomes Result 1 In this section, we report on the general organization change and system-wide outcomes that have been achieved as a result of all four objectives in this result area. The specific changes for each objective are discussed in the relevant sections above. OD Success competencies for National NGOs The national NGOs can be divided into two groups: BOCONGO and BOCAIP, which were relatively strong and functional organizations, and BONASO and BONEPWA+, which had both collapsed by FY12 (year 2 of the project cycle). Organizational change The outcomes for the first group reflect that they have been able to progress well along the organizational capacity spectrum and have reached over 80 percent overall as organizations (Maatla Semi-Annual Report, March FY15). Maatla was able to keep BONASO and BONEPWA+ alive and resuscitate these organizations. They achieved 77 percent and 60 percent respectively. The domain specific achievements are highlighted in the section on capacity assessments above. Organizations report being more confident about their ability to govern and manage themselves. 6 An in-depth assessment of these guidelines has not been conducted by the evaluators 24 BOCAIP reports having improved organizational systems, enhanced coordination “… they made us organize our work” (KII BOCAIP). They also report having become a more strategic organization. BOCONGO reports internalization of the process of continuous improvement in OD using the OCA periodic review, and the reflect and plan model. They also speak of increased ability to help their members with their OD challenges. “The most important thing is that it becomes an institutionalized process” (KII BOCONGO). BONEPWA+ reports being in a stronger position, particularly because they have had their audits done, which means that they are able to apply for donor funding. However, they believe that their strategic approach could be further enhanced, as they felt that they were still too focused on donor demands and requirements for programming. BONASO has managed to secure funding for one year from NACA, which is its main outcome. BONASO failed to achieve noteworthy outcomes from the Maatla project mainly because they and FHI360 could not agree on their role in the project. Both are at fault. BONASO failed to strategically take advantage of Maatla to implement the Operational Plan for Civil Society Strengthening, and FHI360 should have changed strategic direction with BONASO earlier in the project when they realized they were not going to get their cooperation regarding their TA role on the technical side. Health system change The health systems outcome has been the strengthening of partnerships between government and the national sub-partners, and between the sub-partners themselves. Respondents from all the organizations indicated improved relations in this regard. Respondents indicated that they now understand which coordination structures to belong to and how they could share responsibilities between them. “We were able to develop a memorandum of understanding of how we all work together on other aspects; we now have a cooperation model” (KII sub-partner). The second outcome is strengthened CSO reporting to government on their response to HIV and AIDS in Botswana. This has also contributed to the improved relationships between civil society and government. A key component of this has been improved tools and an appreciation of data quality. Related to this is the strengthening of the multi-sectoral approach at district level—especially in terms of the DMSAC—but this will be elaborated under Result 2. The third is that both civil society and government have an enhanced appreciation of the importance of strong systems for organizational development and management, including the importance of maintaining strong regulatory systems to encourage NGO adherence to these good governance requirements (e.g. keeping the NGO registration system updated). The fourth is an enhanced understanding of how gender discrimination and gender-based violence affect the HIV pandemic and how it can be mainstreamed. A key outcome has also been the dissemination and implementation of government guidelines that are in line with government policy. 3.1.6 Lessons Learned • What actionable, prioritized lessons have been learned for the design of potential follow-on activities, i.e. what should continue and what should change from the current activities in a potential future activity? OD The factors that made the governance training successful were that it was a long-term accredited course with practical assignments and it was accredited. Hence, it also provided income generating potential for individuals and organizations to become governance trainers. 25 The strategy of first using embedded and then shared ODAs raised some interesting lessons. Rather than making a wholescale switch, more flexibility should have been introduced depending on the needs of the organization, the domain and the personality of the individual receiving the support. For example, some domains work better with an embedded model, e.g. HR and finance, due to the confidential and day-to-day nature of the task. But for IT, a mentor could be available on demand. Regarding personality and strategic leadership, one director may not need or want a mentor, but another may. We changed from an embedded to a shared model, but we should have had a more customized approach. There were so many projects with multiple partners, and this did not really allow for time to get to know each organization’s style and culture and their specific needs.” (KII sub-partner) Short-term training should always be coupled with mentoring, and tied into a practical deliverable that the participant (or organization) should deliver following the training. For instance, M&E training should be followed up by the development of an M&E plan, with on-going deliverables related to the plan, with a mentor assigned to each participant to help them see this through. The project had too many core sub-partners, which made project management and coordination complicated, and also resulted in an over-supply of capacity building. An adequate feasibility study needs to be carried out to ensure that the objectives of the project could be achieved, given the capacity of the organizations who are expected to implement it. Since this project was building on existing capacity building projects, this information should have been available. OD activities need to be incentivized otherwise organizations tend to focus on meeting their service delivery targets and the OD component becomes neglected. For example, OD could be included in the recipient organization’s work-plans in terms of time spent or completion of activities related to OD plans. The demands placed on the organization without having these in the work-plan was described as follows: “It was like trying to do a PHD while having a full-time job” (KII sub-partner). The capacity building component needed a process model, and a conceptual framework. For example, the domains could have capacity pauses after training to focus on implementation, before the next module is introduced. “We should have taken more consideration of not trying to do everything at once Technical capacity strengthening The cascade model of training worked relatively well, but the national organizations still required TA from FHI360 to help cascade training, and to ensure that the district and field staff were implementing correctly. In other words cascade models still require involvement from the core sub￾partners, but at the same time need to build the capacity of the sub-partners to conduct training and supervision. They need to be supported in the cascade process. There is a need to document methodologies, manuals and guidelines to ensure standardization and continuity in programming. 3.1.7 Conclusions The overall conclusion is that Maatla has helped to strengthen the HIV/AIDS response at national level by contributing to the organizational strengthening of the CSOs responsible for implementing the civil society response. The main OD achievements for Result 1 were the strengthening of the governance of the participating organizations, and creating a group of governance trainers who can continue to strengthen this aspect of civil society in Botswana. A number of respondents indicate that challenges with governance have been the root cause of much of the funding problems of CSOs in Botswana. Three out of the four implementing organizations took full advantage of their participation in Maatla (even though their advancement was hamstrung by their small size), but a strategic opportunity was missed with BONASO to deliver on their mandate as per the strategic framework for Botswana HIV and AIDS service organizations. 26 There has been progress in terms of strengthening the organizational and technical capacities of the other implementing organizations, and these organizations are in a better position than they were at the start of the project. Due to the collapse of BONEPWA+ and BONASO at the start of the project, the ambitions for the project had to be scaled back somewhat, but at least they are now in a position to receive funds from the government directly, and both have done so. The exit of BOCCIM affected the resource mobilization aspect of the project, and, while organizations have managed to raise additional funds, the amounts are not as high as those received from Maatla. A picture emerges of an over-supply of capacity building for a small pipeline. Expectations that the organizations would cascade organizational capacity down to the districts and to non-Maatla districts were not realized due to a lack of funding. USAID’s expectation to meet high service delivery targets serves the purpose of achieving high results during the program cycle, but causes instability in the CSO sector in the long run. The allegory is of a river that floods causing a deposit of fertile soil, but when the flood retreats it leaves devastation in its wake; this is because there are few other funders willing to fund to the scale of USAID. So it is a bit of a catch-22 situation, and unless the dams have been built to catch the rainwater, when the floods are over the benefits are also lost. The main technical contribution of the Maatla project at national level has been to help translate government policies and guidelines into HIV/AIDS programs and to develop models around these that were being implemented in the districts. This is a positive contribution to the health system as long as the implementing organizations are able to continue implementing them. Funding for posts was a critical enabler for success. Organizations were able to progress in certain domains where they had posts funded by Maatla. For example, BOCAIP received funding for two finance posts—finance officer and a finance manager—and this allowed them to run their finances properly. 3.2 Result 2: A Model Developed and Implemented to Strengthen District Level Systems to Provide and Coordinate HIV/AIDS Services in Hard-to￾Reach Areas 3.2.1 Introduction The intention of Result 2 was to develop and implement a model to strengthen district level systems to provide and coordinate HIV/AIDS services in hard-to-reach areas. Creating a vibrant, cohesive program of HIV/AIDS at district level requires the dedicated participation of all key actors: government, civil society, the private sector and intended customers/target groups. They will need to competently play their own roles individually and to collaborate on activities that require joint responsibility”( FHI360, 2012). The sub-objectives were “capacity assessment at district level” in order to tailor-make organizational capacity building to the need of the organization, “strengthening the CSO response to deliver services in line with their objectives and targets”, “capacity building of local government”, “ strengthening of CSO organizational leadership”, “defining/redefining roles to make CSO and government more effective partners through relationship building”, “coverage of beneficiaries” to ensure hard-to-reach communities are reached, “public-private partnerships to strengthen services and sustainability” and “to provide an enabling environment for policy making and service delivery”. 27 Figure 8 Result 2: Interlink between Activities (based on FHI360 2012) Government bodies able to lead and coordinate HIV/AIDS activities • Effective coordination among all partners • Strengthened organizational leadership and service delivery capacity Shared outcomes • Financial, human and other resources prioritized through collaborative decision making • Strengthened and coordinated district level M&E systems • PPPs to strengthen services/sustainability • Enabling environment for policy making and service delivery • ICT platform for improved service delivery and systems functioning CSOs able to deliver services • In line with their objectives • Meeting targets • Demonstrating sustainability and relevance • Strengthened organizational leadership Communities fully engaged, beneficiaries receiving services/customers demanding quality Result 2 refers to a model being developed and implemented in the districts. However, no integrated model was ever developed for this, either before or during implementation. FHI360 respondents indicated that in fact four models were developed and implemented for the district, namely the following: 1) Model for government/CSO coordination (through the support of NASTAD) 2) Model for CSO-CSO coordination (through NCONGO, BOCONGO and BONASO) 3) Technical model for BOCAIP/BONEPWA+ and the GoB integrated service delivery 4) Model for data quality assurance None of these models was ever documented as a model, with the exception of the revised technical model (FHI360, 2014). NASTAD did produce the Botswana’s District HIV and AIDS Coordination Model: A Guide for Civil Society Organizations (NASTAD, 2015). However, this is a guideline with a toolkit and not a model description as such. These are all elements that could still be woven into a district level model. 3.2.2 Capacity Assessment at District Level For capacity assessments at district level, the program description document suggests differentiating between 1) assessment of individual CSOs and District AIDS Coordinators (DACs) as they enter the Maatla project and 2) joint CSO/local government annual reviews to improve annual district wide multi-sectoral planning and program implementation. For 1) the assessment of CSOs and DAC, an assessment tool will be developed based on FHI360 TOCAT and NCONGO’s CSO assessment model for CBOs. Other components, such as gender mainstreaming, would also be included. Based on the capacity assessment, capacity building plans will specify the proposed action steps for organizational and technical strengthening. For 2) the assessment of joint CSO/local government reviews, the program description document refers to NASTAD expanding previous efforts to 28 review and update DAC and District Multi-Sectoral AIDS Committee’s (DMSAC) capacity to develop effective, evidence based annual district level planning and introduce results-based management. NASTAD’s Evidence Based Planning Toolkit would be adapted for this process (FHI 360, 2012). Assessment of DMSAC/Government Coordination and Collaboration in the Model Districts During FY11-12, NCONGO, NASTAD, FHI360 (and BONASO) introduced Maatla to the four districts’ DMSACs, District Commissioners and CSOs. NASTAD wrote a report on the availability of and mechanisms for accessing district level funding in these four districts. NASTAD, in consultation with FHI360, also finalized standards for district level government HIV/AIDS coordination and collaboration systems, and assessments of capacity to coordinate district HIV and AIDS for the four districts were carried out. During FY13 NASTAD supported the four districts, to develop their district level government coordination and collaboration-strengthening plans. NASTAD subsequently supported Ngami, Chobe and Ghanzi to develop a road map to address issues that impede coordination and collaboration. Mid-term assessments conducted in FY13-14 on government and CSO coordination and collaboration revealed that all four districts have reached a minimum threshold or target set for coordination and collaboration (70 percent) and, on average, districts improved by 30 percent. It is uncertain from the annual reports whether an annual reflection and review meeting for AIDS service providers and stakeholders in each of the four districts did take place. Organizational capacity assessment With regard to capacity assessment tools, FHI360’s TOCAT was combined with NCONGO’s tool, as well as the former AED “OCA” tool and later incorporated additional standards relating to gender. WUSC supported the development of the NCONGO OD plan based on the outcome of the OCA. However, facilitated participatory self-assessments of the remaining coalitions did not take place, given the stage of resuscitation by the end of FY13. Throughout the program implementation, NCONGO regularly completed OD capacity assessments with their members in Ngami and Okavango districts. NCONGO supported the development of OD Plans for six of its member organizations. OD assessments were also completed in Ghanzi and Chobe by BONASO with the support of FHI360 district staff and the ODAs. During FY14, five OD plans were drafted in Chobe and seven in Ghanzi. Also, BOCONGO conducted capacity assessments of members around Gaborone. Finally, WUSC facilitated a district OD planning exercise in collaboration with BONASO, using a planning toolkit. BONEPWA+’s District Support Group Coordinators (DSCGs) continued to hold meetings with support groups to capacitate them on OD. BONEPWA+ DSGCs and volunteers also participated in district planning and coordination meetings, workshops and trainings organized by NASTAD in collaboration with DACs, DMSAC and DHMTs. As part of handing over CSO organizational development support from WUSC to BOCONGO, WUSC supported BOCONGO in conducting training on OCA-Lite in February 2015. It was raised in the FY15 semi-Annual Report that ODAs in Ghanzi, Chobe and Maun will also assist BOCONGO and NCONGO to host strategic planning workshops in their respective districts, and WUSC will work closely with NASTAD to enhance the relationships with local government structures to advocate for integration of OD plans in the annual district evidence based plan (EBP). During the evaluation, the OCA-Lite tool was applied to five district organizations and the overall result of the OCA-Lite organizational assessment is presented in the bar graph that follows. Thuso, a member of BOCONGO, was the most highly rated organization, scoring 92 percent overall, 29 achieving the status, “mature”. In contrast, the organization receiving the weakest rating was BOCAIP at 65 percent7, giving it the status, “developing:. NCONGO and BOFWA are both classified as ‘expanding’ organizations. BONEPWA+’s score between 70 and 79 percent earns it the intermediate status ‘developing’. Figure 9 OCA-Lite Overall Ratings The scores for each organization across the domains that make up this total score can be comparatively assessed in the radar chart that follows. Each corner of the heptagon in the chart represents a domain of the OCA-Lite score, with scores ranging from their lowest in the center of the graph at 46 percent to 100 percent at the edge of the heptagon. 7 This finding is interesting when it is compared to the national TCA rate for BOCAIP (section 4.1.1.2) where they scored 97 percent. This could lead to the conclusion that while BOCAIP national is strong on the technical capacity they seem to be weaker on the organizational capacity. 30 Figure 10 OCA-Lite Ratings by Domain The domain performance in the radar graph above shows BOCAIP’s overall weak rating is most significantly influenced by a low infrastructure rating, followed by a weak strategic leadership rating and a comparatively poor service management score. In contrast, BONEPWA+’s weakest domain ratings are infrastructure and process management. While Thuso’s overall score is greater, NCONGO’s performance is more even across domains. BOFWA’s domain ratings appear skewed to strategic leadership, governance and finance, while other domains receive intermediate ratings. Individual domain ratings will be explored in the comparative discussion of each domain in Appendix VIII. Gender Assessments of BOCONGO members in the non-HIV/health sector WUSC assisted BOCONGO to create a gender audit tool for districts, which was then piloted. FHI360 assisted BOCONGO with gender analysis training in Maun. WUSC conducted gender analysis training for WUSC Uniterra members (co-facilitated by BOCONGO’s Gender Officer) in Ngamiland and Okavango districts, and seven organizations were assessed. M&E During 2013, tools and processes for QA/QI were introduced to the national NGOs. District level standards and assessment tools were planned to be finalized in FY14. These were included and incorporated in the SIMS tools. Technical capacity assessment After a thorough consultative process in FY12, technical competency standards were developed and during FY14-FY15 BONEPWA+ and BOCAIP were supported to conduct a technical assessment of their centers. In the same period FHI360 District Coordinators started to extend HIV program related technical assistance beyond Maatla partners (BOCAIP and BONEPWA+) to other CSOs implementing HIV/AIDS programs in Maatla target districts. A total of 20 organisations were 31 supported to conduct technical assessments. The results of their assessment were discussed with each organization, and TA plans developed. During the evaluation, the TCA tool was applied to three district organizations and the overall results of the Technical Capacity Assessment are presented in the bar graph that follows. Figure 11 External Overall Scores TCA for District Offices of BOCAIP, BONEPWA+ and Sekgele The two core Maatla organizations did very well in the assessment using TCA rating criteria, while Sekgele only received a score of 63 percent. This difference could be evidence of the benefits of the Maatla program, as Sekgele only received minimum intervention from the Maatla Program. It should be noted that the majority of the competency criteria for the domains, technical planning and design, and staff capacity were deemed not applicable in relation to Sekgele. Figure 12 External TCA Ratings for District Offices of BOCAIP, BONEPWA+ and Sekgele 32 Table 5 Maatla core quantitative indicators Indicator Latest Result as per Date of Data Source Data Source Date of Data Source # / % of people successfully referred for targeted clinical services (e.g., PMTCT, SMC, HTC, ART, TB/HIV) Not available at time of reporting # of targeted populations reached with individual and /or small group level preventative interventions (adults, youth, PLHIV) 15957 Annual Report FY 15 # of eligible adults and children provided with a minimum of one care service. Removed. Used to report on NGI changed MER. This is indicator discontinued. District reporting rate for CSOs, and mean improvement in reporting rate. Not available at time of reporting Mean improvement in district coordination and collaboration between government and CSOs Not available at time of reporting # of community health workers trained in HIV￾related service delivery. 18 (Kataleng District trained in GBV) Annual Report FY 15 # of patients receiving community DOT Not available at time of reporting # of individuals reached by activities which promote the reduction of gender based violence 18336 (# of people reached with Gender Norms and GBV Prevention in year IV). Annual Report FY 15 # CSOs reached within the model districts Not available at time of reporting # of accredited training/capacity building materials developed by NCONGO Not available at time of reporting # of member organizations receiving trained or mentored by NCONGO Not available at time of reporting Organizational and technical capacity indicators as above, but for targeted districts and district￾level CSOs See detailed report in appendix VII 3.2.2.1 Strengths/Enablers of the Capacity Assessments at District Level With regards to strategic leadership, the various organizations report that the Maatla program and particularly WUSC, have assisted them with the development of a strategic plan to guide their organization. Promotional/communication materials such as T-shirts, sunhats, bags, gazebos, brochures, posters and signage were provided. These materials promote visibility of the organizations, which enables them to do resource mobilization. 33 Maatla on its own has made us more visible than before and has placed us on better ground to go out there and do resource mobilization. It has furthermore capacitated us in resource mobilization. It took us to a different stage. (OCA-Lite, district organization) As a result of the Maatla program, NCONGO is in the process of creating a business arm offering accredited training to generate funding for its members. In terms of organizational structure, a strength is that the Maatla program has funded a six-months’ training course in governance for some of the CSO staff members. Assistance has also been provided in the development or revision of constitutions. We have a stronger board that are all trained and up-to-date with our model, and they have revised our constitution and produced a governance manual for the board. They have also revised a number of management policies on human resources. (OCA-Lite, district organization) Most of the respondents reported that their board has improved and board members have clearer ideas of their role. The board is more playing its real role as an advisory body and leaving management to the secretariat. Trained staff are using the skills learned in the governance course and cascading it to the support group members. (OCA-Lite, district organization) Finally, the Annual General Meetings are held regularly as a result of the Maatla program. A strength in the human resources domain is that NCONGO has developed HR policies and is supporting its members to have the same, with support from WUSC. Through the Maatla program the organizations have increased their financial management capacity, which has enabled the various organizations to be audited and given them better possibilities for successful resource mobilization. Organizations such as NCONGO reported that they have developed financial templates and that they have been capacity built in financial management by or with the support of FHI360. These skills and templates are cascaded to their membership organizations. In terms of infrastructure and IT, organizations in the districts reported that they had received equipment such as computers, faxes, projectors and filling cabinets from the Maatla program. This has enabled them to be more efficient, particularly when they have to report or carry out research. It is hoped that this equipment can be kept by the organizations once the program comes to an end. A strength pertaining to program and service management is that each organization has clarified its mandate. Also clear scope of work, work-plan matrixes and printed annual reports are some of the benefits mentioned by the organizations as a result of the program. With regard to process management, technical M&E support was provided by FHI360 to district organizations. As a result, good quality data was collected and data collectors have been trained on the data collection tools and to understand the importance of M&E. One of the enabling factors was that the FHI360 M&E officer was placed in the district. She assisted the organizations to provide accurate and timeously monitored data. Particularly NCONGO and BOCAIP benefited from this support as they developed an M&E plan and tools with the support of the FHI360 M&E officer. 3.2.2.2 Challenges/Barriers of Capacity Assessments at District Level With regard to strategic leadership, concerns were raised particularly around resource mobilization as there is no dedicated staff in the organizations to undertake this. Despite efforts by NCONGO in facilitating proposal writing. it was felt that more coordination is required and that organizations are still competing for funds. It was also mentioned that going forward promotional materials and marketing is expensive and not possible for district organizations to replenish. In terms of organizational structure, one respondent mentioned that her/his board only has a few skilled members on board. A challenge with this board is that these board members have limited knowledge of the role of board members versus management. 34 Some of the challenges pertaining to human resources are that none of the organizations has a dedicated HR officer and they feel ill-equipped to deal with HR issues. With regard to the volunteer management, it was reported by at least two organizations that volunteers are not paid on time and that their working hours are often beyond the stipulated 24 hours. With regard to financial management, one of the organizations reported that their finances were dealt with at national level and by FHI360, which often results in delays in payment due to the requisition period. It is a concern that FHI360 was unable to establish procedures for accountability purposes so that this organization could manage their own funding. In terms of infrastructure and IT, a few of the organizations reported challenges such as no budget for maintenance of infrastructure and IT, not enough office space, and for one organization, no electricity or access to the internet. One challenge concerning program and service management raised by NCONGO was that only membership organizations in the HIV and AIDS sector are benefiting from the program and service management support from Maatla. With regard to process management, NCONGO stated that it is a challenge to get the right data from the organizations in time, which leaves them with lengthy verification processes for producing the consolidating report for DMSAC. Furthermore, NCONGO does not have a dedicated M&E officer to oversee this process, particularly when the FHI360 M&E officer is no longer there. It is therefore a concern whether the M&E efforts by the Maatla program have equipped the district organizations sufficiently so that they can produce their own M&E plans and continue collecting quality MIS data. 3.2.3 Strengthening the CSO Response at District Level to Deliver Services in Line with their Objectives and Targets NCONGO’s CSO coalition model formed the basis for strengthening the CSO district response. NCONGO’s model unites CBOs to improve results through formal capacity building and joint planning and action, but also focuses on all CSOs contributing with their unique core competencies (FHI360, 2012). The Learning and Innovation Site at NCONGO (LISN) was opened and accredited in FY12, and launched and equipped with computers in FY13. It was then moved to Love Botswana Life Center. During FY15, eight of the 21 training modules of the LISN Center were accredited by the Botswana Qualification Authority (BQA). As a result of organizational capacity assessment of the CSOs, terms of references were developed, consultants identified and training provided. A list of all the training conducted can be found in Appendix IX. Linkage for delivery of computer-based learning was explored during FY12, but it never materialized. Throughout the four years of implementation, NCONGO facilitated regular networking forums with representatives from government, the private sector and CSOs in Ngami and Okavango districts. NCONGO shared its experience and Networking Forum Manual with BONASO and District ODAs in Ghanzi and Chobe, to inform coalition establishment in those two districts. Although quarterly review meetings were planned, NASTAD and NCONGO facilitated annual project review and M&E meetings with partners in the four districts. These reviews informed the annual Maatla plans. During FY13, NASTAD facilitated SWOT (Strengths, Weaknesses, Opportunities and Threats) analyses in the four target districts to identify factors key to achieving district level objectives and strengthen DMSAC relationships. The purpose was also to establish same service forums for CSOs and government working in the same technical areas. While they worked well and are active in Okavango for the sectors of HCT, OVC and CTBC, they never worked in the Chobe and Ngami districts as there were gaps in the service inventories. 35 During FY13, NCONGO drafted its private sector strategy to access funding for its members from the private sector. In FY14, NCONGO facilitated a two day workshop for Mababe, Zokotsama Community Development Trust, on board meeting planning and minute taking, and also shared its appraisal procedures with two organizations. NCONGO also provided financial management mentoring to the Okavango Artists Association. During FY15, with the support of WAR and BOCONGO’s ODA, NCONGO assisted six organizations to develop their strategic plans and review their constitutions. Also, WUSC continued to provide technical assistance to CSOs in the Maatla model districts through ODAs based in Ngami, Chobe, Ghanzi, and through one additional ODA placed with NCONGO, specializing in governance and strategic planning. On-going assistance was provided by the ODAs mainly through training, coaching, assistance with proposal writing, and sharing of technical documents/materials. 3.2.3.1 Strengths/enablers of strengthening the CSO response at district level to deliver services in line with their objectives and targets Enablers for strengthening the CSO response at district level have in general been the financial, technical and organizational development support provided and the use of tools, such as standard operating procedures and a communication guide on community education, for CSOs. Strengths have also been to empower CSOs to fill the service delivery gap that government is unable to fill and to make CSOs and government work together. 3.2.3.2 Challenges/barriers of strengthening the CSO response at district level to deliver services in line with their objectives and targets Challenges mentioned include that some volunteers do not have a strong educational background and are therefore in some cases not trainable. This makes it difficult for them to understand the data collection tools and apply them correctly. 3.2.4 Capacity Building of Local Government This output focused on strengthening local government to lead and coordinate HIV/AIDS activities through the support of NASTAD. A range of consultative meetings was held with TACs, DACs, DMSAC, MLG&RD and the District Health Management Team (DHMT) to introduce the program and to provide feedback on its progress. Throughout the program implementation, NASTAD facilitated and supported the development and implementation of the district level government coordination and collaboration strengthening plan. NASTAD also held consultative meetings with Ngami and Okavango TACs to incorporate Maatla activities into district HIV and AIDS plans. The Maatla program activities were implemented with DMSAC in all four targeted districts. NASTAD undertook an assessment into why some government departments and private sector organizations were not attending DMSAC meetings. The findings of the assessment were disseminated to stakeholders such as DAC, DMSAC and MLG&RD, and in FY14 the Ngami district adopted some of the recommendations and included them in the district HIV/AIDS response plan. NASTAD also developed a DMSAC orientation package for members. According to the FY15 semi￾Annual Report, NASTAD will facilitate an orientation workshop for DMSAC/TAC members in the four project district to familiarize them with the DMSAC orientation package. NASTAD furthermore developed a needs assessment tool for DMSACs, which was shared with MLG&RD and other district stakeholders. In FY13 a workshop was facilitated on gender mainstreaming within DMSAC and local government in the four target districts through the support of Women Against Rape (WAR). In FY14 NASTAD provided technical assistance in Charleshill, Ghanzi, Chobe, Ngami and Okavango DAC offices through on-site visits to provide one-on-one situational coaching and mentoring to key 36 stakeholders. NASTAD shared a service inventory data form with all the districts and the districts developed district service inventories. The directory for Ngami was finalized and others drafted. In FY15 NASTAD disseminated findings from the mid-term assessment on district level CSO and government coordination and collaboration to all four districts and carried out one-on-one situational coaching and mentoring for DACs, TACs and DHMTs, based on assessment findings. Finally, Ngami and Ghanzi districts held annual reflection and review meetings with HIV and AIDS service providers and community leaders and with the assistance of NASTAD. 3.2.4.1 Strengths/enablers of capacity building of local government An enabling factor is that local government officials such as DMSAC, TAC, DHMT members have been trained and that NASTAD has brought district CSOs and local government together. One of the strengths is that district referral systems were developed and are being implemented. There are even statistics on the referrals. In the quarterly DMSAC meetings, progress, including status on referrals, is reviewed. Also through the production of the consolidating report by NCONGO, the CSOs speak with one voice and the report improved significantly. An added strength is the development of guidelines, standard operating procedures and orientation packages. These can be used in the future when the Maatla project is closed down. “We are going to the community as one entity. All CBOs were always there to plan with government as an entity. This brought together limited resources to be used for delivering services to the community” (SSQI-DMSAC). 3.2.4.2 Challenges/barriers to capacity building of local government One of the challenges mentioned is that although the district orientation package has been developed, none of the new DMSAC or TAC members had gone through orientation at the time of fieldwork in August 2015. The DAC office should have had clear and transparent procedures for reviewing and vetting proposals, but apparently these has not been developed yet. 3.2.5 Strengthen CSO Organizational Leadership The purpose of this output is “to build on NCONGO’s success in coalescing CBOs into cooperative coalitions, by replicating or adapting this model in our other target districts” (FHI360, 2012). The first set of activities revolved around the continued strengthening of membership engagement by NCONGO for Ngami and Okavango CSOs. Monthly networking forums were conducted with good attendance in both Ngami and Okavango districts. A CSO mapping study was carried out in both districts. In FY14, BONASO conducted support visits to coalitions to facilitate the development of strategic plans to feed into the development of their year 4 scope of work. Throughout the duration of the program, NCONGO continued to support its members with their activities, such as holding AGMs, participation in the Sixteen Days of Activism and Men’s Sector events. NCONGO also facilitated CSO’s response to calls for proposals with assistance from the volunteers placed at NCONGO, and as a result three NCONGO member organizations were awarded the EU-EIDHR grant. NCONGO members were meant to sign the NGO Code of Conduct, yet this is not reported in the annual reports. The second area of implementation was to establish and support district coalitions in the Chobe and Ghanzi districts. After a slight delay, several consultative meetings were held between FHI360, NCONGO, BONASO and BOCONGO regarding the appropriate model for establishment of CSO coalitions in Chobe and Ghanzi and joint introductory meetings were held in these districts. Roles and responsibilities were clarified between the different partners including BONASO, NCONGO and the national networks and incorporated in the FY12 work-plan. As part of the establishment of a district coalition in Chobe, a CSO mapping exercise was conducted using GIS. Likewise, a CSO service mapping of Ghanzi was carried out in FY13. Also in the same year site visits were planned to Chobe CSOs to administer capacity assessments in preparation for supporting the establishment of a Chobe CSO network. However, it is uncertain if these assessments took place. An interim coalition structure was formed in Chobe in FY12. FHI360 staff and WUSC ODAs based in Ghanzi and Chobe supported BONASO in maintaining the coalitions in Ghanzi and Chobe. NCONGO also made a 37 presentation on its coalition structure at the Chobe and Ghanzi coalitions meeting, providing an overview of its membership base and sectors. The issue of membership for non-BONASO members was a challenge. The district-based ODAs in Ghanzi and Chobe supported CSOs to affiliate to BONASO. At the start of FY15, WUSC seconded ODAs to BOCONGO to provide technical assistance to coalitions in Organizational Development. NCONGO and the BOCONGO ODA supported the Chobe and Ghanzi Coalitions with the registration process, including developing and facilitating a meeting to discuss both a draft constitution and requirements for registration. In the third quarter, both coalitions were registered as Chobe Alliance of NGOs (CANGO) and Ghanzi Alliance of NGOs (GHANGO). The third area of implementation was to provide leadership training for NCONGO members, other CSOs and district government officials. Interim capacity building plans were put in place for NCONGO members at the start of the program. Leadership development plans for NCONGO members were planned but not reported on in the annual reports. During FY14 it was agreed that NCONGO will include Chobe and Ghanzi coalition members in their OD training, based on needs identified through the OCAs. 3.2.5.1 Strengths/Enablers of CSO Organizational Leadership One of the strengths of the approach of organizational capacity training is that it is based on the capacity assessment. Ngami district became the learning site on the district model for CSO coalition through NCONGO. They were not only responsible for coordination and capacity building of members also for starting similar coalitions in Ghanzi and Chobe. The respondents felt that NCONGO did a good job in these roles. The core member organizations felt supported by NCONGO, FHI360 district staff and the WUSC ODAs. “I feel more supported as a member. Maatla provided experts to support the network. Through NCONGO we are more able and they are representing the CSOs (SSQI-district organization). The organizational development support provided by WUSC, together with the tools developed by them, were enabling factors for the success of the program. NCONGO was also able to coordinate and clarify roles and responsibilities of members. Through the financial and organizational capacity support provided by the program, NCONGO increased their authority as a network organization. They have even registered as a training institution so that they can start generating funding for their members. Another of the main enabling factors for supporting the re-establishment of coalitions in Chobe and Ghanzi was that WUSC ODAs who have previous experience in providing organizational development for CSOs were placed in those districts. 3.2.5.2 Challenges/barriers to strengthening CSO Organizational Leadership BONASO’s original mandate was to establish HIV coalitions in the districts, but to have HIV specific coalitions in the districts with few HIV organizations did not make sense. Hence the NCONGO model was adopted as a model for more broad-based coalitions. BONASO was supposed to design their model on NCONGO, but there was no working relationship between the two to make sure that these models were built. It took a considerable amount of time and facilitation by FHI360 to establish this model. Coalitions have been established in Ghanzi and Chobe districts. However, concerns were raised as to how sustainable those structures would be once the program closes, despite being registered and having the support of NCONGO. A core issue is resource mobilization, as the coalition is mainly comprised of HIV/AIDS service organizations without operational funds, and they had experienced difficulty in convening meetings. Although the coalitions benefited from the placement of the WUSC ODAs in the districts, it was raised that they came too late into the implementation of the Maatla program. 38 One additional challenged mentioned by a non-core CSO was that although they had received governance training there was insufficient support for them to implement what they had learned, e.g. having separation of power between the board and management and having a strategic plan. 3.2.6 Defining/Redefining Roles to Make CS and Government More Effective Sub-partners through Relationship Building The focus of this output is strengthening and institutionalizing collaboration between local government and CSOs in delivering health services. Capacity was to be built to strengthen government and CSOs to play their role individually and together to create a cohesive, seamless, cost-effective and sustainable service delivery system. Firstly, NASTAD would build the capacity of DACs and DMSAC to lead, plan and coordinate, assist in CSO technical skills development, to ensure service delivery according to quality practice standards, and to ensure two-way referrals and follow-up of clients. Secondly, FHI360, NCONGO and WUSC would build the capabilities of CSOs to address the needs of most at-risk persons. A third aim was to strengthen district financial and other resource priorities through collaborative decision making in DMSAC, DHMT and DDC before submitting to central government (FHI360, 2012). NASTAD commenced in FY12 by conducting district capacity assessments in order to inform capacity building strategies. During FY13-FY14, NASTAD held workshops in Ngami, Chobe, Ghanzi and Okavango districts to strengthen DMSACs and to clarify roles and responsibilities between local government and CSOs. A model was developed for strengthening the interface between CSOs and district local government in terms of coordination and collaboration for the provision of health services. An additional model was developed in FY14 to ensure strengthened interface for coordination of HIV and AIDS services in hard-to-reach areas. NCONGO represented its members on the Ngami and Okavango DMSACs, while the interim committee in Chobe represented CSOs at the Chobe DMSAC, and the Ghanzi coalition is now a member of the Ghanzi DMSAC. Furthermore, in Okavango, NCONGO and some of its members sit on the District Child Protection Committee and Health Planning Committee. NCONGO also took part in the evidence based planning process for Ngami and Okavango districts. NCONGO membership continued to work with the DAC office on activities that address priority issues identified in the 2013-2014 Ngami Comprehensive HIV/AIDS Plan. NASTAD conducted discussions to strengthen collaboration between MAC’s focal person for Education, Agriculture, and other key ministries and TAC. In FY14, NASTAD successfully supported setting up forums for same service providing sectors/committees in Chobe, Ghanzi and Okavango districts with strategic leadership and direction to ensure collaboration, joint planning and a highly responsive service provision to the community. During FY15, NCONGO mobilized Ngami CSOs to participate in the New Directions for Global Health Forum facilitated by PEPFAR and the DAC office. USAID, in partnership with the DAC, also facilitated dialogue among DMSAC stakeholders including government health care professionals, and civil society organizations. 3.2.6.1 Strengths/Enablers of defining/redefining roles to make CS and government more effective partners through relationship building The Maatla program was successful in developing and testing a model for bringing government and CSOs together to improve multi-sectoral HIV and AIDS service delivery through strengthening DMSACs. The extended consultative process followed for the relationship building was a strength of the program. Before a model was developed, the process started with assessments of each of the DMSACs. Based on the district DMSAC assessment, a workshop was conducted in each district with DMSAC members to clarify roles and responsibilities. Standard operating procedures/guidelines/a manual were developed on how to facilitate DMSAC. The participatory development or update of the District Service Inventory to facilitate referrals and the district referral tool were also an enabler for better collaboration and coordination between the government and CSOs, which ultimately ensured that clients received improved services and stayed on their treatments. Reporting by the CSOs to DMSAC improved through the consolidated report produced by NCONGO (and other 39 coalitions) on behalf of their members. Also CSOs and particularly BONEPWA+ improved their relationship with the health facilities as trust was built of the volunteers’ capacity. One enabler was the realization by government that CSOs are equal partners and closer to the community (which was manifested through, e.g. trusting CSO volunteers to extend health services in the communities). Another enabler was the extensive efforts and government experience of NASTAD to facilitate this relationship building. 3.2.6.2 Challenges/Barriers to defining/redefining roles to make CS and government more effective partners through relationship building Although NASTAD initiated separate thematic area referral meetings for government and CSOs these meetings were not carried out. Hence it was decided to have referrals as part of the agenda of different units/departments, namely S&CD, DHMT, TAC and DMSAC. A challenge was also that the private sector was not on-board of DMSAC. A final challenge is also that referrals are not part of the DHMT’s performance indicators and hence might not be sustained. 3.2.7 Coverage of Program Beneficiaries The approach outlined in the program description document was for FHI360 to concentrate its efforts most directly on primary beneficiaries, namely the CSOs, networks and the local DMSAC, who in turn would improve quality and coverage reaching the secondary beneficiaries, people in the communities. Coalitions of CSOs would be established and supported as a means of achieving extensive coverage in hard-to-reach areas (FHI360, 2012). As part of Result 2, Maatla has been supporting two partners (BOCAIP and BONEPWA+) to provide HIV and AIDS services in four health districts of Botswana (Ngamiland, Okavango, Ghanzi and Chobe). Figure 13 Types and Locations of Services Provided by the Two Partners in the First Three Years of the Program8 8 Concept paper on Revised Maatla Technical Model (August 2014) 40 Figure 14 Draft Maatla Model for Family-centered CSO-government HIV and AIDS Service Integration in the Fourth and Fifth Year of the Program For the first three years of implementation, the Maatla Program service delivery model was focused on where the two partner organizations were providing which services. In the last two years of implementation, the service delivery model was taking more cognizance of the role of the family, community and the health facility plays and emphasized the importance of referrals. In FY12, after initial briefing with the four national organizations, FHI360 signed agreements with BOCAIP and BONEPWA+ including a scope of work and budget for direct service delivery in the four focus districts. BOCAIP recruited community mobilizers, OVC counselors, HCT counselors, and TB support workers for Ngami and Okavango while BONEPWA+ recruited support group volunteers and two district Support Group Coordinators for Ghanzi and Kasane. District staff were provided technical training in service delivery and application of monitoring tools by district-located FHI360 staff. Service delivery started shortly afterwards. During FY13-FY15, FHI360 district staff provided continuous support in the Maatla districts to partners (BONEPWA+, BOCAIP) and their volunteers. In FY15 Kuru Health came on board as a Maatla partner to provide CBTBC services in the Ghanzi district. Kuru staff and volunteers were trained in the integrated family model, which emphasizes collaboration and referral linkages across the continuum of care services, HIV prevention, and CBTBC and gender transformative norms using revised communication guides. The district officers also conducted quarterly support visits to all volunteers to monitor the quality of services provided. Furthermore, FHI360 facilitated regular meetings with district-based stakeholders to improve referral linkages including DACs, DHMT, and S&CD officers. BONEPWA+ and BOCAIP expanded their services to the hard-to-reach areas in the Maatla districts and recruited volunteers. 41 3.2.7.1 Strengths/enabler of coverage of Program Beneficiaries Overall, results of the Maatla project are that there are decreased default rates on treatments (both TB and ART treatments) and there is increased awareness of HIV and AIDS prevention in the communities. Recruiting volunteers from the communities who are trained and intensively supervised by organizations such as BONEPWA+ and BOCAIP and using tools for CSOs such as standard operating procedures and communication guides on community education have all been enablers for the success of the program. “The strengths of the volunteer model is that it provides better quality of service; they are closer to the community and thus understand the community culturally and language wise” (SSQI District organization). With regard to prevention, the strategy of focusing on better packages of messages for partner organizations, coupled with more targeted issues such as safe male circumcision and preventing further spread of infection for people living with HIV are strengths of the program. In terms of HIV counseling and testing, BOCAIP provided mobile testing and used opportunities as community gatherings to do testing. As a result more people have been tested. It was furthermore raised that BOCAIP provided good quality counseling and support services. With regard to community TB care, a strength is the link that has been formed between the health facility and the volunteers. As a result there are fewer defaulting cases. We offered DOT in the comfort of their homes. The mobilisers take the medication to the homes and they work hand-in-hand with the nurses. We used to have 81 patients and we now only have 28 as the rest have finished their treatment. (SSQI District organization) As for care and support, ART adherence has improved as a result of community awareness creation on disclosure, stigma and discrimination, and services such as basic counseling and referrals for psycho-social support. With regard to caring for OVC, the volunteers benefited from the training and BOCAIP’s communication guide. They worked closely with the social workers, assisted them where they could in terms of identifying, reaching, translating and birth registration of the OVCs. As a result the OVCs were provided with more holistic and better services. It was furthermore reported that some of the drop￾outs were back in school. The partner organizations and their volunteers gained greater understanding on gender, gender mainstreaming and How the Technical Capacity Building Model Strengthened relationship with Health System Personnel and Provided Better Coverage of Health Services Before the Maatla Program there was little cooperation between health clinic personnel and CSOs operating in the sector. The health personnel did not trust volunteers from the CSOs to assist them in providing health services. “For a long time government and CSO were not best friends. There were competition and no close collaboration. Government did not understand our role as CSO and they did not know how they could benefit from cooperating with us. Secondly there was no proper referral of clients between Government and CSO” (SSI-District Organisation). The Maatla Program provided training of the volunteers of BONEPWA+ and BOCAIP and facilitated the interaction between the Health System personnel and the CSOs. The technical capacity building of volunteers enabled them to provide better services and as a result they are more trusted by the health system personnel. The health system personnel refer cases to the volunteers for e.g. follow up and tracing patients. The advantage is that volunteers come from the community where they are placed. They know the challenges of the community members well and they do not struggle to create trust with community members as they are part of the community. Maatla project played a big role in making thorough follow-ups. It is difficult for us to do follow-ups (SSI- DG-S&CD) As a result of the Maatla Program the number of people defaulting on treatment has decreased and there is increased level of awareness of HIV among community members. 42 gender-based violence as a result of the program. Working with WAR and WUSC, and including GBV in the communication guide and tools, were all enabling factors. Table 6 Key PEPFAR indicators PEPFAR Indicators Latest Result as per Date of Data Source Data Source Date of Data Source # of targeted populations reached with individual or small group level preventative interventions that are based on evidence or meet the minimum standards required Replaced in FY 14. New Indicator Percentage of individuals from priority populations who completed a standardized HIV prevention intervention, including the specified minimum components, during the reporting period Estimated target: 11000 No results are reported on in the MER Q3 Report (Under PP_PREV_TA). However, the following indicator is reported on: Percentage of the target population who completed a standardized HIV prevention intervention including the minimum components during the reporting period (DSD) in the same document. Target: 14360 Results: 14925 Performance Monitoring Plan (PMP) May 2015 # of targeted populations reached/ with individual or small group level preventative interventions that are primarily focused on abstinence or being faithful and are based on evidence or meet the minimum standards required DISCONTINUED IN FY 14 Performance Monitoring Plan (PMP) May 2015 # of people reached by individual, small group or community level intervention or service that explicitly addresses gender based violence & coercion related to HIV/AIDS Classification of PEPFAR support: (Discontinued in FY 14). FHI 360 and the implementing partners are in the process of finazling the revised materials for the new MER indicators to meet the minimum standards requirements for GEND_NORM. However partners continued to provide Indicators and Targets Table final (Based on the NGI and the current Maatla project PMP) FY 14 Q3 30 July 2014 43 the services using the communication guide to provide messages that address GBV. Achievement versus target a total of 1,865 individuals were reached with interventions that explicitly addresses gender based violence and coercion related to HIV and AIDS this quarter. Their age and sex disaggregation as follows10- 14 male 65; 10-4 female 85;15- 29 male 365; 15-29 female 505; 30+ male 282; 30+ female 563. In addition to the number of people reached this quarter 522 were followed up (repeats) with same interventions. # of people who received testing and counselling services for HIV and received their test results Cumulative for FY15: 2827 Target for FY15: 5000 (Results for Q1 FY15 in the source document seem incomplete). MER Report Q3 FY15 # of people living w/ HIV/AIDS (PLHIV) reached with a minimum package of Prevention with PLHIV (PwP) interventions Classification of PEPFAR support: Level 3 Achievement versus target: a total 516 of people living with HIV were reached with interventions using the PHDP sections of the communication guides in ARV clinics and households this quarter. The people reached were of the following age and sex : 10-14 male 0; 10-4 female 1;15-29 male 68; 15-29 female 111; 30+ male 125; 30+ female 211 . By the end of Q3 Maatla exceeded the FY 14 target. Indicators and Targets Table final (Based on the NGI and the current Maatla project PMP ) FY 14 Q3 30 July 2014 # of eligible adults and children provided with a minimum of one care service Classification of PEPFAR support: (Discontinued in FY 14). FHI 360 and is working with implementing partners to provide OVC and Family Planning Services All the care and support data for this quarter are reported under OVC-SERV Indicators and Targets Table final (Based on the NGI and the current Maatla project PMP ) FY 14 Q3 30 July 2014 44 3.2.7.2 Challenges/Barriers to Coverage of Program Beneficiaries Some of the challenges are that volunteers seem to be difficult to retain as the stipend is low. As the program comes to an end and no more funding will be provided for their stipend, they leave the organization. Another challenge is that the coalitions in Chobe and Ghanzi are still at an early stage, which raises doubts as to whether the structures will be sustained once the program comes to an end. With regard to prevention activities, challenges include the insufficient use of community leaders to access the communities, the training of the volunteers in English when they have to implement their activities using Setswana, and not having enoug) condoms for distribution as part of the program. The volunteers were also up against some barriers related to cultural beliefs and norms such as “older people do not want to talk about prevention with younger people” and “prevention services should be targeted at women only”. “Talking to parents, most of whom are elderly people, about such issues as condom use and sexual matters made them uncomfortable. They felt insulted” (FDG volunteers). In terms of HIV counseling and testing, access to these services still presents itself as a challenge, particularly for people living with disabilities and people living in hard to reach areas. One of the hampering factors is that only certain days are testing days. It also appears that men are more reluctant to going for testing. There is a need to focus on men as they do not test as much as women. Generally they are not as forthcoming—not only in HIV/AIDS issues but in health-seeking behavior in general. When you see a man at the hospital he would be really sick. (SSQI DAC) With regard to TB care, it was felt by several respondents that as funding was provided for DOT only, the community TB care was no longer comprehensive in nature. This shift apparently happened when USAID decided to focus on service delivery. The planned TB cluster task team were never established. In terms of care and support there were cases where nurses were not referring clients to volunteers. Also there was a concern as to how to sustain home-based care once the program comes to an end. With regards to OVC, it was felt by BOCAIP that they did not have sufficient technical capacity to provide counseling for OVC. A further challenge was that the social workers were so over￾burdened, causing a backlog in terms of registration of the OVC, that OVCs often received services late. Attitude, social norms and cultural harmful practice are still barriers for addressing gender-based violence in the targeted communities. An additional challenge raised is that the police are not sufficiently trained to respond to GBV. 3.2.8 Public-Private Partnerships to Strengthen Services and Sustainability The focus of this output was to broker innovative engagement of the local private sectors by partnering and strengthening their peers in civil society and local government. It was envisaged that local business councils through BOCCIM would provide capacity building, and businesses would offer internships, long-term coaching relationships and training at the LISN. The private sector was meant to assist the CSOs to improve their financial sustainability through innovative income generation (FHI360, 2012). During FY12 BOCCIM introduced the Public-Private Partnership (PPP) concept to CSOs, NCONGO and the private sector in Ngami. A framework for mobilizing and identifying mentors was developed. With the assistance of the University of Botswana, a mentorship and coaching toolkit and guidelines were adapted. A mentor training seminar was conducted in Ngami to introduce the mentors to the CSOs. Five companies signed a mentorship agreement with BOCCIM. 45 BOCCIM worked with NCONGO to establish a common understanding of the mentoring process and to agree on the selection of both mentees and mentors. The PPP concept was introduced in Kasane and Ghanzi during FY13. BOCCIM also met with BONASO and the incoming North West BOCCIM Council Chairperson to ensure a smooth transition. By the end of FY13, BOCCIM had signed agreements with eleven mentor companies in Ngami, seven in Ghanzi, and four in Chobe, while no agreements were signed in Okavango. Nine mentors were trained in Ghanzi, and four in Chobe. One mentoring session on human resources began in Ngami. BOCCIM convened a private sector engagement forum in partnership with BOCCIM Northern Regional Office in Kasane. Twenty private sector and four council representatives attended. The purpose was to sensitize the private sector in Chobe to the PPP concept. NCONGO worked with its members throughout the year linking them to the private sector. However, this activity was dependent on mentoring as a starting point for private sector support to civil society. During FY13-FY14, the Private Sector Advisor worked with the district team to resuscitate mentoring activities in Ghanzi, Chobe and Ngami and some mentor-mentee pairing started. During FY15, the Private Sector Advisor resigned and FHI360 TA and WUSC ODAs facilitated pairing. There are two examples of these mentoring sessions happening, namely in Chobe with a business man and Chobe Art and in Ghanzi between Kuru and First National Bank. Other initiatives seemed to have failed during this year. Instead of conducting district level training through the private sector to promote an enterprise mind-set, a Training of Trainers (TOT) for CSOs on proposal writing was conducted. 3.2.8.1 Challenges/Barriers to Public-Private Partnerships to Strengthen Services and Sustainability The building of enterprise models at district level by linking the private sector and the CSOs did not work out. BOCCIM reported that mentee organizations were not making themselves available for mentoring after workshops, even after they had confirmed attendance, due to time constraints. The mentors did not feel they required the orientation training, although it was explained that the training introduced them to CSO and not their areas of expertise. Organizations delayed in signing due to change of personnel, and not understanding the benefits of participating in the project. Another challenge raised was that the differences in organizational cultures in the private sector versus the CSOs were too big and BOCCIM failed to facilitate this and eventually left the program in FY14. The expectations that the private sector would provide support were not met. Also there is nothing in the policy environment to encourage this engagement. 3.2.9 Enabling Environment for Policy Making and Service Delivery The aim for this output is to create a space for CSOs, government, businesses, community leaders, church leaders, women leaders, etc. to address critical but sensitive social, cultural and behavioral topics pertaining to HIV and AIDS. By engaging communities in active dialogues, beneficiaries will be engaged and empowered customers that can demand quality services in response to changing needs (FHI360, 2012). It was also anticipated that health service users would be institutionalized by including community leaders on DHMTs and DMSAC. In FY13, FHI360 supported NACA and the national NGOs to engage community leaders and stakeholders in discussing the underlying causes of HIV infection and prevention strategies in the Maatla districts through the national HIV Pitso held in Francistown. This included consultation with Maatla district leadership regarding HIV and related service delivery. Other activities listed in the program description such as “explore the roles of all actors/stakeholders in addressing critical social and behavioral issues which impact on HIV through the DMSAC”, and “conduct training and coaching for faith-based leaders, women’s leaders, 46 community health workers and health center staff to educate their clients on supportive community norms” have not been reported on in the annual reports and hence it is uncertain whether these activities took place. 3.2.9.1 Challenges/Barriers to Creation of Enabling Environment for Policy Making and Service Delivery It is a challenge that the program did not pursue the planned activities as these could have created a more enabling environment for policy making and social services. 3.2.10 ICT Platform for Improved Service Delivery and Systems Functioning The focus of this output is to build the capacity of district CSOs and develop an ICT ecosystem with local ICT companies to provide long-term, in-depth support to CSOs in using ICT for their business operations and service delivery. The ICT ecosystem will work in rural areas, interface with mobile phones through SMS and broadcast content through social networking (FHI360, 2012). In FY12, Inveneo was selected as the ICT service provider and ICT (capacity) assessments were carried out on the national NGOs and the district CSOs in Ngami and Okavango and in Chobe and Ghanzi in FY13. Three IT companies completed Inveneo Certified ICT partner (ICIP) evaluation processes and Inveneo trained them during FY13. The ICT solution for the NCONGO LISN and other sites was designed, procurement initiated and the roll out of the ICT enhancement plan commenced during FY12-FY13. During FY14, equipment was installed at all sites while three out of sites were awaiting internet connection. The ICIPs provided ICT capacity building and support to the CSOs on how to use the equipment during installation. During FY15 Inveneo provided 12 CSOs with ICT equipment. Three of these 12 CSOs (NCONGO LISN in Maun, and BNYC in Ghanzi and BNYC in Chobe) were supplied with different types of computers to establish district CSO Resource Centers where CSOs without computers could go to access computers and internet services. 3.2.10.1 Strengths/Enablers of ICT Platform for Improved Service Delivery and Systems Functioning It was raised by several respondents that having access to ICT has been an enabler for their organization’s efficiency. 3.2.10.2 Challenges/Barriers of ICT Platform for Improved Service Delivery and Systems Functioning It is a challenge that the program did not pursue the planned activities around interfacing with mobile phones through SMS and broadcasting content through social networking, as these could have provided improved service delivery. 3.2.11 Outcomes 3.2.11.1 Organizational change The technical capacity building of volunteers has enabled them to provide better services and as a result they are more trusted by the health system personnel. Hence the health clinic personnel refer cases to the volunteers for, e.g. follow up. This contributes to overall improved service delivery. “The difference is nurses ask us to assist them, where they cannot to execute the duty. They trust we have the capacity” (SSQI District organization). Organizational capacity building has also contributed to improved organizational performance in a number of key areas, including M&E, data collection and reporting, leadership, the development of strategies, resource mobilization and management, and governance. Some of these are highlighted in the following quotes: We have been able to write proposals and come up with projects. We have a strategic direction as an organization. We were able to develop our own organizational development plans. We came up with our 47 own organizational policies. We managed to do gender mainstreaming and such gender balance is being adhered to. (SSQI District organization) The project gave us valuable technical skills in project management and resource management. We managed to raise more than a million pula from different sources due to skills and stability we enjoyed from Maatla. (SSQI District organization) Organizations have also benefited from capacity building through increased visibility, which was said to contribute to attracting more donors, and by adapting tools, reporting to and sharing skills and materials with other non-Maatla sites. Whatever skills and competencies we learnt we are cascading down to all of our centers. Even if Maatla was not there we were able to cascade skills to other sites—we are more organized in non-Maatla sites— they could see we were more organized and they trusted us. (SSQI District organization) 3.2.11.2 Health system change A significant health systems outcome has been the strengthened relationship and enhanced collaboration between CSOs and government at the district level. This has strengthened the multi￾sectoral approach at the district level. An enabling factor has been the clarification of the roles and responsibility of government and CSO members of DMSAC. The facilitative and validating process by NCONGO for the production of the consolidated DMSAC report by its member organization has contributed to district government officials receivinging better understanding and improved respect from the district CSOs. Likewise, the strengthening of the CSO leadership has been a contributing factor. Before, every organization was reporting for themselves. Through Maatla they were able to consolidate their reports, so they present them as a coalition. They make a lot of sense and reduce work load for our office. (SSQI District Government) NASTAD’s development of the district referral systems has also bridged gaps between government and CSOs and helped contribute to improved service delivery and increased adherence to treatment through referrals, tracing of patients and follow-ups. Using capacitated volunteers from the same communities as the beneficiaries has ensured increased coverage of the program into hard-to-reach areas. It has also contributed to improvements in service delivery and access to services by enabling follow-up with patients, by providing home visits and by extending the reach of services through decentralizing services to designated local clinics. “They [Maatla] improve access in hard-to-reach areas and follow-up. They reach out to areas we are not able to go to. Another strength is that they go into families to follow-up” (SSQI DMSAC). Capacity building of CSOs has also contributed to improving the quality of services, with well-trained personnel whose capabilities are trusted and sometimes even preferred to government services. “The changes [in quality of service] are positive because CSOs have information and capacity. They are really trained well and have information. I am sorry to say they are even better at counseling than some of our nurses” (SSQI DMSAC). Increased visibility of CSOs in the district and improved community engagement, in part through mobilizers, health talks, home visits and support groups, have contributed to increased HIV/AIDS￾related knowledge and increased awareness of services available in the district. More people have knowledge about their HIV status, about which services they need, and where they are able to get them. The increased HIV/AIDS-related knowledge has also contributed to achieving “improved understanding of HIV, a decline in default rate, reduced stigma, and more people in retained care” (SSQI District organization). The health system’s change has contributed to a number of health outcomes, including increased numbers of people testing for HIV, increase in people on ARVs and decreased ARV defaulters; higher uptake of TB services and completion of treatment; and more people seeking counseling services and arguably benefiting from these services. 48 3.2.12 Conclusions Although the Maatla project did not develop an integrated model for the districts, some of their components were developed, documented and implemented. Capacity assessments of the DMSAC structures were successfully completed, and both organizational and technical capacity assessment of the CSOs were carried out. Based on these assessments, plans were developed, roles clarified and capacity building implemented. Tools, manuals and standard operating procedures were developed and applied. District Service inventories and referral tools facilitated better referrals and follow up. As a result, government officials and CSOs are collaborating better, which has led to improved health service delivery. The capacitated volunteers ensured closer links between the communities and the health facilities as well extension of services to hard-to-reach areas. In short, a model has been developed and implemented to strengthen district level systems to provide and coordinate HIV/AIDS services in hard-to-reach areas. However, the sustainability of this program is questionable as the existence of some of the coalitions, CSOs and their volunteers has depended on Maatla funding. Despite efforts, a proper sustainability plan has not been implemented and with the collapse of the PPP initiative it is unlikely that the success of the program will be sustained. 3.2.13 Lessons Learned Capacity building addressing the gaps identified in the capacity assessment and followed up with extensive mentoring on site is a useful model to ensure that organizations are truly developed to provide service delivery. Financial, technical and organizational support of organizations, coupled with standard operating procedures, guidelines, manuals and forms/tools, are enablers for efficient service delivery. An established and capacitated coalition such as NCONGO can, with technical support from FHI360 and WUSC ODAs, facilitate the establishment of coalitions in other districts. However, it is crucial to have ODAs and TAs based there. A multi-sectoral approach to addressing HIV and AIDS is useful. Having a capacitated platform such as the DMSAC, where stakeholders from government and CSOs have a clear understanding of roles and responsibilities, is an enabling factor. It is, however, important that government representation on DMSAC consists of senior decision makers. A district service inventory is a useful tool for facilitating referrals. Also developing a referral form together and providing training in its application ensures greater understanding of it. It is crucial to have follow-up as part of the referral system. Likewise it is important that the health facilities provide feedback to the CSOs on the referral. Having an FHI TA M&E officer based in the district is an enabler for ensuring that data collected by CSOs is validated and that the reports are of good quality. Having a coalition such as NCONGO consolidating the CSO report for DMSAC ensures a better quality report and as a consequence improved respect and recognition of the importance of the CSO sector by government. Using capacitated volunteers from the same communities as the patients and supervised by capacitated CSOs is a useful model to extend health services to hard-to-reach areas. This improves the linkage between communities and health facilities. It is important to facilitate the trust building between health facilities and the volunteers. 3.3 Result 3: Strengthened Institution(s) Able to Provide Sustainable Capacity Building to Civil Society 49 Table 7 Key indicators for Maatla Result 3 Indicator Latest Result as per Date of Data Source Data Source Date of Data Source # of successful funding proposals by BONASO and BOCONGO 4 successful (2 pending) for (As of Q2 Year IV). Annual Report FY 15 # of accredited training/capacity building materials developed by BONASO or BOCONGO BONASO: none. BOCONGO: 3 training/capacity building courses were accredited namely: resource mobilisation, gender and HIV mainstreaming and governance Data source: Interview with BOCONGO August 2015 # of member organizations or affiliates trained or mentored by BONASO or BOCONGO Information not available at time of the report. The BOCONGO ED helped at least one organisation as part of the Governance Training. Data source: Interview with BOCONGO August 2015 % increase in non-USG funding for BOCONGO and BONASO Not certain of the % increase in funding. BOCONGO received funds from the EU and from the Global Environment Facility. And they had a project with OSISA. BONASO received funding for 1 year from NACA Data source: Interview with BOCONGO Interview with BONASO August 2015 3.3.1 BONASO as Technical Lead for ASOS The primary achievement of Maatla with regard to BONASO was that it helped to resuscitate the organization. The activities in FY12 were focused on working with the board to hire staff to run the organization, and to rebuild their membership in the districts, An AGM was held in FY13. The Maatla annual reports for FY12 and FY13 indicate that BONASO’s scope of work was aligned to the national framework for CS capacity strengthening, and that BONASO had accepted its mandated role to coordinate AIDS service organizations (ASOs), assume its technical mandate, lead the CSO response to HIV/AIDS, and to develop a comprehensive technical framework for supporting the CSO response to HIV/AIDS. The annual reports show very little progress in getting BONASO to take on its roles besides the development of technical standards and technical assessment tools to assess BONASO. In FY14 FHI360 agreed to second District Coordinators to BONASO to work as Technical Officers, but this did not take place because of the decision to exit the partners focusing on OD from the project at the end of Y4. At the time of the evaluation BONASO had received one year of funding from NACA. 50 BONASO’s chairpersonship of the HIV/AIDS sector within BOCONGO would have been a good way for them to strengthen their leadership in the sector. However, the ED’s successful bid to become chair of BOCONGO meant that BONASO could not fulfil this role. 3.3.1.1 Strength of BONASO as Technical Lead Maatla managed to resuscitate BONASO and get them in a position to receive funding from the GoB (NACA). Membership value The funding of posts for BOCONGO, such as a sector coordinator and a communications officer, ensured it was able to carry out this sub-objective. Maatla also provided funding for BOCONGO to engage its members in consultative processes and information sharing sessions. For example, the three Maatla-funded sectors attended a workshop on civil society. 3.3.1.2 Challenge – BONASO as Technical Lead Although it is stated in the annual report that the BONASO work-plan was informed by the national framework on CS capacity strengthening, neither the interviewee from BONASO nor FHI360 was cognisant of this when they spoke of BONASO’s role in the project. Somehow this strategic intent was not followed through. Even though it is stated that BONASO agreed to take on its technical strengthening role, BONASO consistently refused to do so throughout the project, and this delayed this response. Numerous interview respondents described the many discussions about BONASO’s role in all-partner meetings and other forums took place. Interview data suggests that BONASO’s ability to fulfil its role was hampered by a lack of technical staff. As one respondent said “They had a technical mandate but no technical staff” (KII sub-partner). BONASO suggested that numerous requests for additional human resources were turned down by FHI360, and as such they had no staff to receive the technical assistance that FHI360 intended to offer. Another reason provided was that BONASO saw their role as capacity strengthening of the ASO sector in both OD and technical domains, whereas Maatla had assigned the lead OD capacity-building role to BOCONGO. FHI360’s focus for BONASO was not a high priority for BONASO’s management. BONASO had different priorities even though their participation in the Maatla project was within their broader mandate. A major challenge for BONASO was that they did not have the required technical staff to implement the technical role required by Maatla, and these posts were not funded by Maatla. By the same token, BONASO failed to take strategic advantage of the Maatla project to fulfil its role as outlined in the national framework on CS capacity strengthening. This failure to meet each other half way resulted in a lost opportunity for sustained technical capacity strengthening of ASOs in Botswana. The main challenge for BOCONGO will be to sustain their level of effort and continue to make strides now that they no longer receive Maatla funding. At the time of the evaluation, the key staff members who had been 100 percent funded by Maatla had already left the organisation. Although BOCONGO had won a number of proposals, this was on a much smaller scale than Maatla. 3.3.2 BOCONGO’s Broader Membership Value The aim of this sub-objective was for BOCONGO to draw on the expertise of its members in other sectors to strengthen the HIV response. This is in line with the integrated response to HIV and AIDS. BOCONGO identified the following three sectors to work with under Maatla: Health and HIV and AIDS, Gender and Development, and Children and Youth. Maatla supported BOCONGO to hold numerous consultative and coordination strengthening activities, such as NGO forums, and sector coordination meetings. It also funded the post of sector coordinator in BOCONGO. Maatla also supported BOCONGO to engage with the new NGO council, set up by the NGO Council (set up by MLHA, together with the European Union) in FY12. BOCONGO was also able to mobilize its members to participate in a number of consultations or conferences so that the CSO sector had a presence in these forums, e.g. “BOCONGO mobilized 51 members to participate at a parliamentary forum where SADC national committees were being discussed” (semi-Annual Report, FY15, p. 39). 3.3.3 Organizational Capacity Building The initial intention of this sub-objective was to enhance BOCONGO’s ability to fulfil its nationally mandated role of building civil society capacity in Botswana. The intention was for them to partner with BOCCIM who could mentor them in becoming a sustainable membership-based organization. Due to BOCCIMs withdrawal, this did not take place. The activities reported in the annual reports reflect that the primary achievement under this sub￾objective was BOCONGO developing their training program and their accreditation as a training provider. In FY14 WUSC and BOCONGO developed transition plans for BOCONGO to take over the role played by WUSC to provide OD support to their members, and BOCONGO completed its first round of OCA under WUSC’s mentorship. 3.3.3.1 Challenges: OD BOCCIM’s withdrawal affected its role of helping BOCONGO to become a more sustainable membership organization. The closing-out of the capacity-building aspect of the program meant that BOCONGO was not adequately prepared to take over the OD function from WUSC. The following quote describes how the civil society partners viewed this. This final year was supposed to be the year where we were to hand over to BOCONGO and mentor and coach them on different aspects we were doing on OD, so they could move forward and continue what we have been doing. The plan has been disrupted because the budget was cut. We had a transition plan but when the cuts came through. BOCONGO just said “give us the documents” and when we have staff members on board we will do it. (KII, sub-partner) USAID, however, argued that this transition plan should have happened sooner in the project, and the fact that it did not was a failure on the part of the core sub-partners. The evaluators believe that donors should make decisions that affect the sustainability of interventions based on what has actually occurred and where the organizations are at. Pulling the rug out from underneath the transition process has resulted in large inefficiencies in the spending of public money because it diminished effectiveness. 3.3.4 Negotiating Public-Private Partnerships The cooperative agreement (CA) foresaw the establishment by BOCCIM and BOCONGO of a joint public-private executive committee that would leverage resources from the private sector and help strengthen BOCONGO’s OD role. However, it was agreed that to avoid duplication they would rather participate through the private sector Technical Working Group (TWG), a sub-committee of NACA (Maatla Annual Report FY13). The activities reported under this objective suggest that the main strategy became to lobby for a CSO funding policy (FY12) and to encourage Corporate Social Responsibility (CSR), which would be a good systemic intervention to encourage private sector giving in Botswana. 3.3.5 Advocacy and Communication for External Audiences Here the intention was to strengthen the advocacy and external relations of BOCONGO and BONASO, and to help the organizations to increase their brand and visibility. The work-plans reflect activities aimed at strengthening communication with members, as well as participation of sub￾partners in national events. 3.3.6 Transition Planning from Phase 1 and Phase 2 The intention of this sub-objective was to begin transferring capacity and responsibility for technical and organizational capacity building to BONASO and BOCONGO, respectively. The CO envisages that the two organizations would develop close working relationships through their participation in Maatla management forums and by conducting joint strategic planning. The annual reports show that 52 the main strategy for achieving this was in fact their continued participation in the project management team as co-partners. Other activities reported under this sub-objective are resource mobilization activities (also reported under other sub-objectives). The transition of OD and TA to BOCONGO and BONASO is reported on in year 4 (FY15). It is clear that while BOCONGO did receive some handover from WUSC, it was not sufficient (as discussed previously). However, as discussed in the strengths below, they had made good strides in other areas, which they could continue with, funding permitting. Also, as discussed previously, BONASO was far behind in terms of their TA role, and would not be in a position to continue with the Maatla project once this was over. 3.3.6.1 Strengths: Transition Phase 1 to Phase 2 There are signs that BOCONGO was able to benefit in the long-run from the Maatla project and really participated fully to draw maximum advantage out of the opportunity. An enabler was that Maatla funded key positions, which allowed BOCONGO to do this. Another was their strong and willing leadership. Some indicators of BOCONGO being a strengthened institution are that they are now able to provide accredited training. This they can use to raise funds and to provide training at cost price or sell to generate income. They can also use it to assist members with their own governance issues, now that they have a Governance trainer, and to assist members with other aspects of OD, e.g. strategic planning. This improved member consultations. They are also able to take over the OCAs, although their confidence level in this regard is not very high. The following quote summarizes the positive commitment of BOCONGO to their on-going OD role: This year we are going to train our members (2015). We have got money from the European Union and the Global Environment Fund. With the EU we have selected 15 organizations and we have developed a training calendar and we can use our facilities and provide the training, but the members have to be able to pay for their expenses. (BOCONGO, KII) 3.3.6.2 Challenges The original Request for Applicants (RFA) clearly saw the transition to phase 2 occurring in year 3 of the project. Further, it makes it clear that resource mobilization activities would take place predominantly in phase 2. The gap in resource mobilization in year 2 (FY13), due to BOCCIM’s withdrawal, affected the speed with which this could be implemented, but FHI360 did step in to focus on this in year 3 (FY14) with some success. There is no indication in the planning documents that FHI360 used the two-phased approach as a guide for program implementation. However, in the feedback session it was noted that progress had been made to graduate the qualifying organizations to direct funding from USAID, and graduation criteria had been developed with partners and the strategic advisory committee. The decision not to apply it was based on feedback from USAID that there will be no direct funding opportunities because of the changes related to PEPFAR 3. 3.3.7 Conclusions Overall, the findings from KRA 3 suggest that while BOCONGO made good strides and took full advantage of their participation in Maatla, they are not yet at the point where they are strong enough technically or financially to carry out this role—at least not in the HIV and AIDS sector. Funding that they have received is for other sectors (e.g. climate change). This raises the question about whether BOCONGO was a good partner for a PEPFAR project in this first place, since their mandate is much broader than HIV/AIDS. BONASO’s new leadership, however, failed to make strategic choices to advance their program, and they are not in a position to fulfil the function as a technical lead in the future. They are in a position to partner with NACA as a funded organization, which is positive. 3.3.8 Lessons Learned The main lesson learned was that it is not realistic to take an organization in crisis, such as BONASO, and expect them to build a full program of action based on a funders’ requirements or mandate, and to take leadership in a sector. This needs the full cooperation of the organization’s 53 leadership to buy-in to the role envisaged for them in the project and to manoeuvre strategically within this. Unfortunately, the leadership of BONASO and FHI360 did not manage to build a cooperative relationship in this regard. The other lesson was that the process for building organizational capacity to take over the project should have been more clearly defined, and time bound. The following quote captures this point: We didn’t have milestones established for what should have been done by the end of each year. The implicit assumption was that it would happen naturally, but we should have been more conscious of transitioning roles earlier in the project. We could have been more deliberate in the project about the milestones. (KII sub-partner) 4 Design Evaluation questions • To what extent has the activity been implemented according to activity design? What changes to design would improve the activity? • What actionable, prioritized lessons have been learned for the design of potential follow-on activities, i.e. what should continue and what should change from the current activities in a potential future activity? 4.1 Findings In this section the process and activities for the design of the Maatla activity are reviewed. FHI360 was awarded the implementation of the Maatla activity under RFP dated 20 January 2011, with a closing date of March 2011. This gave FHI360 six weeks to develop the proposal. The document review reveals that after signing the CA in May 2011, FHI360 began a series of consultations and conducted a literature review and mapping exercises. A process of engaging in dialogue with the sub-partners also took place. Besides the original proposal that is contained in the CA, there is no evidence of detailed program design documents or a five-year program strategy. The project seems to have been implemented on the basis of annual work-plans, without an overarching log frame or theory of change (in any meaningful form). The project monitoring plan is dated February 2012, and describes the result areas as contained in the CA, and then lists indicators for each. It is our contention that for a project this size, there was inadequate design and operationalization of the proposal upfront. This means that key steps in the project cycle were skipped. The implications of this were that there was a lot of duplication and repetition of activities under result areas and sub-objectives. This made reporting confusing, and also meant that some areas did not have sufficient activities to ensure their effectiveness. Further, had a proper design process been followed, the misunderstanding that arose in FY12 about the implementation of a service delivery component would not have occurred. FHI360 was clearly under the impression that it was delivering a capacity building program (with organizational development and technical strengthening components), but did not conceptualise activities nor have the correct inputs to deliver HIV and AIDS programming. However, USAID in Botswana clearly expected FHI360 to be able to report on the numbers of people reached by HIV/AIDS-related services. This meant that in year 2, FHI360 had to re-allocate resources that it thought were meant for a capacity-building program to a service delivery program. The main challenge that resulted from this was that FHI360’s implementation team was crafted based on a capacity-building brief, and did not have the right skills upfront to implement services. How this misunderstanding came about is still contended between USAID and FHI360, and USAID are clear that since money was allocated from service line areas (e.g. OVC), it should have 54 been obvious that it was intended for services. Nevertheless, the point is that if a proper program design process was followed, this would not have occurred. The way the result areas were conceived also caused a challenge for the allocation of activities. R1 and R2 often had the same target groups and activities but were split between National and District level implementation. Again, this could have been sorted out through a design process if USAID had agreed to change the results areas as defined in the RFP. Another challenge with the design was that the result areas were not well defined. For result 2, for example, the understanding of what constituted a “model” and who was responsible for designing and implementing the model was not well defined. “From the beginning it has to be very clear what is understood by “model developed”. How do we identify and define this? When do we know that we have reached this?” (KII, sub-partner). Another implication is that the project was not adequately operationalized in a time-bound manner. This meant that the two-phase approach did not come through in the work-plans, and also that milestones and processes for upgrading the implementing organizations to direct USAID support did not materialize. Related to this was that there was no proper process design for the capacity-building component. There was the idea that capacity building would be informed by the OCA, but not for how the capacity building should take place. This resulted in a fairly ad-hoc approach to capacity building. An example of an OD process with training, follow up mentoring and a practical component is provided in Error! Reference source not found.. Another challenge in the design was that there were too many core sub-partners and too many domains (and both the OCA and the TCA in two separate work-plans), which resulted in an over￾supply of capacity building. This was exacerbated by the lack of a structured capacity building process. The NGO sub-partners had very few staff members to receive the capacity building, and were often expected to participate in more than one process at a time, including training and project management structures or mentoring groups. Further, since this was a cascade model and they were then expected to deliver on service delivery targets, they also had to be in the districts, which left little time for them to implement what they had learned. More consideration should have been given to the fact that this project was being implemented in hard-to-reach areas, which required a lot of traveling time. A visit to the district could take up two days’ travel time. These challenges affected the achievement of the targets for reaching beneficiaries, which partners also suggested were set at too high a level. Setting high targets requires organizations to become bloated for the delivery period of the project, and they become unsustainable. The cascade model was itself poorly conceived, because the assumption was that the organizations would be able to conduct the training themselves once they had received it, but there were a number of barriers to this. Firstly, they did not have adequate funding to do this. Secondly, they needed more mentoring to implement what they had learned. The cascade model would have worked better if phase 2 had been implemented with a funding component as per the initial RFP. The other main design challenge was that the GoB was only involved in the project in an advisory capacity. In the interests of sustainability, USAID should consider implementing projects together with or through the government in partnership. In other words, the GoB should have been a sub￾partner. Since they were not, they tended to view donor-funded projects as parallel work, and were not well positioned to incorporate the systems and activities once the donor moved out. This would be in line with point 6 of the Paris Declaration for AID Effectiveness: “6. Country structures are used to implement aid programs rather than parallel structures created by donors” (Paris Declaration for AID Effectiveness, OECD, 2005). This point is discussed in more detail below in the section on relevance. 55 On a positive note, FHI360 proved that it is good at learning from implementation and experience, and has managed to develop good models which they have documented, but only in 2015 after the SIMS identified gaps in program documentation. These models are described in the relevant sections below. Two critical assumptions in the PMP were not adequately managed: • Resources will be available from other sources to sustain the civil society sector. • There will be willingness and interest by all stakeholders to assist BONASO. “There were a lot of assumptions in the proposal about capacity building and service delivery. For example, we assumed that the World Bank would fund organizations if we built their capacity” (KII, sub-partner). Even though the project built on previous projects that had implemented OD, no one was able to identify a particular capacity-building conceptual model that underpinned the design of the project. Such a model or program theory would have helped with designing the capacity-building process and with the design of indicators. 4.2 Conclusions The program proposal incorporated into the cooperative agreement was a good concept document. The Maatla project went from concept to activity planning, but skipped key steps of project design and operational planning. For example, there was no logical framework or overall five-year plan with key milestones. This had a number of important negative ramifications for how the project was implemented, including a capacity-building program that did not have a well-designed process for training and mentoring, and misunderstanding by FHI360 regarding the supply of HIV and AIDS￾related services, which they thought were not part of the project. Another key design challenge was that there were too many core sub-partners and an over-supply of capacity building for the small organizations that were receiving support. This resulted in them being over-stretched and not being available for training or mentoring. It meant that core sub￾partners were competing for sub-partners’ time for training and mentoring. This was one of the reasons why BOCCIM left the project. The following quote sums up the sentiment from most quarters that the project design was too complicated: “I think Maatla has really tried their best; they were disadvantaged by the project mix￾up but who is to blame? They proposed to deliver these things in that manner—they tripped themselves. It was too complicated in design” (KII, USAID). 4.3 Lessons Learned The primary lesson regarding project design is that if the steps in the project cycle are not followed, confusion, duplication and inefficient processes follow. A work-plan and results framework do not constitute good design principles. The design of the capacity-building process encompassed a needs analysis via the use of the OCA tool, followed by the development of a plan of action. It was not supported by a structured process that ensured that classroom training was followed by implementation, supported through mentoring. Creating an enabling environment for resource mobilization requires intensive lobbying at a policy level to get governent to change legislation in favor of tax benefits for corporate and private giving. OD and TA should not have been separate processes, and should have been more closely linked to improved benefits in the health system; there should have been more defined health system outcomes from the start. We should have been asking how organizations make an impact on the health system. This would have given us health system objectives in the first place; it would have been better if it were clear how the CSOs were supporting the health systems.” (KII, sub-partner) 56 5 Relevance Evaluation questions: • To what extent has each activity been implemented according to activity design? Where there have been deviations or shortfalls, what are the causes? • How likely is each activity to achieve its desired end results, as defined in each activity’s results framework? • What actionable, prioritized lessons have been learned for the design of potential follow-on activities, i.e. what should continue and what should change from the current activities in a potential future activity? 5.1 Findings Maatla activities were found to be aligned with government policy regarding HIV and AIDS, TB, OVC and GBV, as defined by the NSF II and to government-supported strategies for civil society capacity building. Relevance to government policy With regard to government policy, Maatla actively engaged with government, including the Ministry of Health (MoH) and NACA, during its program design process and developed activities that are aligned with government protocols, for example, HCT service delivery standards for district-based CSOs were to be reviewed for their alignment with national guidelines (Annual report FY12). This alignment enabled Maatla activities to strengthen and complement the GoB HIV and AIDS response and related programs. Examples of where the Maatla activities were able to support and strengthen government programs include the ability to boost government services in hard-to-reach geographic areas and most at risk communities, and to mobilize communities to access services, to provide OD and capacity building for government network sub-partners. Another example of the strengthened district level TB program is highlighted by the quotes below: Maatla project was addressing what was needed in the district. We had a target of 65 percent of enrolment in TB care and the target was reached due to Maatla volunteers. (KII, Government) Five technical working group (TWG) meetings were held in May 2012, one for each of the five Maatla service delivery areas: HIV prevention, HCT, palliative care, OVC and community TB care. Representation was drawn from MoH, MLG, the national NGOs, other CSOs, the UN, and other key stakeholders. The discussion centered on national guidance and priorities to be used to inform CSO service delivery standards. (Maatla Annual Report, FY12) Hence, the resources provided through the Maatla project were able to strengthen the HIV/AIDS response, or, as one respondent phrased it, “Basically they came from behind and gave a push” (KII, Government). Maatla’s relevance was also strengthened in that it addressed the inter-sectionality of HIV/AIDS with the core issues of gender and TB, which are also priorities of the GoB. “Currently national priority is GBV, as it is a key driver of HIV”(KII, Government). Maatla’s multi-sectoral approach is also an appropriate response to the inter-sectionality of HIV with gender and TB. Respondents to the interviews from the Government of Botswana suggest that because of the strong alignment of USAID activities to the government programs, the government does not provide funding to organizations in areas with USAID presence/funding. This allows government to divert resources to address other developmental needs and also helps government meet its objective of implementing high impact interventions. This is made clear by the following quotes: 57 The amount of funds used in the project by Maatla means that government was able to divert such resources that otherwise would have been used on CSOs capacity building elsewhere to other developmental needs. (KII, Government) One of the key objectives of NACA is to direct resources to areas of greatest need and impact. By virtue of the active inclusion of NACA in the design and implementation of Maatla, the priorities were very much aligned. (KII, Government Respondent) In as much as this is an indicator of the relevance of the program to the GoB, there is a threat in working in parallel. If the GoB is not adequately informed on changes in the USAID (or PEPFAR) objectives or funding areas, they will not be able to effectively step in and take over should USAID funding change direction or come to an end Unless government is able to effectively do this, the organizations and services previously funded by USAID run the risk of being undermined and collapsing. This is a question of sustainability. The critical key assumption underpinning the results framework, that CS would be funded by other parties, did not hold true, and since it is clearly acknowledged in Maatla annual reports that the donor funding landscape had changed in Botswana (already mentioned in FY12 report), it would have been necessary to work more closely with the government to ensure continuity of Maatla services. It is a problem with a coordinating agency seeing donor programs as parallel programs, not with the donor. We were very involved in PEPFAR even at the contracting stage. We have our own money, we are not totally donor dependant. We are busy with our programs and let them run theirs. (KII, Government respondent) Relevance to needs of Civil Society strengthening Maatla’s interventions were found to be relevant to the needs of this target group. In order to implement the NSF II successfully, government relies heavily on a strong civil society. The important role of CSOs is highlighted by the quotes below: HIV/AIDS is a top priority in this country and for the ministry and we embrace CSOs as our key partners. Some of them don’t know how to manage their organizations so when Maatla came in to say they were strengthening them, we thought it was a good idea. (KII, Government) The idea of focusing on strengthening the network organizations was also deemed relevant: “This was definitely needed—to have strong coordinating bodies—especially to be able to leverage funds for their members, monitor implementation, show results and direct the donor funding” (KII, sub￾partner). The degree to which the OCA and the TCA required professionalization of CSOs in Botswana was also deemed relevant by most respondents, although some felt that they were too far-reaching, or that the domains could have been fewer. 5.2 Conclusions The Maatla project was relevant to both the GoB’s National Strategic Framework for HIV/AIDS II (NSFII), and it’s strategy for strengthening of CSO AIDS Service Organizations. While the project clearly boosted the efforts of the Botswana Government to address HIV and AIDS, the challenge is that USAID and the GoB tend to work in parallel. This threatens the sustainability of the intervention because government systems may not be able to absorb the outputs of the project once it is over. 5.3 Lessons Learned Even though relevance and alignment of the program to the GoB is clearly an enabler of the program, as it allows strengthening of Government programs, if continuity is not ensured when USAID/PEPFAR change direction, the benefits are likely to be lost. Hence, alignment and collaboration are not sufficient conditions for success. To ensure sustainability, the government needs to be brought more into the center of programs as a sub-partner. 58 The following quote highlights this point: “They are 100 percent aligned. This helps us to avoid duplication of services, as if I offer food why should they offer food if they can offer something different” (KII, Government). The converse is also true, for example, if USAID offers food where government does not, and government is not ready to take over this service provision when the donor leaves, when USAID service does end this could cause hunger in an area. Another example is that if USAID programs support access or adherence to treatment for HIV or TB, when USAID leaves, their treatment regime may be jeopardized. 6 Sustainability Evaluation questions: • How likely is each activity to achieve its desired end results, as defined in each activity’s results framework? • What actionable, prioritized lessons have been learned for the design of potential follow-on activities, i.e. what should continue and what should change from the current activities in a potential future activity? Sustainability is about the extent to which the benefits will be continued once the project activity is complete. The document review shows that sustainability of interventions were built into Maatla from the beginning, and a number of activities were designed to strengthen sustainability early on in the project. For Maatla, result 3 was in itself a sustainability objective. The focus here was for the national NGO member organizations (BONASO and BOCONGO) to take over the technical capacity strengthening and the organizational capacity assessment components, respectively. This is discussed in detail under the section Result 3. The second aspect of sustainability is that of the organizations themselves—their own funding situations—and concerned resource mobilization; and the third, is that the systems and models that have been developed should be able to continue to be used into the future. Some progress was made in the first quarter of 2012 when WUSC organized a visibility and communication training for BOCONGO and BONASO. But as discussed in the section on managing the organization as an enterprise, these efforts only started taking off again in FY14 after the withdrawal of BOCCIM from the project. BOCONGO was helped to develop their business services, particularly as a provider of accredited training. It is reported in the FY14 Annual Report that BOCONGO managed to develop three modules that were submitted to the Botswana Qualifications Authority and those these modules were accredited. The courses were resource mobilisation, gender and HIV mainstreaming and governance. Respondents from BOCONGO and from other core and implementing organizations were concerned that they were not adequately prepared to do this because of the sudden phase out of the OD component and fast-tracked handover by WUSC. This handover was supposed to happen in 2015 but because of the close-out phase it didn’t really work well. At this point it is a role that we would like to play but we are going to do strategic planning, as there have been some changes in the context and we want to see how Maatla will be adjusted. (Sub-partner) Another change in the landscape is the creation of an NGO council in Botswana that may supersede the role of BOCONGO. BOCONGO will need to work with the council and become recognized as their capacity building partners, and the partner for the NGO code of conduct. Interviewees from USAID mentioned that BOCONGO had to be cut short due to changes in PEPFAR priorities. They mentioned that if WUSC had transferred capacity to BOCONGO earlier that would have prevented the failure of the program. 59 The resource mobilization and income generation efforts that were conducted under the objective “managing the organization as an enterprise” were also intended to help with sustainability. An assumption was made that resources would be available from other sources to sustain the civil society sector (Maatla PMP, no date) In the last quarter of 2011, all core sub-partners and sub￾partners participated in an internal project meeting to discuss the vision of Maatla beyond the project. The Strategic Advisory Committee (SAC) also discussed a sustainability strategic plan in 2011. Several activities were planned and achieved in FY12 to strengthen the sustainability of Maatla. Some positive results were that during FY12 BOCAIP was supported to develop a property portfolio and business plan; and BONEPWA+ was assisted to identify opportunities for generating income. In FY14, as part of supporting national NGOs and networks to position themselves for government and others to procure their services, FHI360 invited SHOPS (Supporting Health Outcomes through the Private Sector) to make a presentation to members of RM and the Sustainability Committee on how organizations can develop their value propositioning statements and use them to market services to the private sector. Data collected from interviewees gives conflicting views on the sustainability of Maatla as some believe that the project will be sustainable as the implementing organizations were taught to source funding, but others feel that there was no sustainability plan to address how the gap Maatla will leave will be filled. Responses from BOCONGO and other stakeholders Interviewees from implementing organizations mentioned that without donor funding CSOs will struggle to continue with their services. If they do manage to continue, the quality of service given to communities may deteriorate. Interview respondents from USAID mentioned that if CBOs failed to mobilize resources and write reports it would be a failure to themselves as they were trained on these aspects in the implementation phase of the project, but they did not do much until last year when they realized that the project was soon to be phased out. They have all been trained to write proposals but they relax during implementation because they are getting funding and then at the end all the people who have been trained are out because they are the first ones to find other jobs. (KII Donor respondent) But they should have diversified their funding by now. It was only last year when I visited them I saw that they didn’t have proposals. (KII Donor respondent) However, at the time of the evaluation, as noted in section 3.1.3, all four sub-partners had received funding at the time of the evaluation, including from the European Union and the GoB. BOCONGO had also secured a project in Environmental Justice from the Open Society for Southern Africa (OSISA). Hence, it is a success of the Maatla project that all the sub-partners were going to be able to continue with their work into the future, even if it is not directly HIV/AIDS-related, as in the case of BOCONGO. The alignment of Maatla to the programs and policies of the GoB has been an important sustainability factor as it could mean a relatively seamless transition from Maatla to government funding for the CSO. An example of this is provided by an interviewee from the Ministry of Gender who mentioned that the activities will continue as they are integrating gender mainstreaming into their daily work and they are also mobilizing communities to develop action plans to respond to GBV on their own. However, the early close-out of the OD component has meant that there is a gap in funding for the capacity-building component. Further, Government has only funded areas where Maatla is not operating, so communication and coordination with government would be necessary in order for a smooth transition to government funding in these areas. The extent to which this has taken place in preparation for the next COP under PEPFAR 3 is not clear, as USAID will no longer be working in the Maatla districts. 60 The GoB has shown a commitment to sustainability by already agreeing to fund BONASO, BOCAIP and BONEPWA+, however, to a smaller degree than Maatla did. “In fact, as of now, the government has mobilized resources under global funds and also under government to continue engagement with CSOs” (KII Government respondent). WUSC mentioned that the governance component should be an enabler of sustainability as one of the main challenges for NGOs in Botswana was to secure funding. This is one of the reasons BONASO fell, and others will struggle to be sustainable when the funding from USAID stops. They tried to help the NGOs raise funds but there was not enough time to achieve this. The systems and procedures that have been developed by Maatla are a positive factor for sustainability. These include a number of toolkits and guidelines that were produced for use in the CSO sector in Botswana such as a financial management toolkit for small CBOs. Respondents from interviews felt that even if the organizations do have to scale back their services and scope, these organizational and technical systems will remain, and they will continue to use OCA and SIMS. OCA and SIMS has helped to formalise the CSO system, and SIMS will ensure that they continue to give quality services to the communities even after Maatla. The process of institutionalizing the tools has been very important for sustainably, e.g. using the OCAT, doing improvement plans, making sure this enters into the strategic and annual work-plans, and this helps to attract funding because we are more systematic and sustainable. (KII Implementing organization) When comparing primary and secondary data, it is evident that there were plans that were put in place to ensure sustainability during the implementation stage of the project but that these were not well followed through until FY14, which is possibly a little late considering the early close-out of the OD component. The withdrawal of BOCCIM, the lack of PPPs, and the early close-out of the OD component are the biggest barriers to the sustainability of the activity and the ability of CBOs to mobilize funding to ensure sustainability. 6.1 Lessons Learned Alignment to government objectives is a good practice but to really ensure continuity and lasting benefits in the government health system it is important to try to bring government into the project as a sub-partner from the start. USAID should identify how to achieve this within their regulations, possibly through their bi-lateral relationships with the GoB. 7 Project management Evaluation questions: • To what extent has USAID fulfilled its role in the activities by providing adequate funding, responsive managerial oversight and technical support, and assistance in solving problems in implementation as they arise? • What actionable, prioritized lessons have been learned for the design of potential follow-on activities, i.e. what should continue and what should change from the current activities in a potential future activity? 7.1 The Role of the Core Sub-Partners and USAID FHI360 was the overall project manager. The personnel were hired and the project management structures and systems were established in FY12. During this year offices were established and equipment secured. Sub-agreements were signed with all sub-partners by May 2012 following financial pre-award assessments, and were renewed annually based on new annual work-plans. Cooperation agreements between WUSC and these organizations were in place. 61 FHI had to manage relationships between the core sub-partners and sub-partners, USAID and the GoB. FHI360 was responsible for all financial and contracting arrangements, project management, monitoring, evaluation and reporting. FHI360 submitted quarterly and annual reports as per the schedule. USAID were satisfied with the role that FHI360 played. They found them to be a responsive learning organization that accommodated requests and kept them informed. The following management and coordinating structures were put in place: 1) Strategic advisory committee (SAC): The SAC reviewed and approved work-plans, and was supposed to meet regularly (every six months from Y2). 2) Senior Management Team meetings (SMT): Directors of core sub￾partners and sub-partners attended and it met quarterly. 3) Project Implementation Committee (PMT): This was attended by project implementation staff. It’s aim was to review progress and make recommendations to the SMT. The PMT was supposed to meet monthly, but it met seven times in FY13. The PIC was dissolved during quarter 4 FY14, as it was concentrating on what was already contained in the reports, and replaced with two focused committees (the Sustainability/Resource Mobilization Committee and the Technical Committee). 4) Project partner review meetings: FHI360 conducted these with individual partners on a periodic basis. A key task of FHI360 was performance management of the sub-partners. The project review meetings were useful as a performance management tool, and for sharing challenges and solutions between FHI360 and the sub-partner. FHI360 contends that the performance management of the sub-partners was made difficult by the five-year funding contract, which was not performance related. The core sub-partners indicated that they had sufficient resources to implement the project, but this was not true for the sub-partners who all felt that they could have had more people to implement the large scope of work. One of the main challenges that FHI360 faced was with the allocation of resources to partners. Questions about whether additional staff should be hired directly into the sub-partner or provided through additional TAs at FHI360 have been asked by respondents, but there is not one clear answer. This became more of a concern when BOCCIM withdrew and their capacity to provide resource mobilization support was lost. The interviews also show conflicting views on this. As discussed in the section on design, the main challenge for the design and implementation of the project was that its structure was too complicated. This posed challenges for coordination. Role of the core sub-partners For the most part the core sub-partners played their roles well, except for BOCCIM, who pulled out. WUSC was able to fulfil its role of OD advisor, although they recognized that they did not have sufficient experience in implementing a long-term capacity-building program and hadn’t developed a process model for this. The training they provided was well-received, especially the governance training, which was longer-term training with a very structured approach. There was also a mixed reaction to the appropriateness of some of the ODAs in terms of skill and level of experience. NASTAD focused on district level coordination and collaboration between the government and civil society and while they did this well, they concentrating on their own scope of work and could have contributed more to developing a district model. Instead, different pieces of the model were developed by different partners. NASTAD, for example, did not want to develop the referral system, which FHI360 eventually did. NCONGO was both a core sub-partner and a recipient of capacity building, which was confusing, but played a very important role in the success of the project in the districts. NCONGO played an important role at district level, and fulfilled two roles, namely the building capacity of their members and helping CSOs in other districts to form coalitions like their own. Respondents are generally positive about the role of NCONGO. FHI360 found BONASO a difficult partner with which to implement the technical strengthening aspect of the project, and so this did not progress well. The project had to rebuild BONASO, which had collapsed, while getting them to implement a program of work they did not necessarily buy into. This is discussed in detail in the section on technical capacity strengthening under effectiveness. 62 FHI360 responded well to the changes in the program that came from the misunderstanding about the implementation of the service delivery component, and to the inclusion of a stronger gender component at the request of the GoB and USAID in FY14, and to the changing requirements of USAID with regard to the new direction for PEPFAR 3. However, they were possibly too accommodating of USAID demands to cut back on the OD component in FY15, as this meant that many of the OD outcomes were not coming to fruition. We are not allowed to support anything that is not in line with PEPFAR 3 from October, and we had to cut all the OD aspects, and release staff because they were not focusing on the new PEPFAR 3 objectives. It is a difficult decision for sub-partners; it means some of the results won’t come out as they would have because these elements were just coming to fruition, so we won’t see the results. It is very frustrating when things change, because you are just building momentum. (KII sub-partner) This begs the question about the role of the FHI360 under a cooperative agreement, and whether they should have insisted on seeing these aspects of the project through. This leads to a discussion of the role of USAID. Role of USAID • To what extent has USAID fulfilled its role in the activities by providing adequate funding, responsive managerial oversight and technical support and assistance in solving problems in implementation as they arise? USAID played a good oversight role and picked up a number of challenges with implementation, such as the lack of implementation of the service delivery component during FY2012, and that Maatla had an insufficient youth focus. The insistence on including a service delivery component did strengthen the program as, in reality, organizations such as BOCAIP and BONEPWA+ may not have had sufficient resources to implement programs, and as such the core sub-partners would have been providing capacity building in a vacuum. However, this did pose a challenge for FHI360, which had allocated resources and hired a team to implement an OD program. The wisdom of USAIDs directive to close-out the OD component of the program has also been questioned in this evaluation. It may have been an expedient decision with regard to how USAID Botswana responds to its corporate needs, but flew in the face of development principles of sustainability and efficiency. Another challenge has been the extent to which USAID brings the GoB into the center of its programming rather than offering parallel programs. This is also raised elsewhere in the report in more detail. Both of these issues are discussed in detail in the relevance section. Apparently the contracts with FHI360 and the sub-partners prevented direct communication with USAID. Sub-partners took this to mean that they could not communicate with USAID on other matters either, regarding, for example, funding for other work or invitations to the US Ambassador to speak at a function. This hampered their ability to build stronger relationships with USAID, who is obviously a key player in the development arena in Botswana. The program seemed to have been reasonably funded, although how this was costed is not clear to the evaluators as we were not privy to the bidding process. The main challenge was that implementing organizations still seem to have been short-staffed, and the over-reliance on volunteers to deliver program objectives is inefficient in the long-run due to their high turn-over rate. 7.1.1 Strengths The project management structure and consultative meetings and processes were a strength of the project, but as discussed below these also had challenges. The SMT helped with information sharing between partners. The feedback was mostly positive “I liked the meetings we had in Maatla—senior management were there, FHI and WUSC, they were 63 showing our achievements and challenges, and we discussed next steps. They worked well because there was cross-fertilization and we shared lessons learnt” (KII, sub-partner). In some cases FHI360 showed good adaptive management and flexibility. They were able to respond well to challenges such as the findings presented by the SIM, as they can draw on their international resources and other offices for assistance. They were also able to identify and fill gaps even if their response time was perhaps a bit slow. For example, a year went by with BOCCIM failing to perform before they stepped in with resource mobilization activities. This could also have been because of the need to negotiate resource allocations and placement of human resources with USAID. The establishment of program guidelines, models and SOPS has been a strength, and was helped by the SIMS process, which identified further gaps in this area. The development of SOPs and guidelines was a process that began in FY13 and was ongoing through the life of the project. Partners were mostly satisfied with the respectful relationships that were developed in the project, and appreciated the transparent way that FHI360 managed the project. 7.1.2 Challenges The SMT was supposed to provide a strong communication channel between Maatla and the GoB (mainly MoH and NACA), but in year 2 and year 3 it did not meet as regularly as expected, but MoH indicate that they believed they were sufficiently informed about the Maatla project. The SMT was revived again in FY15. The SMT meetings were criticized for not being strategic enough in nature and focusing too much on activity planning and reporting. “The SMT meetings could have helped to coordinate and weave the different elements of the project together, but they did not seem to focus on these issues. They were not very strategic” (KII sub-partner). This sentiment was echoed by a number of respondents. The primary challenge in the project was coordinating the activities of the many sub-partners. Even though FHI360 put coordination systems in place, there were still issues of duplication and communication. A gap related to the work-plans was that there was no annual share for all sub-partners, which could have helped with coordination. “If this had happened we could have been around the table to discuss what the plans were for the coming year, and we could have shared travel to the districts and coordinated our visits” (KII, sub-partner). The process of negotiating sub-agreements with sub-partners delayed implementation, and took up much of FHI360’s time. This was reported in FY13 and FY14. According to one respondent, FHI360 should have had more coordination capacity. They had one person doing coordination and it was not sufficient. The partner review meetings for the first two years only took place between FHI360 and the sub￾partner, and WUSC argued that they should have been included, as they were also managing the performance of these sub-partners on the OD component. This was changed in year 3, which was positive. All-partner meetings did not work as effectively as intended. Firstly, sub-committees were established, which slowed things down. Despite the many coordinating structures and all-partner meetings there were still challenges with coordinating the work-plans of the various core sub-partners. This resulted in some overlapping activities competing for the participation of the sub-partners. The following quote captures the sentiment expressed by many respondents. The meetings, workshops, etc. were useful but took a lot of our time. We spent too much time in meetings—there were too many partners in the project—it needed too much coordination. The project could have been implemented with fewer partners, some roles could have been merged. Having both FHI360 and WUSC was demanding of our time. All three organizations (WUSC, FHI and BOCCIM) had a 64 resource mobilization person and they all needed our time—there was too much duplication—it needed more coordination. (KII sub-partner) The responses also indicate that sometimes ODAs came with competing advice, which left organizations confused. There were also occasions where two sub-partners visited the districts at the same time to offer similar workshops. There was some criticism about the way that changes in the project direction were communicated to partners: “There were major changes to the project that we were not consulted on, it was a directive “we are cutting your budget”, “move out of these districts.” etc.”(KII, sub-partner). The project was not costed in terms of time and personnel at an operational level, and as a result unrealistic demands were made on a few people—the same people had to attend training, cascade training and implement organizational improvements. Had a costing and time allocation been done, it would have been possible to see how much in-office worktime would be left for people to implement project activities. The alleged prohibition on communication between partners and USAID was an obstacle to the sub￾partners seeking out other kinds of support from USAID that could have helped with resource mobilization. A challenge for FHI360 was changes to PEPFAR indicator definitions, which meant that their M&E tools had to be changed accordingly. A number of respondents complained that this was inconvenient because people had to be retrained in the use of new tools after they had become accustomed to existing ones. 7.2 Monitoring and Evaluation Monitoring and evaluation was one of the built in components of the Maatla project. CSOs need to have their own data collection system as they are expected to contribute 25 percent of the HIV/AIDS national response. Several activities had to be undertaken to successfully build the M&E system into the project, as well as to instil and encourage an M&E culture in Botswana organizations. These M&E activities were undertaken at different stages of Maatla projects, starting from the implementation phase. Establishing M&E systems and tools The focus of the first phase of the project was the establishment of M&E systems and tools, and recruitment of M&E TAs for all aspects. There was an attempt by Maatla to integrate the first two M&E objectives, so that the M&E tools were aligned to government. In FY11 and FY12, FHI360 made efforts to align the Maatla M&E system with those of NACA and MoH. NACA shared the national operational plan, including new indicators to accompany the NSF 2, with FHI360. By the end of quarter 3, Maatla staff had programmed new NACA NSF 2 indicators into their electronic system, which was called the District Health Information System (DHIS), and organizational units representing all CSOs were mapped in the district. The PMP indicators and targets were revised and submitted to USAID for approval in October of 2011, and the final PMP was approved in March 2012. Several tools were developed and piloted, including an Organization Capacity Assessment tool (OCA), District Government Coordination tool, Technical Capacity Assessment tool (TCA), and a data quality assessment tool. For a full list of M&E tools for Maatla please see Appendix X. An evidence-based approach was used for programming and the capacity-building strategies for the sub-partners and the district coordination systems were developed once baseline data had been collected using the Districts Government CSO Coordination tool, and the OCA—OCA-Lite and TCA. There was a lot of emphasis in Maatla on data quality, and helping organizations to improve their data quality systems can be counted as one of the successes of the project. Maatla benefited from the piloting of “a new global data quality assurance tool developed by FHI360”, called the Rapid Data 65 Verification and Improvement tool, which was piloted in two different sites, and the assessment yielded some valuable information for the project and for improving the tool itself. On-going technical assistance, training and the provision of SOPs guided both CSOs and their counterparts in the District AIDS Coordinator’s office in terms of CSO reporting in the districts (Maatla Annual Progress Report, September 2012). This kind of support included compiling of quality and timely monthly reports, and reviewing and providing feedback on the indicator definitions. A number of changes took place in PEPFAR reporting and, in FY14, FHI360 also revised the M&E toolkit to realign it to the MER (Monitoring, Evaluation and Reporting) for PEPFAR indicators. The revised tools, data collection guidance and reporting timelines were disseminated to all sub-partners. There were also several monitoring and support activities that took place in all four districts. These activities included reviewing the availability of data on sites, conducting data quality checks and meeting with stakeholders. 7.2.1 Strengths Findings from interviews indicate three main points as being the strengths of the Maatla M&E component. These include data quality assessment, strengthening CSO reporting at a district level— including improvement understanding of the importance of M&E—and thirdly, the use of data for evidence-based planning. Under data quality assessment, respondents from USAID mentioned that they gave training to FHI360 on routine data and quality assessments, understanding the PEPFAR indicators, reporting and reporting on time. The most important piece was for FHI360 to have the capacity to manage the M&E system. Respondents from FHI360 verified this by saying that they worked with USAID to develop quality verification of data and they trained partners in how to collect data. The DHIS is an impressive electronic query-based system that allows Maatla to rapidly analyze and report on key data. For the first two years of this project much of the focus of the M&E team was on building this system. Respondents from implementing organizations mentioned that quarterly progress reporting and quarterly supervisory visits to volunteers were done. The main purpose of these visits was to do data quality verification and identify areas for improvement. In districts they managed to conduct data quality assessments and they worked together to do M&E activities on a monthly and quarterly basis. Measuring tools for OCAs were put in place to periodically see if they improved on domains. The second point is strengthening CSO reporting at a district level, including improved understanding of the importance of M&E. With the support of the FHI360, the M&E Officer partnering NGOs mentioned that they managed to develop M&E plans, and processes were put in place for data collection and reporting. Capacity building received by the CSOs helped them to develop their M&E plan and as a result they are more focused on indicators, and they know how the information that they collect fits into the overall projects targets and the national targets. Respondents from implementing organizations mentioned that the referral tool used at district level to monitor and check process worked well but they still need to continue collaborating with government and each other as they roll out to other areas. Lastly, implementing organizations mentioned that they have learnt how important M&E is as a motivating force and how having data and the evidence on a regular basis and sharing with their sub￾partners helped them to monitor and highlight their progress. 7.2.2 Challenges The attempt to align to the national indicators was not successful. Apparently NACA were not yet ready with their indicators and systems when FHI360 was designing the PMP for Maatla. For this 66 reason, current reporting on indicators, such as referrals, that are not yet in the governmet M&E system will probably not continue. Even though the capacity of CSOs to collect monitoring data was enhanced, the challenge lay in analysing the data. For the most part, FHI360 still collated the data for reporting to USAID. This was mainly because the implementing organizations did not have M&E personnel or software. FHI360 only began to feed this data back to the organizations later in the project, which organizations found really beneficial, but would have preferred this earlier on as it helped them to understand how data can be used for planning. The district level data analysis meetings and presentations were apparently quite rich and highly appreciated (according to FHI360 feedback), but the challenge was that these only started in year 4. A lot of time was spent with the project database in the initial two to three years which mitigated against a stronger focus on data analysis and use earlier in the project. Another barrier to using M&E for learning was that the emphasis of the Maatla M&E system was very much on reporting on targets, and had insufficient focus on qualitative reflection, or on a lessons learned component to the M&E system. They focused on reporting, understanding indicators, doing DQAs, but there has not been any learning component. Before with EAD, the learning component was big and helped them to use info to make decisions and target the right groups. You could see that AED were actually analyzing the data and using it and feeding it back to the communities. You didn’t see this in FHI.” (KII Donor Organization) BONASO was expected to help collect reports on behalf of the ASOs and collate reporting to NACA, but BONASO didn’t do this because they had no staff to do so. BONEPWA+ mentioned that even though M&E was beneficial to them as an organization and to other NGOs, a disadvantage is that it was only output focused rather than outcome based. It was not used it to see any behavior change as a result of the project. An interviewee from USAID mentioned that SIMS was a good tool because it helps identify gaps in the systematic documentation of the program, but it is an external monitoring tool to check compliance and is was not well suited to the nature of a cooperative agreement. There is no standard operating procedure for the OCA. It was mentioned by FHI360 in the feedback workshop that they have kept close tabs on changes that have been made over the years to ensure that the results of the Y1 and Y3 OCA are comparable. However, this was not provided to the evaluators. Further, if BOCONGO are expected to continue with the OCA and do the scoring, a more detailed guideline of how to tally the scores would be necessary. The evaluation team spent at least one day cleaning up the excel spreadsheets and correcting tally errors. Further, it is not clear how not applicable N/A should be scored because there is no standard operation procedure or guidelines for the OCA. This needs to be clarified for future use of the tool. 7.3 Conclusions Most interviewees agree that the M&E component was built into Maatla project, CSOs were assisted to develop M&E plans, and from those M&E plans they have learnt how to keep focused on their indicators, to document everything that they do and be able to share it with others as part of evidence of work that they do. The negative part is that there was not enough capacity building to ensure that CSOs document and analyse data themselves. Most interviewees from CSO felt that not having an M&E officer in their own organizations deprived them of an opportunity to use data to learn and improve their programs. Despite the possible mismatch between SIMS and a cooperative agreement, the exercise was very beneficial for the project and ensured that FHI360 adequately documented the processes and models that it had developed during the project. Perhaps the requirements of SIM could be adapted to a PMP in the future, which would make it more in line with cooperative agreements. 67 7.4 Lessons Learned Do not create projects with too many core sub-partners who are trying to work with sub-partners. Perhaps one core sub-partner and the GoB is sufficient. The importance of using M&E data for adaptive management, and not just reporting on targets, has been reiterated by the findings. This takes place on a number of levels. Firstly, the importance of closing the feedback loop and sharing the analyzed data with the sub-partners helps to inform programs and policies. Secondly, quarterly reporting allows managers to pick up challenges that can be addressed timeously. Thirdly, it is important to have a reflective component to monitoring systems so that lessons for improvement can be learned, and lastly, to report on outcomes as well as to not focus on outputs only, which deprives implementers of an opportunity to see a behavior change and to see whether the program is achieving what it is meant to achieve. Another key lesson has been that it is possible to get CSOs to collect good quality data on an on￾going basis in a systematic way. It requires good simple systems, including quality control and data validation systems, mentoring and much repetition over extended periods of time. Providing feedback to these organizations on the data that they collect is also important as it helps them understand why they collect data, and how it can be used. This provides meaning for the M&E so that it is not just an exercise for the donor, and hence increases their commitment to doing it well. The results from capacity-building projects are often process oriented and hard to measure. The OCA could be expanded to include more questions about the implementation of systems, and about how learning is taking place in the organization. 8 Recommendations Evaluation question 2 • What modifications, if any, to the strategic approach and theory of change have potential to substantially increase program effectiveness? 8.1 Result 1 Recommendations about improving the design of capacity building projects follow under Result 1. FHI360 1. When engaging in a capacity building program make sure that it follows a well-designed structured process, with milestones that are time bound, and that progress on the OD components is incentivised. 1.1 A technical guideline for a structured process for capacity building of CSOs should be developed out of this project by FHI360 to inform the future capacity building projects. The structure of the Governance course could be adopted as a model as it ensures that learning is transferred by the learner into the organisation as a practical project and cascaded to other organisation through case studies. A mentor or ODA can support both internal practical projects and the case study projects. 1.2 The project description document clearly identifies the need to have an integrated capacity building framework that includes a number of different methods such as training (short and long term), mentoring, technical assistance, provision of guidelines and SOPs. What is missing is a clearly defined process for how these components should fit together. The combination of the elements is likely to depend on the needs of the organization and hence a fluid approach as was adopted by Maatla is understandable. However, certain basic processes needed to be in place to ensure that what skills were learned in formal training were transferred into the workplace. A good capacity￾building process should have structure to ensure that participants have a chance to implement what is learned in the classroom and do practical projects at work, supported by mentors for individualized learning. Participants should be assessed on completion of the practical project to 68 ensure that they have absorbed the skills and transferred the learning to strengthen the organizational systems. Thus indicators for transfer and use of skills should be developed. The governance training followed a similar model and this worked very well. The training in M&E, however, left the organizations with a draft M&E plan and no follow-up to help them operationalize this in their organizations. Perhaps the intention of the Maatla designers was to have a more structured process, but with the over-supply issue this was not manageable. 2. It is recommended that in future FHI360 make the extension of sub-agreements performance based. 3. The OCA tool is not nuanced enough and organisations can score well for having systems in place even if they are not implementing them well. The OCA tool needs to have more questions about the organizations’ ability to implement the systems that they have put in place, and about their ability to learn and improve on these systems. 8.2 Result 2 For FHI360 4. It is recommended that extensive consultation and facilitation take place to ensure local buy-in into new programming direction. This will ensure that the needed working relations exist between the various sub-partners such as BONASO, BOCCIM, NCONGO. 5. Although sustainability was part of the concept and addressed in the beginning of the program, efforts were not maintained, particularly with the collapse of the BOCCIM cooperation. It is recommended that sustainability is built into the program and is constantly addressed throughout the program cycle. 6. Having insufficient personnel kept being a recurrent issue raised by respondents. It is recommended that a sufficient number of supervisors be recruited in the service delivery organizations in order to supervise the volunteers in hard-to-reach areas. 7. It is important to provide incentives for the volunteers both in monetary terms and through recognition. Particularly an organization like BONEPWA+, who provides services to people living with HIV by volunteers who are themselves infected or affected by HIV, should invest in income￾generating activities for their volunteers. It is also recommended that volunteers help social workers identify OVC as they are in the community. 8. Due to the link between HIV and GBV it is recommended that continued awareness creation should take place in the communities regarding gender and GBV. This should be targeted specifically at men and involve men as mobilizers. It is recommended that community dialogues should be facilitated. 9. It is recommended to continue supporting refresher courses in the Maatla target districts to sustain the knowledge gained. Particularly more M&E capacity building is needed. For USAID 10. The program should be rolled out into more hard-to-reach areas to ensure extended coverage of the program. This is particularly the case with the mobile HIV testing and counseling services. Also, strategies for reaching men should be developed to ensure that they go for testing. 11. As the CSOs are assisting government in providing services in hard-to-reach areas, it is recommended that government fund the CSOs for this. CSOs should adopt a business-like costing approach, not just charge government for transport and food, but should cost personnel and overheads to work towards sustainability. 8.3 Result 3 For sub-partners 69 12. BOCONGO must enter into an agreement with the NGO council regarding its role and functions relative to the council—in particular it’s capacity building function for civil society. 13. BONASO should fulfil the role provided for it in the civil society capacity building strategy, and enhance its monitoring and evaluation function significantly. 8.4 Design General 14. Every project should have proper planning documents behind it, including a log frame or some other detailed results framework and an operational plan that covers the full length of the project lifespan. While annual planning is necessary, it needs to be done within the framework of an overall project plan covering the lifecycle of the project, with key milestones and critical pathways for a detailed theory of change. 14.1 The results framework should have outputs worded so that it is clear what the project is expected to deliver. The annual workplans have activities linked to results areas, but these activities are too narrow and the results areas are too broad. It seems the level of outputs is missing from the design documents. 14.2 A better line of sight (or pathway of change) to health systems outcomes would have made the project design more coherent. A more detailed theory of change could have assisted with this. 14.3 The theory of change should also make explicit what evidence is being used to inform program design. This would have been extremely useful. For example, why is the capacity building model that has been selected the right one to bring about the desired changes? 13.4 Much of the justification is contained in the project document, but it is embedded and dififcult to extract. A clearly defined and described theory of change would help the implementors and evaluators to understand why things were done in certain ways, and to check whether the theory holds true in practice or needs to be adjusted. 15 More realistic targets should be set or a different way of calculating targets needs to be implemented, based on the delivery capacity of the organizations that are implementing the project and the funding provided by USAID – organisations are over-stretched by the targets required by PEPFAR which also pushes services up to levels that are not sustainable. 8.5 Relevance & Sustainability for USAID 16. USAID must ensure that it considers the sustainability of interventions that have already been funded before closing-out or changing direction. The requirement to be able to re-pivot all USAID programming on six months’ notice is not developmental, and results in inefficiencies, gaps in service delivery and collapsed organizations. The findings show that government tends to work in parallel with USAID and hence better mechanisms need to be found to ensure sustainability and continuity of the benefits of USAID programs,. The key assumption in the USAID Botswana results framework that Government will be ready to absorb systems and structures built up through USAID projects needs to be central to the design of the activities so that it can be influenced to occur, and needs to be monitored closely. For most Governments, at least one budget cycle is necessary to ensure that they are able to take over programming. This is particularly important when PEPFAR is going to change strategic direction and Governments are expected to pick up where PEPFAR has left off in order to avoid gaps in services which PEPFAR mobilises, as will now be the case in the hard to reach areas. This is because PEPFAR 3 is no longer going to be working in hard to reach areas. 17. Two strategies are recommended. The first and less sustainable option is that USAID tries to ensure that the GoB is ready to absorb the program activities once the program closes out through consultations and keeping them informed of project progress through their participation in advisory 70 boards, such as with Maatla. The second, and possibly more reliable option, is that government is brought into the center as a core sub-partner. The former means that Government is aware of, and prepared for, changes in USAID programming. The second means that Government will also have systems in place to absorb the projects into their structures and work-plans. There could be a recurrent component that government is financing from project inception, so that it will be evident that there is support and this will also put the funding payment mechanisms in place that can be stepped up incrementally every year in place. Both of these require that USAID provide for longer timelines to allow for other sub-partners to catch up with their course corrections. not be implemented in parallel to government programs. 8.6 Project Management, M&E 18. A conceptual framework needed to have underpinned the design of the M&E system. There are some good ideas in the document produced by Measure in 2003 regarding the monitoring and evaluation of health systems. Designing the M&E system around health system outcomes would have made it easier to report on changes that had occurred and how the project contributed to these changes. 19. Projects of this scale should not forego baseline and mid-term reviews as occurred for Maatla. 20. It must be ensured that those who gather data are provided with timely feedback so that they can use it to improve planning and implementation. Providing feedback also assists with M&E compliance as people have an improved understanding of its purpose and benefits. 21. Because M&E is a fairly specialized skill, it is necessary to provide training, on-going mentoring and repetition to enhance quality compliance. Ideally, organizations should have specialized monitoring capacity or be provided with sufficient technical assistance to support these tasks. 22. An SOP for the OCA and TCA must be developed so that they can be used by others (such as BOCONGO) in the sector. 23. Outcomes reporting into monitoring systems must be included so that behavior change can be reported on, even if it is highly qualitative. One idea is to have “change stories” that are reported monthly. These accumulate over the period of implementation and eventually there is enough data to do a thematic analysis of the stories. 24. It would be beneficial if government officials were more involved in piloting the monitoring tools, e.g. the referral system, so that they are able to help roll them out to areas that are not part of the Maatla project. 25. Government needs to continue to collaborate in monitoring and evaluating the pilot so that all parties can learn as they roll out to other areas. 71 9 Appendices LIST OF APPENDICES: I. Bibliography II. List of Documents III. Evaluation Statement of Work IV. Maatla Results Framework V. Detailed description of methodology and evaluation matrix VI. Maatla Final Instruments (available separately for download) VII. Schedule of Capacity Assessments conducted VIII.Capacity Assessment Results IX. List of training conducted X. Lists of Toolkits and Guidelines Developed XI. USAID Criteria to ensure the quality of te evaluation report (USAID Evaluation Policy) 72 APPENDIX I: BIBLIOGRAPHY BONASO and NACA. 2010. “National Capacity Building, Strategic Framework for Botswana HIV and AIDS Service Organizations, 2010-2016.” Central Intelligence Agency. 2015. “The World Fact Book” [online] Available at: https://www.cia.gov/library/publications/the-world-factbook/rankorder/2155rank.html [Accessed: 30 September 2015]. FHI 360. 2011. “Organisational Capacity Assessment (OCA), Capacity Building Series”. FHI 360. 2012. “Attachment B: Program description, Botswana Civil Society Strengthening Program”. FHI 360. 2014. “Concept paper on Revised Maatla Technical Model”. Government of Botswana. 2010. “The Second Botswana National Strategic Framework for HIV and AIDS 2010-2016”. MEASURE Evaluation. 2003. “A Guide to Monitoring and Evaluation of Capacity building Interventions in the Health Sector in Developing Countries”, MEASURE Evaluation Manual Series, No. 7. NASTAD. 2015. “Botswana’s District HIV and AIDS Coordination Model—A Guide for Civil Society Organizations”. Republic of Botswana. 2015. “Progress Report of the National Response to the 2011 Declaration of Commitments on HIV and AIDS.” Southern Hemisphere. 2015. “Proposal for USAID Botswana Projects Performance Evaluation.” Statstics Botswana. 2013. “Botswana Impact Survey IV (BAIS IV), 2013”. USAID. 2010. Botswana Performance Evaluation RFTOP No. SOL-674-15-000007 73 APPENDIX II: LIST OF DOCUMENTS Document Type File Name Reports Annual Progress Report: Maatla October 1, 2013-September 30, 2014 Annual Progress Report: Maatla May 17, 2011-September 30, 2011 FHI360 Maatla project Cooperative Agreement No. 674-A-00-11-00049-00. FY12 Q4 Annual Report Narrative October 1, 201- September 30, 2012 Maatla project: Cooperative Agreement No.674-A-00-11-00049-00. Annual Report October 1, 2012-September 30, 2013 Cooperative Agreement No. 674-A-00-11-00049-00. FHI, Attachment B Program Description Maatla project Year IV Semi-Annual Report Log Frames, project proposals and strategic plans Maatla Program Description Application submitted—May 2011.pdf Maatla work-plan year 4—APPROVED.Doc Maatla work-plan year 3 narrative—revised July 10 final.doc Maatla work-plan year 2 matrix-submission 2012 OSMM.doc Maatla work-plan year 2 narrative submission 2012 OSMM.doc Maatla work-plan year 4 matrix. APPROVED.doc Revised Work-plan year 3 matrix—July 10,2014.doc Project implementation documents Article Botswana CSR Breakfast forum.docx BOTS Maatla PTQA Report (5 April) kt & AC.docx Concept paper on Maatla Technical Model. Oct 2014.docx Findings + Recommendations BOTS Maatla 2014—HQ DEBRIEF (April 7) (2).ppt Governance Success Story Feb 2015.pdf Maatla all partners presentation April 2015.pptx Maatla FY15 Implementation sites updates 21_Oct_2014.xls Maatla service delivery site support and supervisory checklist Revised 14 Nov 2014.docx 74 NACA CATCH report final 240914.pdf NASTAD final Maatla toolkit New word format 03-03-2015 (2).docx OVC success story Mar 2015.pdf Pre-co visit self-assessment—BOT Maatla.doc Referrals success story March 2015.docx Sides on SA learning visit May 2014.ppt Trip Report Nairobi June 2014.doc WUSC Presentation for SMT Feb 27 2015.ppt M & E documents￾monitoring tools (OCA, TCA) Technical assessment tool TAT II-Maatla FY 19 Sept 2014 Referral form—03 02 2015—revised 13 03—2015.rtf Referral MSF 16 Oct 2014 _4 Dec 2014 Revised 20 March 2015.xls Prevention session registration form—Maatla FY 12—3- Jan 2014.doc OVC and care tool June 13h 2014.xls OCA—2013 final. xls NCONGO DMSAC Reporting Tool Maun.xls Individual follow up form May 3112.doc Individual assessment tool draft May 3112.doc HIV prevention tools draft June 12 ’14. xls Final workshop report Chobe 12.pdf DCs tech assistance results Jan 2015.xls Counseling and testing feedback and TB tools BOCAIP Baseline and mid-term capacity assessment results district collaboration & coordination. xls Network Organisations Documents Fully executed fy 14.pdf Mod 2 fully executed.pdf Mod 2 partially executed.pdf Partially executed.pdf 75 Botswana 0599 0002-601131 Mod # 4 WUSC Partially executed Botswana 0599 009-601138 Mod # 2 BONASO Partially executed Fully signed amendment # 5 Partially executed Signed Agreement Amendment # 4 Fully executed Mod #4 Mod # 2 Partially executed Partially executed Mod # 4 Signed Documents Amendments # 2 BOCAIP SOW 19 September 2014 BOCAIP Work-plan Y14 19 September 2014 BONEPWA+ Year 4 SOW final Budget amendment # 4 Budget modification # 3 Signed amendment # 4 Signed modification # 3 Subagreement Modification # 3 Subagreement modification # 4 BONEPWA+ Year 4 SOW final Budget amendment # 4 Signed amendment # 4 Subagreement modification # 4 Budget modification # 5 Fully signed modification NCONGO NCONGO SOW—FY 15 NCONGO work-plan matrix year 4 final Subagreement modification # 5 76 Amendment #3 BOCONGO Amendment 3 BOCONGO MAATLA Work-plan YR4 MATRIX final BOCONGO SOW Y4 15 September 2014 MODIFICATION # 3 signed 07 Oct 2014 final BONASO SOW narrative year 4 07 Oct 2014 final BONASO work-plan year 4 BONASO subagreement # 3 Modification # 3 Signed MOD # 3 Modification # 5 Signed agreement—Year 1V WUSC Modification # 5 WUSC SOW Year IV Draft September 25 Final work-plan Year 14-15 Final Y4 Scope of work MOD # 6 NASTAD Modification # 6 NASTAD Subagreement modification # 6 USAID Documents Request for Applicants (RFA) No. 674-11-002 Orphans and Vulnerable Children (OVC) & Gender Project in Botswana, 2010 Amendment No. 1—RFA 674-11-002 Orphans and Vulnerable Children (OVC) & Gender Project in Botswana, 2010 Request for Applicants (RFA) No. 674-11-003 Botswana Civil Society Strengthening Project, 2011 Amendment No. 1—RFA 674-11-0002 Orphans and Vulnerable Children (OVC) & Gender Project in Botswana, 2010 77 APPENDIX III: EVALUATION STATEMENT OF WORK RFTOP No. SOL-674-15-000007 Botswana Performance Evaluation Page 2 of 43 SECTION B – SUPPLIES OR SERVICES AND PRICE B.1 PURPOSE To provide support to the United States Agency for International Development’s Mission in Southern Africa (USAID/Southern Africa) to conduct a performance evaluation of two USAID activities that are both using a community-based approach to involve and mobilize communities in their response to HIV/AIDS. The two activities to be evaluated are the Maatla Civil Society Strengthening activity and the Tsela Kgopo Orphans and Vulnerable Children and Gender activity. In this performance evaluation, USAID seeks to understand if the activities have been implemented well so far, and whether any mid-course adjustments need to be made. USAID will use the results of this performance evaluation to determine whether an impact evaluation of the same activities is warranted in the near future. B.2 CONTRACT TYPE This is a Firm, Fixed-Price Task Order. For the consideration set forth in the contract, the Contractor shall provide the deliverables or outputs described in Section F and comply with all contract requirements. B.3 FIRM FIXED PRICE The firm, fixed-price is $________________. B.4 PAYMENT The paying office is set forth in section G.4. B.5 OTHER RFTOP INFORMATION The final statement of work for the task order that will result from this RFTOP will be incorporated at the time of award and shall be based on the proposal by the successful Offeror. B.6 PAYMENT SCHEDULE Payment of the fixed price shall be made on the schedule specified below: Deliverable (Percent (%) of Firm Fixed Price) 1. Work Plan approved by USAID _____40%_______ 2. Debriefings to USAID and stakeholders completed _____30%_______ 3. Final evaluation report approved by USAID _____30%_______ TOTAL _____100%______ Payments of the amounts specified above are subject to approval by the Contracting Officer’s Representative (COR) and acceptance of the milestone event or deliverable by the COR. The paying office is USAID/SA as indicated in section G.4; payment processing shall be in accordance with the payment schedule in section G.4 of this RFTOP. END OF SECTION B RFTOP No. SOL-674-15-000007 Botswana Performance Evaluation Page 3 of 43 SECTION C – DESCRIPTION/SPECIFICATIONS/STATEMENT OF WORK PROJECTS TO BE EVALUATED: 1. Project name: Maatla (Botswana Civil Society Strengthening Program) Cooperative Agreement No.: 674-A-00-11-00049-00 Project Dates: May 17, 2011 through May 16, 2016 Agreement Value Range: $15 - $20 million Implementing Organizations: Family Health International 360 (FHI360, Prime). Sub-contract: Inveneo. Sub￾agreements: Botswana Network of AIDS Service Organizations (BONASO); Botswana Christian AIDS Intervention Program (BOCAIP); Botswana Network of People Living with HIV/AIDs (BONEPWA); Botswana Council of Non-Governmental Organizations (BOCONGO); Botswana Confederation of Commerce, Industry and Manpower (BOCCIM); National Alliance of State and Territorial AIDS Directors (NASTAD); Ngamiland Council of Non￾governmental Organizations (NCONGO); and World University Service of Canada (WUSC). 2. Tsela Kgopo (Botswana Orphans & Vulnerable Children and Gender Project) Cooperative Agreement No.: 674-A-00-11-00048-00 Project Dates: June 13, 2011 through June 12, 2016 Project Funding: $15 - $20 million Implementing Organizations: Project Concern International (PCI) (prime). Sub-awards: Hope Worldwide (HWW); Catholic Relief Services (CRS); Humana People to People (HPP); Marang Child Care Trust (MCCNT); Kagisano Society Women’s Shelter Project (KWSP); Kgetsi Ya Tsie (KYT); Bakgatla Bolokang Matshelo (BBM): Mother’s Union (MU); Botswana Association of Positive Rehabilitation (BAPR); and House of Hope (HOH). 1) Objective USAID/Botswana’s PEPFAR (President’s Emergency Plan for AIDS Relief) Program seeks to strengthen local health-service-provider organizations in Botswana to improve health outcomes among beneficiaries and communities. The aim of this solicitation is to conduct a performance evaluation1 of two USAID activities that are both using a community-based approach to involve and mobilize communities in their response to HIV/AIDS. The two activities to be evaluated are the Maatla Civil Society Strengthening activity and the Tsela Kgopo Orphans and Vulnerable Children and Gender activity. In this performance evaluation, USAID seeks to understand if the activities have been implemented well so far, and whether any mid-course adjustments need to be made. USAID will use the results of this performance evaluation to determine whether an impact evaluation of the same activities is warranted in the near future. This evaluation will include immediate collection of information on the quality of implementation so far, to allow for mid-term corrections in programming, and on the organizational and technical capacity of sub-partners. Routine and internally collected data on the institutional and technical capacity of the local organizations being supported by the activities should be used by evaluators to monitor trends in their 1 The previously published evaluation solicitation requested an impact evaluation, but USAID has decided to have evaluators conduct a performance evaluation first. If results of the performance evaluation suggest the activities have been well implemented and are likely to be resulting in a change in outcomes among beneficiaries, USAID will consider soliciting for an endline impact evaluation of one or both activities at some time in the future. According to USAID’s evaluation policy (http://www.usaid.gov/evaluation/policy), USAID defines evaluations as either being “impact” evaluations or “performance” evaluations. Impact evaluations are those that use a rigorously defined counterfactual (experimental or quasi-experimentally derived) to attribute changes in outcomes to an intervention. Performance evaluations are everything else. RFTOP No. SOL-674-15-000007 Botswana Performance Evaluation Page 4 of 43 organizational capacity. This internal data should be supplemented by additional data collected by the evaluators to assess changes in organizational outcomes, without attribution to the prime partners. This evaluation should inform USAID of the quality of activity implementation so far and whether a later impact evaluation is warranted. This evaluation will shape the nature of potential future USAID activities. 2) Purpose and Use The evaluation’s purpose is to assess: (1) how well the activities are being implemented so far, whether according to plan; (2) what prioritized, immediate changes should be made in activity implementation to improve performance; (3) how follow-on activities might be informed from the experience of these activities; and (4) whether the activities are being implemented well enough to warrant an impact evaluation before the activities end. Evaluation results will be used as formative data for the activities so that they can better tailor their interventions to beneficiary needs during the final years of implementation. The evaluation will also document lessons learned and best practices from the significant investments made in the activities. The primary audience of the evaluation will be USAID/Botswana and its implementing partners. A secondary audience of the evaluation is the Government of Botswana and other stakeholders, such as other USG agencies, bilateral and multilateral donors, other NGOs, etc. 3) Background of activities2 The Maatla Civil Society Strengthening activity and the Tsela Kgopo Orphans and Vulnerable Children and Gender activity were designed in tandem and share complementary mandates to work with local non￾governmental organizations (NGOs). The two activities are intended to contribute toward the achievement of USAID Botswana’s Results Framework (see Appendix 1), which includes three objectives: (1) Strengthen community agency to seek, support and provide HIV/AIDS related services; (2) Enhance NGO capacity to complement HIV/AIDS response in Botswana; and (3) Strengthen GOB capacity to lead HIV/AIDS response in Botswana. Additionally, Botswana’s activities also contribute to the Regional Development Corporation Strategy (RDCS) Development Objective 2, “Reduced impact of HIV/AIDS in the region,” via its Intermediate Result 2.1, “Mitigate the spread and impact of HIV/AIDS in non-presence countries.” 3.1. Maatla (Botswana Civil Society Strengthening Program) Maatla is a five-year activity managed by FHI 360 and funded by USAID/Botswana with a start date of May 17, 2011. The goal is to significantly and sustainably strengthen the capacity of the civil society sector in Botswana to support HIV and AIDS and related health service delivery. This is meant to be achieved through collaboration with national non-governmental organization (NGO) networks, district￾level coalitions and government structures, providing both support for organizational development and technical capacity building. Maatla works in close consultation with USAID/Botswana, the Ministry of Local Government and Rural Development (MLGRD), the Ministry of Health (MoH), Ministry of Labor and Home Affairs (MLHA) and the National AIDS Coordinating Agency (NACA). 2 In a previously published version of this solicitation, the implementing partner activities were referred to as “projects.” Efforts have been made to remove such references, in accordance with USAID’s current definitions of “project” and “activity”. RFTOP No. SOL-674-15-000007 Botswana Performance Evaluation Page 5 of 43 In order for Maatla to achieve its goal of significantly and sustainably strengthening the capacity of civil society in Botswana in support of HIV/AIDS service delivery, it was designed to achieve the following results: ● Result 1: Strengthened technical competence and organizational capacity of national NGOs and networks to support field offices, affiliates, and member organizations to expand and strengthen delivery of strategic and high quality HIV/AIDS services. ○ Select indicators: • % CSO meeting minimum standards for leadership, governance, management of finances, human resources, inter-organizational linkages, infrastructure, etc. • Mean improvement in Organizational Capacity Assesement (OCA) scores per domain (see Appendix 5) • % CSOs implementing HIV/AIDS prevention, care, and treatment programs according to minimum standards. • Mean improvement in Technical Capacity Assessment (TCA) scores per domain (see Appendix 5). ● Result 2: Model(s) developed and implemented to strengthen district-level systems (CSOs and local government) to provide and coordinate HIV/AIDS services in hard-to-reach areas. ○ Select indicators: • # / % of people successfully referred for targeted clinical services (e.g., PMTCT, SMC, HTC, ART, TB/HIV) • # of # of targeted populations reached with individual and /or small group level preventative interventions (adults, youth, PLHIV) • # of eligible adults and children provided with a minimum of one care service • District reporting rate for CSOs, and mean improvement in reporting rate • Mean improvement in district coordination and collaboration between government and CSOs (see Appendix 5 for this tool) • # of community health workers trained in HIV-related service delivery • # of patients receiving community DOTs • # of individuals reached by activities which promote the reduction of gender based violence • # CSOs reached within the model districts • # of accredited training/capacity building materials developed by NCONGO • # of member organizations receiving trained or mentored by NCONGO • Organizational and technical capacity indicators as above, but for targeted districts and district-level CSOs ● Result 3: Strengthened institution(s) able to provide sustainable capacity building to civil society. ○ Select indicators: • # of successful funding proposals by BONASO and BOCONGO • # of accredited training/capacity building materials developed by BONASO or BOCONGO • # of member organizations or affiliates trained or mentored by BONASO or BOCONGO • % increase in non-USG funding for BOCONGO and BONASO To achieve Result 1, Maatla focuses national-level capacity building on four national NGOs with mandates as leaders of civil society and its response to HIV/AIDS: BONASO, BONEPWA+, BOCAIP and BOCONGO. According to the GOB National Operational Plan (NOP), BONASO, BONEPWA+ and BOCAIP are expected to play lead roles in delivering the CSO technical response to HIV through their members and affiliates; yet, they require significant capacity upgrading to play these roles. BOCONGO RFTOP No. SOL-674-15-000007 Botswana Performance Evaluation Page 6 of 43 was elected as the fourth NGO for capacity building because of its mandate of CSO organizational development and its multi-sectoral CSO membership. Together, the four organizations provide maximum reach into Botswana‘s civil society sector so that Maatla’s efforts should strengthen the technical and organizational capacity of district-level and local CSOs. For Result 2, Maatla aims to strengthen district level capacity by: (1) expanding the participation of local CSOs, the technical quality of their interventions, and their basic management and financial systems to facilitate effective operations through the building of strong multi-sectoral coalitions; and (2) measurably enhancing the ability of government and CSOs to work together to accomplish shared responsibilities for district-level design, implementation, reporting, advocacy and community and private sector mobilization. Maatla will work to develop models in Northwest Ngamiland, Okavango, Ghanzi and Chobe districts in a phased approach. To achieve Result 3, the Maatla targets support to two organizations, BONASO and BOCONGO, to sustain capacity building of civil society over the long term. As the lead program coordinator of AIDS service organizations in Botswana, BONASO is expected to monitor and respond to technical capacity needs, particularly as the epidemic evolves and new evidence-based best practices emerge to address them. BOCONGO is the key national coordinator for institutionalizing NGO capacity in Botswana. Maatla aims to address both organizational capacity building as well as increasing the technical competency in a wide range of HIV/AIDS-related services in the following areas: Sexual prevention, HIV Testing and Counseling (HTC), community TB/HIV care, adult and pediatric care and support, Gender Based Violence (GBV) and support for orphans and vulnerable children (OVC). Maatla aims to advance Botswana‘s ability to achieve its national goals for HIV/AIDS service delivery in the areas of treatment, care, and prevention through implementation of capacity-building strategies to move civil society partners closer to independence and program sustainability. Maatla should also develop models and strategies that will help guide Botswana’s response to HIV/AIDS and inform the National Strategic Framework on HIV/AIDS (NSF) II and other strategic documents. The theory of change assumes that by building the capacity of CSOs and local government coordination structures, the activity will leave structures more capable of supporting the HIV/AIDS response and broader health needs in Botswana. This program focuses on capacity building for the delivery of a wide range of HIV/AIDS related services and provides support to civil society organizations, particularly to those working in the areas of HIV prevention, emphasizing behavioral, biomedical, and structural relevance; HIV testing and counseling; care and support, including OVC; community TB/HIV; and adult and pediatric palliative care. The capacity building for CSOs covers both organizational and technical domains. The organizational aspect includes the following domains: Strategic Leadership, Organizational Structures (Governance), Human Resource Management, Financial Management, Infrastructure, Program and Service Management, Process Management and Inter-organizational linkages. The technical aspect includes program areas of prevention and care and support for adults, youth and OVC. With organizational capacity support for the national-level NGOs, the activity at year four intends to have groomed one or more organizations to be able to build capacity of its affiliates/members. The activity is currently supporting BONASO and BOCONGO to assume this role. The activity aims to strengthen district-level systems to enable them to better provide and coordinate HIV/AIDS services in hard-to-reach and underserved areas. This support should result in local government bodies being able to lead and coordinate HIV/AIDS activities at the district level and it RFTOP No. SOL-674-15-000007 Botswana Performance Evaluation Page 7 of 43 should result in more effective coordination among all partners working in the district. The activity should also result in strengthened organizational leadership and service-delivery capacity. Maatla Activities BOCAIP and BONEPWA+ Sites • Gumare Counseling Center in Okavango. The center has volunteers placed in several villages, including Gumare, Etsha1, Etsha 6 and Etsha 13, Ngarange, Nokaneng, Tsau, Sepopa, Shakawe and Seronga. • Maun Counseling Centre in Ngami. The center has volunteers operating in Maun and several neighboring villages, including Makalamabedi, Phuduhudu, Toteng, Shorobe and Sehitlwa. BOCAIP has plans to expand its sites to Chobe with the current work plan. BONEPWA+ has sites in Okavango, Ngamiland, Chobe and Ghanzi districts. The main site in Ngami is the BONEPWA+ Center of Excellence in Maun, which has volunteers operating in Komana, Toteng and Sehitlhwa. • Okavango/Ngami: BONEPWA+ has volunteers operating in Etsha 1, Etsha 6, Etsha 13, Shakawe, Maun,Gumare, Nokaneng, and Sehitwa • Chobe: The main site is BONEPWA Center of Excellence in Kasane, which has volunteers placed in Pandamatenga, Parakarungu, Kachikau and Kazuungula. • Ghanzi: The main site is the BONEPWA+ Center of Excellence in Ghanzi Township, which has volunteers covering D’kar, Charles Hill, New Xade, and Ghanzi Township. Table 1 below shows the Maatla implementing partners’ original indicators and 5-year targets3 . Appendix 2 gives a map of Maatla activity sites. Table 1. Maatla indicators and life-of-activity targets for number of beneficiaries served PEPFAR Indicators Totals Maatla 5-year targets, by district and partner Ngami Okavango Ghanzi Chobe BOCAIP BONEPWA BOCAIP BONEPWA BONEPWA BONEPWA # of targeted populations reached with individual or small group level preventative interventions that are based on evidence or meet the minimum standards required 26,600 6,605 3,472 8,489 1,205 5,186 1,642 # of targeted populations reached /with individual or small group level preventative interventions that are primarily focused on abstinence or being faithful, and are based on evidence or meet the minimum 18,850 5,669 1,372 6,487 1,760 2,370 1,192 3 Indicators and targets may have changed slightly since the project has started. We provide the originals to give bidders a sense of what activities are taking place and where, to help prepare their proposals. Final indicators and targets will be shared with the winning bidder. RFTOP No. SOL-674-15-000007 Botswana Performance Evaluation Page 8 of 43 PEPFAR Indicators Totals Maatla 5-year targets, by district and partner standards required # of people reached by individual, small group or community level intervention or service that explicitly addresses gender based violence & coercion related to HIV/AIDS 9,350 2,312 1,671 2,419 749 1,497 702 # of people who received Testing and Counseling services for HIV and received their test results 15,340 8,557 6,783 # of People Living w/ HIV/AIDS (PLHIV) reached with a minimum package of package of Prevention with PLHIV (PwP) interventions 3,800 217 2,057 185 1,342 # of eligible adults and children provided with a minimum of one care service 11,400 1,973 2,359 1,946 2,956 1,340 826 N.B Please note that partner targets may change over the life of the activity due to partner performance, funding, and level of effort. Some of the indicators in the above table have also been retired or modified during the course of the activity. Changes to Maatla since Implementation Started Maatla is now in year four of implementation. Changes to the activity since implementation began include the following. a) Inclusion of service delivery The Maatla Activity started as a Health Systems Strengthening (HSS) capacity-building activity; however, in year two, FHI 360 was advised to include support to service delivery in the four model districts. Service delivery was started in the last quarter of year two. b) Two partners in crisis In the first two years of the activity, two partners were in a crisis situation that was not anticipated. BONASO had lost all secretariat staff in year one, while BONEPWA+ lost its three key personnel (Executive Director, Programs Manager and Finance Manager) in year two. Therefore, for the first two years, BONASO and BONEPWA+ were being resuscitated. This limited the type and intensity of capacity building services for these organizations. c) Technical support to the government The Maatla Activity was asked in year two to include technical support to the National AIDS Coordinating Agency (NACA) to develop national combination prevention strategies. This included: RFTOP No. SOL-674-15-000007 Botswana Performance Evaluation Page 9 of 43 • National HIV Prevention Pitso, a strategic activity for stakeholders and community members working on HIV prevention to reflect on past activities and ongoing activities, interventions, and future plans so as to re-focus prevention interventions. • Designing, printing, and disseminating the HIV Prevention Pitso report. • Training of 30 combination-prevention champions (program officers from various stakeholders). • Consultative meetings for the development and dissemination of the CATCH (Communities Acting Together to Control HIV) model. The goal of CATCH is to establish a new comprehensive service-delivery model at the community level that would integrate HIV prevention, treatment, and care to improve the realization of zero new HIV infections in Botswana. The above was additional, unanticipated work for the activity personnel. d) Support to GeAD In year three, Maatla was asked to support the Ministry of Labor and Home Affairs (MLHA), Gender Affairs Department (GeAD) to implement a GBV referral pilot in two pilot districts (Kgatleng and Ngami). This also added additional work onto the limited number of FHI 360 activity personnel, more so that FHI 360 was advised by USAID against recruiting additional staff from year 3 onwards. e) Exit of BOCCIM from Maatla BOCCIM was initially spearheading the private-sector mentorship component, but it has exited from the activity. The process of exit affected the private-sector mentorship deliverables and FHI 360’s ability to meet cost-share targets. FHI 360 is now implementing the Private Sector-CSO Mentorship program directly, although without additional staff to make up for the two previously employed through BOCCIM. This also has implications on FHI 360’s outputs regarding technical assistance and mentorship from the private sector. f) Budget cuts In year four, the Maatla activity will experience challenges with the planned Organizational Development (OD) capacity building interventions due to cuts to the HSS budget by approximately one-third in order to meet the PEPFAR-mandated care and treatment earmark. WUSC and partner NGOs and networks had to scale back or drop several of their planned OD activities, which might constrain the activity’s ability to achieve desirable outcomes in OD capacity building. g) Changing of indicators and introduction of SIMS The activity has had to respond to substantial changes in PEPFAR indicators, together with new tools, such as PEPFAR’s Site Improvement through Monitoring System (SIMS). Changes such as these have led to additional time and resources being spent on updating activity training manuals, job aids, and M&E tools. RFTOP No. SOL-674-15-000007 Botswana Performance Evaluation Page 10 of 43 3.2 Tsela Kgopo (Botswana Orphans & Vulnerable Children and Gender Project) Tsela Kgopo is a five-year activity managed by Project Concern International (PCI), with funding from USAID/BW and a start date of June 13, 2011. The goal is to improve the quality of life of orphans, vulnerable children, and their caregivers and to improve gender-based responses and programming in HIV/AIDS activities in Botswana. With a family/household perspective, the activity intends to achieve this goal through implementing activities that achieve the following results: ● Result 1: Livelihoods of households for vulnerable adolescents, especially girls and vulnerable women, improved, through achievement of the following results: (1.1) Community structures to identify and support vulnerable households and linkages to other relevant services strengthened; (1.2) Participation of vulnerable households, including men, in economic strengthening activities increased; and (1.3) Livelihoods opportunities for vulnerable youth increased. ○ Select indicators: • Mean (or median) increase in income • % employed (by age, sex, etc.) • % enrolled in vocational training • % vocational-tech graduates who are currently working (including self￾employed) • % of communities with community action plans implemented • % of households able to access money to meet important family needs ● Result 2: Developmental interventions for vulnerable children and adolescents improved, through achievement of the following results: (2.1) Access of most vulnerable children to quality early childhood development services increased; (2.2) Strengthened collaboration between parent/s caregivers and Integrated Early Childhood Development (IECD) centers for improved child development; and (2.3) Access of adolescents to mentoring, life skills, sexual/reproductive health and HIV prevention programs increased. ○ Select indicators: • % enrolled in IECD • % increase in enrollment in IECD programs • % change in enrolment, grades, retention and attendance in primary school • % youth with correct knowledge of HIV/AIDS transmission and prevention • % with safer sexual behaviors • % youth with stigmatizing attitudes toward PLHIV ● Result 3: Risk of neglect, exploitation and abuse of vulnerable children, adolescents and women reduced, through achievement of the following results: (3.1) Strengthened systems (referrals networks and community structures) to prevent abuse and meet the needs of those neglected, exploited and abused; (3.2) Services that cater for and protect abused children and women increased; (3.3) Child welfare work force skills to recognize signs of and reduce potential abuse and exploitation improved; (3.4) Involvement of men as peer educators, mentors, care-givers and role models increased; and (3.5) Awareness of rights and ability to seek help by vulnerable children, adolescents and women increased. ○ Select indicators: • % women, youth, and children suffering from physical, emotional, or sexual abuse • % of men expressing agreement with or practicing abusive beliefs or practices • % with knowledge of rights and services available • % accessing child- or woman-protection services • % with protection case resolved or acted on RFTOP No. SOL-674-15-000007 Botswana Performance Evaluation Page 11 of 43 ● Result 4: Development, implementation and coordination of national frameworks and policies that address the needs of children and women improved, through achievement of the following results: (4.1) Support to implementation of OVC and gender acts improved; (4.2) Support to development and implementation of OVC policy and draft national policy on gender and development; (4.3) National management of data for OVC and gender services strengthened; (4.4) National curriculum on gender analysis and gender mainstreaming developed and implemented; (4.5) Participation of children and women in policy planning and implementation increased; and (4.6) Logistical support to Department of Social Protection (DSP) and Gender Affairs Department (GAD) annual meetings. ○ Select indicators: • # of District Child Protection Committees (CPCs) and Village CPCs formed or strengthened in activity target areas. • # of districts where Gender and development policy and domestic violence act has been disseminated and implemented. • # of districts where the Children’s Act has been disseminated and implemented. To achieve activity objectives, PCI will partner with Catholic Relief Services (CRS), Hope World Wide Botswana (HWWB), Humana People to People (HPP), Marang Child Care Network Trust (MCCNT); the Government of Botswana (GOB); and other community and private sector organizations. The activity operates in Central, Kweneng, Southern, Southeast, Chobe, Francistown, Northwest, Ghanzi, and Kgalagadi Districts, while helping to catalyze enhancements in other districts through national initiatives. The Tsela Kgopo OVC and Gender activity uses Journey of Life (JOL) and Community Capacity Enhancement – Community Conversations methodologies to sensitize and mobilize communities for effective action. These methodologies bring a diverse range of community actors together to analyze problems that touch on issues of neglect, abuse, and gender, and to then develop and commit to solutions. The use of these methodologies has an almost immediate effect of strengthening linkages and coordination among families, school, health centers, social services, faith institutions, village development committees, and other community members leading to increased sense of community ownership and responsibility. The program will serve 18,922 (as per the CA) people directly with services over 5 years. Table 2 below shows the numbers that each partner will contribute towards the overall life-of-activity target of 19,600, and Appendix 3 gives a map of Tsela Kgopo sites. Table 2. Targeted Tsela Kgopo beneficiaries by partner and site. Partner & Site Number of beneficiaries Kgetsi Ya Tsie, Lerala 1,200 Kagisano Women’s Shelter Activity sites 1,000 • Gaborone & Molepolole Humana People to People sites 3,600 • Ghanzi 600 • Selibe Phikwe 1,200 • Charleshill • Mabutsane 1,800 Hope World Wide Botswana sites 4,100 • Molepolole, Mmakanke, Mosokotso, Mmanoko, Mmatseta, Kopong, Mogonono Catholic Relief Services sites 4,400 • Francistown 600 • Kasane 700 • Gumare 900 • Maun 800 • Sebina 700 • Palapye 700 House of Hope, Palapye 2,000 Mother’s Union, Mahalapye 1,500 Bakgatla Bolokang Matshelo, Mochudi 1,200 Botswana Association of Psychosocial Rehabilitation, Lobatse 600 N.B Please note that partner targets may change over the life of the activity due to partner performance, funding, and level of effort. Changes to Tsela Kgopo Since Implementation Started Tsela Kgopo is now in year three of implementation. There have been no major changes to the activity other than the decision not to award a sub-grant to Positive Innovation for the Next Generation (PING). 4) Existing Data Sources Existing data for the two activities is summarized in Table 3 below. Data collection tools used by both implementing partners can be found in Appendix 5. Data from these tools will be made available to the evaluators. Table 3. Summary of existing data sources for potential use in the evaluation Data available Brief description of data Organisation RFTOP No. SOL-674-15-000007 Botswana Performance Evaluation Page 13 of 43 Organizational capacity assessment results Results from standards-based assessments of partner organizations’ institutional capacity Maatla Technical capacity assessment results Results from standards-based assessments of partner organizations’ technical capacity Maatla District OCA & TOCA reports Results from standards based assessments of District Maatla Maatla & Tsela Kgopo progress reports Program data on people and organizations reached by the activities USAID, Tsela Kgopo & Maatla Child Status Index Longitudinal, client-level data on status of the child monitoring the needs, services provided, and progress of child wellbeing Tsela Kgopo District profiles HIV prevalence data, names and number of service providers per district Tsela Kgopo & Maatla 2011 Population & Housing Census; BAIS I-IV reports; other GOB survey reports (e.g., Family Planning, Income & Expenditure) - Government-produces reports on numbers of people in Botswana, by geography and demographics - Other reports are topic specific and present frequencies of traits of interest, based on sample surveys Central Statistics Office Botswana HIV prevention program data Numbers of people reached by activity partners with prevention services Tsela Kgopo & Maatla Care and Support Numbers of people reached by activity partners with care and support services Tsela Kgopo & Maatla Community TB/HIV data Numbers of people reached by activity partners with TB services Maatla GIS data Coordinates of partner and other CSO/GOB services in target districts Maatla CSO HTC data Results from BOCAIP HIV Testing and Counseling activities Maatla Training data/TraiNet Results from all activity training activities Tsela Kgopo & Maatla SIMS service assessments Standard PEPFAR tool for assessing the quality of services at facility and community sites, and at above-site assessment points. Tsela Kgopo & Maatla 5) Additional Data Potentially to Be Collected via this Evaluation 5.1 Maatla • Mid-term organizational and technical capacity assessments of sub-partners and districts • Qualitative interviews with program staff, beneficiaries, and other stakeholders 5.2 Tsela Kgopo • Mid-term technical capacity assessments of sub-partners • Qualitative interviews with program staff, beneficiaries, and other stakeholders RFTOP No. SOL-674-15-000007 Botswana Performance Evaluation Page 14 of 43 • Qualitative interviews with program staff, beneficiaries, and other stakeholders 6) Evaluation Questions The evaluation questions to be answered are: 1. How likely is each activity to achieve their desired end results, as defined in each activity’s results framework? (Three results for Maatla, as above, in Section 3.1; four results for Tsela Kgopo, as above, in Section 2.) 2. What modifications, if any, to the strategic approach and theory of change have potential to substantially increase program effectiveness? 3. To what extent has each activity been implemented according to activity design? Where there have been deviations or shortfalls, what are the causes? 4. To what extent has USAID fulfilled its role in the activities by providing adequate funding, responsive managerial oversight and technical support, and assistance in solving problems in implementation as they arise? 5. What actionable, prioritized lessons have been learned for the design of potential follow-on activities? I.e., what should continue and what should change from the current activities in a potential future activity? 6. Has implementation so far been successful enough to warrant an impact4 evaluation of each activity before they close? If not, why not? 7) Evaluation Design and Methodology USAID is currently interested in a performance evaluation of these two activities. USAID expects that both quantitative and qualitative methods may be used. USAID expects the evaluators to make extensive use of existing, routinely collected data by the implementing partners, as listed in Table 3 above. Offerors should propose the most rigorous but cost-effective methods they think feasible to assess performance so far, given budget limitations. As part of their proposal, ALL OFFERORS must complete the table in Appendix 4 and give their ideas for methods to be used in answering each evaluation question. Offerors should also consider whether secondary data from national surveys and other studies may be useful in assessing performance. See list of potentially relevant household surveys at the Botswana Central Statistics Office website5 . Such surveys may include the Botswana Core Welfare Indicator Survey, the Botswana AIDS Impact Survey, the Botswana Demographic Survey, and the Botswana Family Health Survey. To answer questions on the quality of implementation, USAID expects the methods to include thorough, rigorous reviews of activity documents (activity awards, M&E plans, annual workplans, progress reports, OCA assessments, and any special studies undertaken by the 4 According to USAID’s evaluation policy (http://www.usaid.gov/evaluation/policy), USAID defines evaluations as either being “impact” evaluations or “performance” evaluations. Impact evaluations are those that use a rigorously defined counterfactual (experimental or quasi￾experimentally derived) to attribute changes in outcomes to an intervention. Performance evaluations are everything else. 5 http://www.cso.gov.bw/index.php?option=com_content1&parent_id=370&id=430&nid=370 RFTOP No. SOL-674-15-000007 Botswana Performance Evaluation Page 15 of 43 activities), key informant interviews and focus groups, online surveys (if appropriate), visits to service-delivery sites, and extensive use of routinely collected program data to analyze changes in the reach, efficiency, and service quality of the service-delivery CSOs whose capacity is being built by the prime partners (FHI360 and PCI). The evaluators will have access to routine program data, collected by FHI 360 and PCI, listed above, but will be expected to collect complementary primary data nonetheless in order to get the most objective evaluation possible. Although there is routine data from the Child Status Index on some beneficiaries (orphans and vulnerable children), the reliability of this data for evaluation purposes has been questioned by studies in other countries where the CSI is in use. Upon award, but before fieldwork is conducted, the contractor will submit a detailed evaluation design, methodology and implementation plan for review and approval by USAID. Offerors shall develop innovative approaches to conduct this evaluation, using the attached program description for the two activities, the USAID Evaluation Policy6 and any other relevant information. The methodology will be comprised of a mix of tools appropriate to the evaluation’s research questions. It is expected that the offeror will discuss the relative strengths and limitations of the methodology proposed within the proposal. The methodology should take into account – and perhaps independently assess, where possible – the quality of the routine monitoring data being collected by the activities. Additionally, the offeror should discuss data disaggregation (by gender and other categories) and gender considerations in the evaluation. Before data collection, the contractor in coordination with USAID will finalize the data analysis methods as part of the methodology plan. Potential Limitations to Evaluation Methods There are several important limitations to answering the evaluation questions posed in this solicitation. For assessing the change in institutional capacity that occurred among sub-partners during the activity, evaluators are expected to use a simple pre-post analysis, or propose whatever they believe is most likely to be most effective, even if the change cannot be attributed to the activity in a rigorous way. Although evaluators may want to collect their own data on organizations’ capacity, they will have access to internally collected data on organizational capacity, but this data may be subject to bias because it was collected by the implementing partners. Although evaluators will have access to beneficiary outcome data in the form of Child Status Index data (beneficiary cohort data on nutrition, psychosocial, educational, and other outcomes), collected by program implementation staff, this data has proven unreliable for evaluation purposes in other settings, so use of it in this evaluation should be done with caution and the appropriate caveats. 6 http://www.usaid.gov/evaluation/policy RFTOP No. SOL-674-15-000007 Botswana Performance Evaluation Page 16 of 43 Offerors are encouraged to suggest ways to mitigate the weaknesses mentioned above, noting that weaknesses in the design and data must be stated explicitly in the final evaluation report in alignment with USAID policy END OF SECTION C RFTOP No. SOL-674-15-000007 Botswana Performance Evaluation Page 17 of 43 SECTION D – PACKAGING AND MARKING D.1 TASK ORDER PACKAGING AND MARKING Task order packaging and marking shall be performed in accordance with the Evaluation Services IQC, Section D. END OF SECTION D RFTOP No. SOL-674-15-000007 Botswana Performance Evaluation Page 18 of 43 SECTION E - INSPECTION AND ACCEPTANCE E.1 TASK ORDER INSPECTION AND ACCEPTANCE Task order inspection and acceptance shall be performed in accordance with the Evaluation Services IQC, Section E.2. END OF SECTION E RFTOP No. SOL-674-15-000007 Botswana Performance Evaluation Page 19 of 43 SECTION F – DELIVERIES OR PERFORMANCE F.1 PERIOD OF PERFORMANCE This projected Task Order is expected to begin on or about February 18, 2015 and be completed no later than June 8, 2015. This would include preparation days, in-country work, and report writing and finalization of all deliverables F.2. DELIVERABLES The offeror should propose a timeframe for each deliverable, not exceeding the final date of performance. 1. Team-planning meeting At least one team planning meeting will be conducted in advance of any international travel. This can be done over the phone. This meeting will allow USAID to present the evaluation team with the purpose, expectations, and agenda of the assignment as well as provide any feedback on the evaluation design and methodology submitted as part of the proposal. 2. Work plan Before international travel is authorized; the team will prepare a detailed work plan that will include the methodologies to be used in the evaluations. The work plan will be submitted to the COR at USAID for approval after the activity is awarded. 3. Methodology plan & study protocol with local IRB approval Written methodology plans (e.g., evaluation design, key questions, methods, tools, data collection instruments, and data analysis plan, operational work plan) will be prepared by the evaluation team and approved by USAID prior to international travel. The written design of the evaluations will be shared with country-level stakeholders before being finalized. After USAID approval, the study protocol must be submitted to, and approved by, a local Institutional Review Board (IRB) in Botswana (see Appendix 6 for the schedule of IRB meetings). 4. In-country team planning meeting On day one of fieldwork, the evaluation team will hold a meeting at USAID/BW to discuss the methodology plans with in-country stakeholders, including the USAID and its implementing partners, the GOB, and other relevant partners. USAID or other stakeholders may join the evaluation team. 5. Interim briefings, including status reports The team leader will provide weekly status reports to USAID on work plan implementation. The evaluation team will also conduct at least two interim briefings with USAID while in Botswana to review the progress and methodology of the evaluations. 6. Debriefing with USAID The team will present the major findings of the evaluations to USAID after submission of each of the draft reports and before the team‘s departure from Botswana. 7. Debriefings with other stakeholders The team will independently present the major findings of the evaluations to the USAID partners (as appropriate and as defined by USAID) and to GOB through PowerPoint presentations prior to the team’s departure from Botswana. The debriefing will include a discussion of achievements, activities, and recommendations. The evaluation team will consider partner comments and revise the draft reports accordingly, as appropriate. 8. Draft evaluation report Draft reports of the findings and recommendations should be submitted to the USAID Task Order Contracting Officers Representative (TO COR). Proposals should specify when the partner anticipates this RFTOP No. SOL-674-15-000007 Botswana Performance Evaluation Page 20 of 43 to happen. The written report should describe the mid-term data collected, and any findings, conclusions, or recommendations. The report will be shared with the two activity partners. 9. Final evaluation report USAID has 14 working days to review the draft report and provide written comments to the contractor on the draft report. The contractor will have 14 working days to incorporate these comments into a final report. Contractor must provide both an electronic version and 5 hard copies of the final report to USAID. The reporting format is listed in Section 6. The report will be released as a public document on the USAID Development Experience Clearinghouse (DEC) (http://dec.usaid.gov), which the partner will need to submit to USAID for upload, after the TO COR provides formal written approval. 10. Data sets Upon activity completion, the contract will submit to USAID clean qualitative and quantitative anonymous data sets with code books and interview guides and questionnaires. F.3. REPORTING FORMAT AND CRITERIA Two separate reports will be produced for USAID, one for each activity. a) Format The report format should be restricted to Microsoft products and 12-point type font should be used throughout the body of the report, with page margins 1 inch top/bottom and left/right. The final evaluation report should not exceed 35 pages, excluding references and annexes. Final Evaluation Reports Each final evaluation report produced by the successful offeror shall: (1) Use rigorous evidence to answer the evaluation questions listed above; (2) Identify and articulate succinctly best practices and lessons learned; (3) Make clear, prioritized recommendations for future community-based health implementation and the design of forthcoming interventions. Additionally, the final reports should include: 1. Mid-term data on organizational and technical capacity of the sub-partners being strengthened by Maatla and Tsela Kgopo 2. Qualitative and/or quantitative interview data with program staff and other stakeholders on the quality of activity implementation, what things are working, what needs to be changed, etc. 3. Review and summary of internally collected program data to assess progress toward meeting the activity results/objectives. The evaluation reports (one for each activity) should include the following sections: Each Final Evaluation Report 1. Executive Summary: Summarizes activity purpose and background, key evaluation questions, methods, findings, and recommendations. (3-5 pgs.); 2. Table of Contents (1 pg.); 3. Introduction and Background: Purpose, audience, and synopsis of task, brief overview of Maatla / Tsela Kgopo activities in Botswana, USAID program strategy and activities implemented in response to the problem, brief description of implementing partners (1-2 pages); 4. Evaluation Design and Data Collection Methodology: Describes evaluation methods, including constraints and gaps (1-2 pg.); 5. Findings/Conclusions/Recommendations: For each evaluation question; also include data quality and reporting system that should present verification of spot checks, issues, and RFTOP No. SOL-674-15-000007 Botswana Performance Evaluation Page 21 of 43 outcome (17–20 pgs.); 6. Issues: Provide a list of key technical and/or administrative, if any (1–2 pgs.); 7. Future Directions: To inform the design of any further activities (2-3 pgs.); 8. Dissemination plan: The evaluation report should include a dissemination plan for how the findings of the evaluation will be disseminated to relevant stakeholders (e.g. reports, presentations, publications, agency websites, annual reports, policy briefs). 9. Use of findings plan: The evaluation report includes a stated plan for how the evaluation findings will be used for decision-making and program improvement (e.g. mid-course corrections, new procurements, resource allocation, and intervention uptake) and timeframe, if appropriate. 10. Total cost: The evaluation report should include the total cost of implementing the evaluation. 11. References (including bibliographical documentation, meetings, interviews and focus group discussions); 12. Annexes: Annexes must include the evaluation tools, schedules, interview lists, tables, all sources of information, the evaluation statement of work, statements of differences, weakness of the data, and a copy of USAID Criteria to Ensure the Quality of the Evaluation Report7 , all of which must be met. b) Final Report Criteria • The evaluation report should represent a thoughtful, well-researched, and well organized effort to objectively evaluate what is working in the activity, what is not, and why. • Evaluation reports shall address all evaluation questions included in this Scope of Work. • The evaluation report should include the scope of work (as written in the signed contract) as an annex. All modifications to the scope of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology, or timeline need to be agreed upon in writing by the Contracting Officer’s Representatives. • Evaluation methodology shall be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists, and discussion guides will be included in an Annex in the final report. • Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, reporting bias, etc.). • As much as possible, evaluation findings should be presented as analyzed facts, evidence, and data, rather than anecdote or hearsay. Findings should be specific, concise, and supported by strong quantitative or qualitative evidence. • Sources of information need to be properly identified and listed in an annex. • Recommendations should be action-oriented, practical, and specific, with defined responsibility for the action. F.4. TECHNICAL DIRECTION AND DESIGNATION OF RESPONSIBLE USAID OFFICIALS TASK ORDER CONTRACTING OFFICER: Adam Walsh Task Order Contracting Officer 7 USAID’s Criteria to Ensure the Quality of the Evaluation Report is available as Annex 1, pg. 11, of USAID’s Evaluation Policy, online here: http://www.usaid.gov/sites/default/files/documents/1868/USAIDEvaluationPolicy.pdf RFTOP No. SOL-674-15-000007 Botswana Performance Evaluation Page 22 of 43 U.S. Agency for International Development 100 Totius Ave Groenkloof 0027 Pretoria, South Africa Telephone: +27 12-452-2230 / Fax: +27 12-460-3177 Email: awalsh@usaid.gov The Task Order Contracting Officer Representative (TO COR): Dinah Zeltser Task Order Contracting Officer Representative U.S. Agency for International Development 100 Totius Ave Groenkloof 0027 Pretoria, South Africa Telephone: +27 12-452-2062 Email: dzeltser@usaid.gov F.5. PLACE OF PERFORMANCE The place of performance under this Task Order is Gaborone, Botswana, as specified in the Scope of Work. END OF SECTION F RFTOP No. SOL-674-15-000007 Botswana Performance Evaluation Page 23 of 43 SECTION G – TASK ORDER ADMINISTRATION DATA G.1 CONTRACTING OFFICE The Contracting Officer with authority to administer the order is Adam Walsh at: USAID/Southern Africa 100 Totius Street P. O. Box 43 Groenkloof 0027 Pretoria South Africa The Contracting Officer is the only person authorized to make or approve any changes in the requirements of this task order and notwithstanding any provisions contained elsewhere in this task order, the said authority remains solely in the Contracting Officer. In the event the Contractor makes any changes at the direction of any person other than the Contracting Officer, the change shall be considered to have been made without authority and no adjustment shall be made in the contract terms and conditions, including price. G.2 TECHNICAL DIRECTION The Contracting Officer will formally appoint a Contracting Officer’s Representative (COR) and an Alternate Contracting Officer’s Representative in a formal letter to the contractor. The Contracting Officer and the COR are the only official representatives of USAID for this contract and are the only ones authorized to provide technical direction to the contractor throughout the evaluation. The contractor is expected to work together with the COR to implement the scope of work. USAID will provide overall direction to the contracted evaluation team, identify key documents, and assist in facilitating a work plan. The evaluation team will be responsible for arranging meetings with key stakeholders and will be required to advise USAID prior to each of those meetings. The evaluation team is also responsible for arranging transportation in and around Botswana. The evaluation team will be responsible for procuring its own office space, computers, Internet access, printing, and photocopying. Evaluation team members will be required to make their own payments. USAID personnel will be made available to the team for consultations regarding sources and technical issues, before and during the evaluation process. USAID will send letters of introduction informing key GOB staff and other high-level partners of the nature, timing, and scope of the evaluation and of the evaluation team members. G.3 ACCEPTANCE AND APPROVAL In order to receive payment, all deliverables must be accepted and approved by the TO COR. G.4 INVOICES One original of each invoice shall be submitted on an SF-1034 “Public Voucher for Purchases and Services Other Than Personal” to the Financial Management Office at USAID/Southern Africa. One copy of the voucher and the invoice shall also be submitted TO COR. Electronic submission of invoices is encouraged. The SF-1034 must be signed, and it must be submitted along with the invoice and any other documentation in Adobe PDF to the Regional Financial Management Office to this address: invoice@usaid.gov RFTOP No. SOL-674-15-000007 Botswana Performance Evaluation Page 24 of 43 Paper invoices may be sent to the following address: USAID/Southern Africa, P.O. Box 43, Groenkloof, Pretoria 0027 or via courier to 100 Totius Street, Groenkloof, Pretoria If submitting invoices electronically, please do not send a paper copy. G.5 ACCOUNTING AND APPROPRIATION DATA (To be completed with included with the final task order): Budget Fiscal: Operating Unit: Strategic Objective: Team/Division: Benefiting Geo Area: Object Class: Amount Obligated: $ END OF SECTION G RFTOP No. SOL-674-15-000007 Botswana Performance Evaluation Page 25 of 43 SECTION H – SPECIAL TASK ORDER REQUIREMENTS H.1 AUTHORIZED GEOGRAPHIC CODE The authorized geographic code for the purchase of goods and services under this task order is 937. H.2 LANGUAGE REQUIREMENTS All deliverables shall be produced in English. H.3 GOVERNMENT FURNISHED FACILITIES OR PROPERTY The Contractor and any employee or consultant of the Contractor is prohibited from using U.S. Government facilities (such as office space or equipment) or U.S. Government clerical or technical personnel in the performance of the services specified in the Task Order unless the use of Government facilities or personnel is specifically authorized in the Task Order or is authorized in advance, in writing, by the TO COR. H.4 CONFIDENTIALITY AND OWNERSHIP OF INTELLECTUAL PROPERTY All reports generated and data collected during this project shall be considered the property of USAID and shall not be reproduced, disseminated or discussed in open forum, other than for the purposes of completing the tasks described in this document, without the express written approval of a duly-authorized representative of USAID. All findings, conclusions and recommendations shall be considered confidential and proprietary. H.5 CONTRACTOR’S STAFF SUPPORT, AND ADMINISTRATIVE AND LOGISTICS ARRANGEMENTS The Contractor shall be responsible for all administrative support and logistics required to fulfill the requirements of this task order. These shall include all travel arrangements, appointment scheduling, secretarial services, report preparations services, printing, and duplicating. H.6 ENVIRONMENTAL COMPLIANCE The Foreign Assistance Act of 1961, as amended, Section 117 requires that the impact of USAID’s activities on the environment be considered and that USAID include environmental sustainability as a central consideration in designing and carrying out its development programs. This mandate is codified in Federal Regulations (22 CFR 216) and in USAID’s Automated Directives System (ADS) Parts 201.5.10g and 204 (http://www.usaid.gov/policy/ads/200/), which, in part, require that the potential environmental impacts of USAID-financed activities are identified prior to a final decision to proceed and that appropriate environmental safeguards are adopted for all activities. [Offeror] environmental compliance obligations under these regulations and procedures are specified in the following paragraphs of this [RFTOP]. In addition, the Contractor/recipient must comply with host country environmental regulations unless otherwise directed in writing by USAID. In case of conflict between host country and USAID regulations, the latter shall govern. No activity funded under this task order will be implemented unless an environmental threshold determination, as defined by 22 CFR 216, has been reached for that activity, as documented in a Request for Categorical Exclusion (RCE), Initial Environmental Examination (IEE), or Environmental Assessment (EA) duly signed by the Bureau Environmental Officer (BEO). (Hereinafter, such documents are described as “approved Regulation 216 environmental documentation.”) An Initial Environmental Examination has been approved for projects funding this RFTOP/Task Order/Contract. The IEE covers activities expected to be implemented under this Contract. USAID has determined that a Categorical Exclusion applies to the proposed activities. This indicates that if these RFTOP No. SOL-674-15-000007 Botswana Performance Evaluation Page 26 of 43 activities are implemented subject to the specified conditions, they are expected to have no significant adverse effect on the environment. As part of its initial Work Plan, and all Annual Work Plans thereafter, the Contractor, in collaboration with the USAID COR and Mission Environmental Officer or Bureau Environmental Officer, as appropriate, shall review all ongoing and planned activities under this contract to determine if they are within the scope of the approved Regulation 216 environmental documentation. If the Contractor plans any new activities outside the scope of the approved Regulation 216 environmental documentation, it shall prepare an amendment to the documentation for USAID review and approval. No such new activities shall be undertaken prior to receiving written USAID approval of environmental documentation amendments. END OF SECTION H RFTOP No. SOL-674-15-000007 Botswana Performance Evaluation Page 27 of 43 SECTION I – CONTRACT CLAUSES All applicable clauses as outlined in the basic Evaluation Services Indefinite Quantity Contract (IQC) and its subsequent amendments (if any) are incorporated by reference. In addition, the following clauses are added: 52.232-39 Unenforceability of Unauthorized Obligations. (JUN 2013) 52.244-6 Subcontracts for Commercial Items. (JUL 2013) END OF SECTION I RFTOP No. SOL-674-15-000007 Botswana Performance Evaluation Page 28 of 43 SECTION J – LIST OF DOCUMENTS EXHIBITS AND OTHER ATTACHMENTS Appendix 1 – Results Framework for USAID/Botswana Appendix 2 - Map of Maatla’s districts Appendix 3 – Map of Tsela Kgopo sites Appendix 4 – Matrix of proposed methods by evaluation question Appendix 5- Maatla’s Organizational and Technical Capacity Assessment Tools Appendix 6 – Botswana Institutional Review Board Schedule END OF SECTION J RFTOP No. SOL-674-15-000007 Botswana Performance Evaluation Page 29 of 43 SECTION K – REPRESENTATIONS, CERTIFICATIONS, AND OTHER STATEMENTS Not required. END OF SECTION K RFTOP No. SOL-674-15-000007 Botswana Performance Evaluation Page 30 of 43 SECTION L - INSTRUCTIONS, CONDITIONS, AND NOTICES TO OFFERORS L.1 GENERAL The Government anticipates the award of one (1) task order as a result of this RFTOP. L.2 ACQUISITION SCHEDULE RFTOP issued 12/15/2014 Offeror Questions due 12/23/2014 Proposals due 01/26/2015 Performance begins 02/18/2015 L.3 GENERAL INSTRUCTIONS TO OFFERORS a) RFTOP Instructions: If an Offeror does not follow the instructions set forth herein, the Offeror’s proposal may be eliminated from further consideration or the proposal may be downgraded and not receive full or partial credit under the applicable evaluation criteria. b) Accurate and Complete Information: Offerors must set forth full, accurate and complete information as required by this RFTOP. The penalty for making false statements to the Government is prescribed in 18 U.S.C. 1001. c) Offer Acceptability: The Government may determine an offer to be unacceptable if the offer does not comply with all of the terms and conditions of the RFTOP. d) Proposal Preparation Costs: The U.S. Government will not pay for any proposal preparation costs. L.4 DELIVERY INSTRUCTIONS (a) Submission, Marking and Copies Proposals shall be submitted in two separate parts (a) technical proposal and (b) cost proposal in accordance with the structural format set forth below. Do not include these two parts in the same document. The Offeror must submit the proposal electronically - internet email with up to 5MB per email compatible with Microsoft Office (MS WORD or Excel) in a MS Windows environment. Multiple emails may be sent to accommodate the proposal size and content, but each must contain very clear identification of the attachment and instructions for assembling the proposal. Offerors may also send an Adobe Acrobat portable document format (.pdf) for electronic submission; however, zipped files attachments are not allowed. Any budget spreadsheets should be sent in unlocked Excel format (formula shown). The subject line for every such email must include the following “SOL-674-15-000007– BOTSWANA PROJECTS IMPACT EVALUATION” (b) Closing Date and Time All proposals in response to this RFTOP shall be due at the following address, no later than the Time and Date indicated on the cover page to this RFTOP. (c) Address Offerors must submit their proposals electronically via email to Lauda Mthembu at lmthembu@usaid.gov and proposals@usaid.gov RFTOP No. SOL-674-15-000007 Botswana Performance Evaluation Page 31 of 43 L.5 INSTRUCTIONS FOR THE PREPARATION OF THE TECHNICAL PROPOSAL A. Technical Proposal The technical proposal must be concise, and complete. The proposal will demonstrate the Offeror’s expertise with respect to achieving the evaluation program results. (1) General The Technical Proposal in response to this solicitation should address the technical evaluation factors listed in Section M. The technical proposal shall consist of the technical proposal itself and an annex. These together will constitute the Offeror’s technical proposal. The proposal must be well organized, complete, clear, and succinctly presented. (2) Page Limitation The technical proposal itself shall not exceed twenty-five (25) pages (page limit does not include, cover page, dividers, table of contents, or annex). Any additional pages will not be evaluated. Proposals shall be written in English and typed on standard 8 1/2" x 11" paper (210mm by 297mm paper) or A4 paper, single spaced, 12 characters per inch with each page numbered consecutively. Font size of less than 12 characters per inch may be used only in tables, charts and footnotes, and abuse of such exceptions may be a basis for a reduction in the evaluation score of any such Offeror, at the Contracting Officer’s discretion. Margin should normally be 1”. (3) Organization The technical proposal should be organized into the following sections as follows: Table of Contents Cover Page Executive Summary Technical Proposal Body 1. Technical Approach 2. Personnel Annex (4) Introductory Materials As stated above, a cover page and executive summary are required at the beginning of the technical proposals. The Offeror may also want to include other introductory materials such as a table of contents at the beginning of the technical proposal. The Cover Page should be the first page of the proposal and include the names of the organization proposed as the prime partner involved in the proposed evaluation. Major sub-contractors (those with over 20% of the total estimated amount) if included should also be listed separately. The Cover Page should include the name of the proposed Task Order Program Director (or other principal point of contact as determined by Offeror) and the name of a contact person (if different from the Program Director) for the prime Offeror and her/his contact information. The Executive Summary should summarize the key elements of the Offeror’s technical approach, personnel plan, implementation plan, expected results. The Executive Summary should be no more than two (2) pages long. (5) Technical Proposal Body (a) Technical Approach: Technical Approach part of Offerors’ proposal must include the following: 1. Specific methodology and techniques used to gather information and answer the evaluation questions. RFTOP No. SOL-674-15-000007 Botswana Performance Evaluation Page 32 of 43 2. A level of effort (LOE) matrix and timeline that illustrate key tasks and milestones for the evaluations. The timeline should show a realistic schedule that lists specific activities that will occur each week that the evaluation research is conducted. It should also specify when the final evaluation report will be submitted to USAID. It is anticipated that the Task will be awarded on February 05, 2014. The contract start date should be no later than February 16, 2015. Offerors should use these dates when developing a timeline. Timelines should include an initial in-briefing with USAID and an oral presentation of the final evaluation (delivered prior to the submission of the written evaluation). The latest possible date for receiving proposals is November 10, 2014. Proposals will be evaluated on the basis of the evaluation criteria in Section M. (b) Personnel: The contractor will identify at minimum a core evaluation team composed of a team leader, three experts, and a logistics/administrative consultant, most of who should be local Batswana hires. Please review Section M for a detailed list of requirements for each evaluation team member. USAID strongly recommends that the evaluation team include the part-time services of a qualified statistician. All attempts will be made for the team to be comprised of a representative number of male and female members. Core team members should include: 1. Team Leader/Senior Evaluation Specialist. The Team Leader will take specific responsibility for assessing and analyzing the organization‘s progress towards targets, factors for such performance, benefits/impact of the strategies, and compare with other possible options. S/he will also suggest ways of improving the present performance, if any. S/he will provide leadership for the team, finalize the evaluation design, coordinate activities, arrange periodic meetings, consolidate individual input from team members, and coordinate the process of assembling the final findings and recommendations into a high quality document. S/he will write the final report. S/he will also lead the preparation and presentation of the key evaluation findings and recommendations to the USAID/SA team and other major partners. 2. Senior Technical Advisor for Organizational Capacity Qualifications: Senior Technical Advisor for Organizational Capacity should have a post-graduate degree in communications, public health or an applicable social science. S/he should have several years’ experience with organizational capacity building in Sub-Saharan Africa. S/he should be knowledgeable in program assessment and evaluation methodologies. S/he should have extensive experience, and demonstrate state-of-the-art knowledge, in conducting evaluations/assessments of activities that aim to build organizational capacity. Roles and responsibilities: The Senior Technical Advisor for Organizational Capacity will assist the Team Leader in developing the evaluation methodology, creating the data collection tools, conducting interviews, analyzing the data, and writing the report. S/he will pay special attention to the organizational capacity aspects of the evaluation, and will take the lead in creating or identifying tools for the evaluators’ own assessment of the local organizations’ capacity. S/he will take part in meetings with USAID and other stakeholders, both in the planning phase, but also in the dissemination of findings. As a lesson learned from previous evaluations, the evaluation needs to be carried out in a participatory fashion, forming a team that, in various places and times, includes a range of USAID staff, GOB staff, and other stakeholders, as feasible. As a lesson learned from previous evaluations, the evaluation needs to be carried out in a participatory RFTOP No. SOL-674-15-000007 Botswana Performance Evaluation Page 33 of 43 fashion, forming a team that, in various places and times, includes a range of USAID staff, Government of Botswana staff, and other stakeholders. Additionally, Offeror’s proposed evaluation team members should provide a signed statement attesting to a lack of conflict of interest or describing an existing or potential conflict of interest relative to the program being evaluated that could lead reasonable third parties to conclude that the evaluator or evaluation team member is not able to maintain independence and, thus, is not capable of exercising objective and impartial judgment on all issues associated with conducting and reporting the work. Real or potential conflicts of interest may include, but are not limited to: 1. Immediate family or close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant/ material though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant/material though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. (6) Technical Proposal Annex The technical proposal annex shall contain position descriptions, CVs/resumes, letters of commitment from personnel or partners, letters of support, experience/performance references and any other supporting documentation requested by this RFTOP. Each CV/resume must not exceed 3 pages in length and must address specific Personnel requirements listed in Section M. B. Cost Proposal (1) Firm, Fixed Price Offerors must complete Section B.3 of this RFTOP, entitled “Firm, Fixed Price” and submit (as a separate section) with its Technical Proposal. A firm-fixed-price contract provides for a price that is not subject to any adjustment on the basis of the Contractor’s cost experience in performing the contract. The Firm, Fixed Price includes all Contractor’s costs to perform the work and includes Contractor’s fee (profit). This contract type places upon the Contractor maximum risk and full responsibility for all costs and resulting profit or loss. It provides maximum incentive for the Contractor to control costs and perform effectively and imposes a minimum administrative burden upon the contracting parties. (2) Costs Offerors must also submit a cost proposal that includes the following information that demonstrates how costs were calculated to arrive at the Firm, Fixed Price. Failure to include all information, or to organize the proposal in the manner prescribed, may result in rejection of the proposal as being unacceptable. The following guidance is provided with respect to the organization of the cost proposal. Line Item Organization: In order to undertake a meaningful analysis of price, Offerors shall use the following standard line items and cost elements organized as presented below. 1. Direct Costs RFTOP No. SOL-674-15-000007 Botswana Performance Evaluation Page 34 of 43 Labor Costs 1. Offeror’s individual labor category (i.e.Team Leader, Senior Technical Advisor, etc.) times the Level of Effort to arrive at individual labor category’s cost. Such costs shall be all inclusive and include salary and wages, fringe benefits and labor overhead. 2. Consultants (if any) - Consultant’s title and his/her fixed daily rate times the Level of Effort to arrive individual consultant’s cost. Travel & Allowances Travel, Transportation, Per Diem and Miscellaneous (Visas, Inoculations, travel allowances, etc. as one line item), Other Direct Costs (if any) 2. Fixed Fee (Profit) 3. Total Firm, Fixed Price END OF SECTION L RFTOP No. SOL-674-15-000007 Botswana Performance Evaluation Page 35 of 43 SECTION M – EVALUATION FACTORS FOR AWARD M.1 GENERAL INFORMATION (a) The Government may award a contract without discussions with Offerors in accordance with FAR 52.215-1. (b) The Government intends to evaluate Offerors in accordance with Section M of this RFTOP and make contract award to the responsible Offeror whose proposal represents the best value to the U.S. Government. (c) The submitted technical information will be evaluated by a technical evaluation committee using the technical criteria shown below. The evaluation committee may include industry experts who are not employees of the Federal Government. When evaluating the competing Offerors, the Government will consider the written qualifications/capability information provided by the Offerors, and any other information obtained by the Government through its own research. (d) For overall evaluation purposes, all evaluation factors other than cost or price, when combined, are significantly more important than cost or price. M.2 TECHNICAL PROPOSAL EVALUATION CRITERIA The following criteria will be used by the technical evaluation committee (TEC) to evaluate the proposals. The TEC members will assign values totalling 100 points to score the various components of the proposal set forth below. The specific evaluation criteria are as follows: Technical Evaluation Criteria Weight A. Technical Approach 50 points B. Personnel 50 points Total 100 points Technical Approach (50 Points): Evaluation of how the proposed technical approach is consistent with and supports the project objectives and evaluation questions as outlined in this RFTOP. The evaluation will include an assessment of the following: 1) Methodologies and Techniques (35 points): • Understanding of local capacity building measurement and evaluation methods and challenges/limitations to these methods • Proposed design/methods to answer evaluation questions is sound and feasible and likely to produce robust evidence for answering the evaluation questions Balance between efficiency and rigor 2) LOE matrix and timeline which illustrate key tasks and milestones for the evaluations (15 points) Personnel (50 Points): The personnel factor evaluates the extent to which the qualifications, skills and experience of proposed personnel meet or exceed those required below. Evaluation includes the overall staffing plan and approach, as well as the capabilities of specific key personnel. The following considerations will be evaluated under this factor. These considerations will not be scored separately as individual elements. They are included to RFTOP No. SOL-674-15-000007 Botswana Performance Evaluation Page 36 of 43 provide potential Offerors with additional information regarding this evaluation criterion and to help Offerors prepare their proposals: • Diversity of experience amongst team members • Relevant educational credentials of the team “Personnel” must include two key personnel: 1) Team Leader/Senior Evaluation Specialist , and 2) Senior Technical Advisor for Organizational Capacity. Required (“must”/”will”) and recommended (“should”) qualifications, skills, and experience of proposed personnel are as follows: Team Leader/Senior Evaluation Specialist (20 points): Team Leader/Senior Evaluation Specialist should have a post-graduate degree in public health or social sciences. S/he should have extensive experience in conducting mixed methods (combining quantitative and qualitative) evaluations in Sub-Saharan Africa and strong familiarity with process/performance evaluation methods and HIV/AIDS activities. Excellent oral and written skills in English are required. The Team Leader should also have experience in leading evaluation teams and preparing high quality documents. Roles and responsibilities: The Team Leader will take specific responsibility for assessing and analyzing the organizations’ progress towards targets, factors underlying their performance, benefits of the strategies, and compare with other possible options. S/he will also suggest ways of improving the present performance, as needed. S/he will provide leadership for the team, finalize the evaluation design, coordinate activities, arrange periodic meetings, consolidate individual input from team members, and coordinate the process of assembling the final findings and recommendations into a high quality document. S/he will write the final report. S/he will also lead the preparation and presentation of the key evaluation findings and recommendations to the USAID/Botswana team and other major partners. Senior Technical Advisor for Organizational Capacity (10 points): Qualifications: Senior Technical Advisor for Organizational Capacity should have a post-graduate degree in communications, public health or an applicable social science. S/he should have several years’ experience with organizational capacity building in Sub-Saharan Africa. S/he should be knowledgeable in program assessment and evaluation methodologies. S/he should have extensive experience, and demonstrate state-of-the-art knowledge, in conducting evaluations/assessments of activities that aim to build organizational capacity. Roles and responsibilities: The Senior Technical Advisor for Organizational Capacity will assist the Team Leader in developing the evaluation methodology, creating the data collection tools, conducting interviews, analyzing the data, and writing the report. S/he will pay special attention to the organizational capacity aspects of the evaluation, and will take the lead in creating or identifying tools for the evaluators’ own assessment of the local organizations’ capacity. S/he will take part in meetings with USAID and other stakeholders, both in the planning phase, but also in the dissemination of findings. Additional Personnel (20 points): Additional personnel will be evaluated based on the following: • Documented appropriateness of proposed staff and organization chart for technical approach proposed RFTOP No. SOL-674-15-000007 Botswana Performance Evaluation Page 37 of 43 • Qualifications and experience of the proposed key personnel vis-à-vis the positions for which the Offeror is proposing them. Demonstrate how these qualifications fit the function of their position particularly with regard to your technical approach • Established lines of authority and communications internal and external to the project M.3 PRICE/COST EVALUATION Price/Cost will not be scored but will be used for best value determination with Technical score being significantly more important than cost. USAID will evaluate the Offeror’s proposed price for completeness, reasonableness, and realism. USAID will evaluate to determine whether the estimated proposed costs elements are realistic for the work to be performed, reflect a clear understanding of the requirements, and are consistent with the unique methods of performance described in the Offeror’s technical proposal. M.4 DETERMINATION OF CONTRACT AWARD In accordance with FAR 52.215-1(f), the Government intends to award a Contract from this solicitation to the responsible Offeror whose proposal represents the best value after evaluation in accordance with the factors and sub-factors as set forth in this solicitation. This procurement also utilizes the trade-off process set forth in FAR 15.101-1. If the Contracting Officer determines that competing technical proposals are essentially equal, price may become the determining factor in source selection. Conversely, if the Contracting Officer determines that competing price proposals are essentially equal, technical factors may become the determining factor in source selection. Further, the Contracting Officer may award to a higher priced Offeror if a determination is made that the higher technical evaluation of that Offeror merits the additional cost/price. END OF SECTION M Appendix 1. Results Framework for USAID/Botswana DO: Strengthened Response to HIV/AIDS and other Health Services in Botswana IR 2.1 Increased management and technical capacity of the Batswana health workforce Intermediate Results Critical Assumptions or Risks to Program:  Sufficient yearly funding to support and implement Botswana Health HIV/AIDS and other health care services  GOB will be able to absorb and fund results  GOB/NGO policies will support efforts. llustrative Activity Types • Journey of Life • Go Girls! Community mobilization approach • Child Protection Committees/DMSAC • Community Conversations • Livelihood opportunities • Community Health Care workers supported • Comprehensive program for MARP  Transformation of the Central Medical Stores;  Strengthening DQAs and Data Flow of M&E systems  Mapping of PEPFAR activities and results by District;  Improve coordination between DAC, DMSACs, DHMT and NGOs at district level  Increase resources at district level through PPPs;  Better use of ICT ;  Costing to increase efficiency  Gender analysis training curriculum development and implementation;  Expansion of GBV services;  Increased GBV awareness;  Logistical Support to DSS and Women’s Affairs Department; Sub-Intermediate Results Objective 1 Mobilized Communities to support & provide HIV/AIDS, OVC & Health related services & practices Objective 2 Strengthened GOB and civil society capacity to support HIV/AIDS response in Botswana Objective 3 Strengthened GOB Policies and programming especially for Women and Children IR 1.2 Strengthened ref. networks to increase uptake of Prevention, Care and Treatment services IR 1.3 Expanded resources to support community efforts IR 2.2 Strengthened Botswana HIV/AIDS M&E Systems IR 2.3 Improved MOH Supply Chain Management System Cross cutting strategies: Gender –Focused Programs; Country ownership and Investment in Country –Led Plans; Policy-Informing Research; Strategic Coordination and Integration; Synergies between IRs, Partners, and Implementers IR 3.1 Improved implementat ion of the Children’s Act IR 3.2 Increased mainstreaming of gender in programs IR 3.4 Improved implementati on of the Domestic Violence Act Country Context:  Botswana is a sparsely populated country of 2.1 million people.  HIV adult prevalence rate of 25%.  In 2009 an estimated 350,000 Batswana were living with HIV with nearly 19,000 new infections.  The TB/HIV co-infection rate is 68% and an estimated 40% of deaths among persons living with HIV are due to TB. IR 2.4 Strengthened business systems for NGOs and National Networks IR 1.1 Community structure strengthened /leaders motivated RFTOP No. SOL-674-15-000007 Botswana Performance Evaluation Page 39 of 43 Appendix 2. Rough Map of Maatla’s districts RFTOP No. SOL-674-15-000007 Botswana Performance Evaluation Page 40 of 43 Appendix 3. Map of Tsela Kgopo sites RFTOP No. SOL-674-15-000007 Botswana Performance Evaluation Page 41 of 43 Appendix 4. Matrix of proposed methods by evaluation question Evaluation Questions Illustrative indicators Data Source/ Collection Methods Sampling / Selection Criteria Data Analysis Method How likely is each activity to achieve their desired end results, as defined in each activity’s results framework? (Three results for Maatla; four results for Tsela Kgopo.) What modifications, if any, to the strategic approach and theory of change have potential to substantially increase program effectiveness? To what extent has each activity been implemented according to activity design? Where there have been deviations or shortfalls, what are the causes? To what extent has USAID fulfilled its role in the activities by providing adequate funding, responsive managerial oversight and technical support, and assistance in solving problems in implementation as they arise? What actionable, prioritized lessons have been learned for the design of potential follow-on activities? I.e., what should continue and what should change from the current activities in a potential future activity? Has implementation so far been successful enough to warrant an impact evaluation of each activity? If not, why not? RFTOP No. SOL-674-15-000007 Botswana Performance Evaluation Page 42 of 43 Appendix 5. Maatla’s Organizational and Technical Capacity Assessment Tools Maatla tools for assessing organizational capacity, technical capacity, and district capacity OCA - 2013 clean copy!.xlsx Technical Assessment Tool TAT FINAL DISTRICT ASSESSMENT TOOL 2 Tsela Kgopo tools for technical visit assessments TA Short Visit Monitoring Tool A.doc TA Short Visit Monitoring Tool A.doc TA assessment and findings Presentation SIMS tool and SOP SIMS Community Site Visit SOP_for use_9.2 SIMS Community Tool_for use_9.25.do RFTOP No. SOL-674-15-000007 Botswana Performance Evaluation Page 43 of 43 Appendix 6. Botswana Institutional Review Board Schedule HRDC SCHEDULE FOR 2014-2015 DATE MONTH & YEAR TIME & VENUE 15 May 2014 0800 hrs 17 July 2014 0800 hrs 04 September 2014 0800 hrs 14 November 2014 0800 hrs 22 January 2015 0800 hrs 12 March 2015 0800 hrs Notes: • All submissions for new applications should be made at least five (5) weeks before the scheduled meetings. • Continue review application should be made at least three (3) months before the permit expiry date. • Amendments and close-out applications can be made anytime during the course of the study. • The Principal Investigators should make sure that all documents are complete before submissions and all other required documents are attached. 0 APPENDIX IV: MAATLA RESULTS FRAMEWORK Critical Assumptions 1. Viable government entities at the district level for adequate CSO coordination 2. Resources will be available from other sources to sustain the civil society sector 3. There will be willingness and interest to assist BONASO by all stakeholders Project Goal: To significantly and sustainably strengthen the capacity of the civil society sector in Botswana to support HIV and AIDS and related health service delivery Result 1: Strengthened technical competence and organizational capacity of national NGOs and networks to support field offices, affiliates and member organizations with expanded and strengthened delivery of strategic and high quality HIV and AIDS services. Result 1: A model developed and implemented to strengthen district level systems to provide and coordinate HIV and AIDS services in hard-to-reach areas. Result 3: Strengthened institution(s) able to provide sustainable capacity building to civil society Summary of Indicators for Result 1 • OD success competencies for National NGOs/Networks • Technical competencies for HIV and AIDS service delivery by National NGOs/Networks • Gender • Direct indicators of training • National (indirect) indicators for service delivery by National NGO partners in non￾target districts Summary of Indicators for Result 2 • OD success competencies for district-level CSOs and coalitions • Technical competencies for HIV and AIDS service delivery by district-level CSOs • Direct indicators for HIV and AIDS service delivery by Maatla NGO partners in target districts • Gender • District coordination • National (indirect) indicators for service delivery by non-Maatla partners in target districts Summary of Indicators for Result 3 • BONASO and BOCONGO capacity strengthening for members and affiliates – expertise and leadership in the field is also demonstrated through Result 1 indicators • BONASO and BOCONGO sustainability indicators 0 APPENDIX V: DETAILED DESCRIPTION OF METHODOLOGY AND EVALUATION MATRIX Annex V: Methodology for evaluation of Maatla Project, for USAID 1 Appendix V: Evaluation methodology Contents 1 Evaluation matrix ................................................................................................................................................................................................... 1 2 Data collection ....................................................................................................................................................................................................... 7 2.1 Geographical information.............................................................................................................................................................................. 7 2.2 Field Work Set Up .......................................................................................................................................................................................... 7 2.3 Conducting Field Work................................................................................................................................................................................... 7 3 Data analysis........................................................................................................................................................................................................ 10 1.1.1 Qualitative data analysis ..................................................................................................................................................................... 10 4 Southern Hemisphere’s model ............................................................................................................................................................................ 12 4.1 An overview of the Southern Hemisphere model ....................................................................................................................................... 12 5 Evaluation model assumptions............................................................................................................................................................................ 14 6 Southern Hemisphere’s quality assurance mechanisms ..................................................................................................................................... 15 1 Evaluation matrix Evaluation question Indicators Data source / Collection methods Sampling/ Selection Criteria Data analysis method How likely is each activity to achieve their desired end results, as defined in each activity’s results framework? Maatla • % CSO meeting minimum standards for leadership, governance, Semi-structured Interviews with a sample of organisations including both core sub-partners and sub-partners, as well • Representatives were purposively selected from these organisations, e.g. implementing staff; • Qualitative analysis using NVIVO10 qualitative software; thematic analysis – identifying the Annex V: Methodology for evaluation of Maatla Project, for USAID 2 management of finances, human resources, inter￾organizational linkages, infrastructure, etc. • Mean improvement in Organizational Capacity Assessment (OCA) scores • % CSOs implementing HIV/AIDS prevention, care, and treatment programs according to minimum standards. • Mean improvement in Technical Capacity Assessment (TCA) scores per domain • # of successful funding proposals by BONASO and BOCONGO • # of accredited training/capacity building materials developed by BONASO or as Government officials. (see sample below) Document review of Maatla programme documents. OCA / OCA lite spreadsheets from Maatla TCA spreadsheets from Maatla FHI360 MIS II FHI360 PEPFAR indicator reporting directors salient themes from these interviews • Scoring using the rating scale in the OCA and TCA rolling up, averaging using excel. Annex V: Methodology for evaluation of Maatla Project, for USAID 3 BOCONGO • # of member organizations or affiliates trained or mentored by BONASO or BOCONGO • % increase in non￾USG funding for BOCONGO and BONASO • Strengths of the capacity building process • Challenges of the capacity building process What modifications, if any, to the strategic approach and theory of change have potential to substantially increase program effectiveness? Indicators are not necessarily relevant here. Rather the design of the programme will be reviewed for its alignment to priority issues in the local and international context. Semi-structured Interviews with: • USAID Head Office • FHI360 • Government stakeholders • Core sub-partners • Sub-partners • DMSAC members (District Multi￾Sectoral Aids Committee) Representatives will be purposively selected from these organisations, such as Programme Managers and M&E representatives. Thematic analysis – identifying the salient themes from these interviews Annex V: Methodology for evaluation of Maatla Project, for USAID 4 Document review of internal programme records. Focus groups with: • Community based volunteers in the sample districts To what extent has each activity been implemented according to activity design? Where there have been deviations or shortfalls, what are the causes? Extent to which activities have been implemented according to plan, and to desired quality Strengths to implementation Barriers / challenges for implementation • SSIs with Maatla’s core sub-partners and sub-partners • Focus groups with community based volunteers Document review of internal programme records. An extensive analysis was conducted comparing the annual work-plans to the annual reports. Representatives will be purposively selected from these organisations, including: Director, Programme management, Representative of the Board of Directors, Finance/Administration Management, implementing staff Thematic analysis – identifying the salient themes from these interviews using NVIVO 10 qualitative software. Annex V: Methodology for evaluation of Maatla Project, for USAID 5 To what extent has USAID fulfilled its role in the activities by providing adequate funding, responsive managerial oversight and technical support, and assistance in solving problems in implementation as they arise? Defined programme activities and output level indicators have not been provided, however some general questions for exploration may include: • Whether key activities/outputs have been delivered within time and with sufficient quality • Strengths and challenges in implementation. These may be related to the context, implementation or design of the programme. • Relevance of support Semi-structured key Informant Interviews (KII) with: • USAID Head Office (national) • FHI360 (national) • Government officials from NACA, Ministry of Health Representatives will be purposively selected from these organisations – e.g. Programme managers Thematic analysis – identifying the salient themes from these interviews What actionable, prioritized lessons have been learned for the design of potential follow-on activities? I.e., what should continue and what should change from the current activities in a potential future activity? Defined programme activities and output level indicators for Maatla and Tsela Kgopo are not provided. When these are available, they will be incorporated into the methodology. semi-structured Key Informant interviews with: • Maatla’s key organisations: BONASO, BONEPWA, BOCAIP and BOCONGO, WUSC, NCONGO, FHI360 • Government officials including Purposive sampling, addressing the programmes’ relevant target groups (adults, youth, ECD centre staff, etc) Thematic analysis – identifying the salient themes from these interviews Annex V: Methodology for evaluation of Maatla Project, for USAID 6 members of the DMSAC, National ministries and local government. Focus groups with: • CBO site staff • Beneficiaries Annex V: Methodology for evaluation of Maatla Project, for USAID 7 2 Data collection 2.1 Geographical information The Maatla fieldwork took place in Gaborone, and in the Okavango and Ngami districts. The districts were selected following discussion with USAID and FHI 360. The Maatla programme is also conducted in Chobe and Ghanzi districts. 2.2 Field Work Set Up Ethical clearance was obtained for the study from the Ministry of Health Chief Research Officer, Health Research Division, Department of Health Policy Development Monitoring and Evaluation. Letters of support for the fieldwork were also obtained from the key Government ministries to encourage participation at the district level. USAID was instrumental in assisting with these process. The sample design and identification of people for inclusion in the evaluation were facilitated by FHI360 and the implementing partners (BOCAIP, BONASO, BOCONGO and BONEPWA), who were all extremely helpful in setting up the fieldwork. The initial arrangements for interviews were made from the Southern Hemisphere office in Cape Town, but our Botswana colleagues followed up with final confirmations, as suggested by USAID in Botswana. 2.3 Conducting Field Work The fieldwork was conducted between 17 August and 4 September 2015, both in Botswana and via telephone / skype. As can be seen from the table below the fieldwork was 87.5% completed in total, the biggest challenge was identifying sufficient respondents who had been involved in the Maatla project at the district level. This speaks to one of the key themes arising; the small size of the CSO sector in the districts, and the limited staff compliment and resources that they had to implement the project. Please see the challenges section below for more details. Table 1 summary table SUMMARY TABLE: SSI Planned SSI Actual FG Planned FG actual OCA/TCA planned OCA / TCA actual National 23 23 4 3.5 District 33 28 5 5 11 8 Annex V: Methodology for evaluation of Maatla Project, for USAID 8 TOTAL 56 51 5 5 15 11.5 National: USAID , FHI 360, Government, Implementing partners / member organisations Stakeholder Focus area (TCA and OCA) Method Sample USAID SSI x 3 AOR, Technical Support HTC, M&E FHI 360 SSI x 4 Deputy Director, Senior Technical officer x 2 ( STO HIV prevention , Gender and OVC; STO HTC, TB and Care ), Chief of Party, STO Monitoring and Evaluation World University Services of Canada SSI x 2 Country Director, organisational development manager National Government SSI x 5 (6) NACA x 2 MoH x 1 DSP x 1 Gender Affairs x 1 BOCAIP TCA SSI x 3 Director, programme manager, M&E officer BONEPWA TCA SSI x 3 Director, programme manager, finance & admin officer BONASO OCA (half complete as of today) SSI x 1 Director BOCONGO OCA SSI x 2 Director, Organisational development manager/ capacity building manager District network organisation offices Annex V: Methodology for evaluation of Maatla Project, for USAID 9 Stakeholder Focus area (TCA and OCA) Method Sample BOCAIP: Gumare / Okavango – counselling centre TCA and OCA lite SSI x 1 centre supervisor (also known as, District Coordinator) finance / admin support staff BOCAIP (Maun) Counselling Centre SSI x 2 Centre supervisor, admin officer BONEPWA+: Ngami (Maun) centre of Excellence TCA and OCA lite SSI x 1 District co-ordinator BONEPWA+: Gumare Centre of Excellence sub-office SSI x 1 Assistant District Coordinator NCONGO (Maun) OCA lite SSI x 2 Acting Director /organisational development manager, Director FHI 360 Maun SSI x 2 District coordinator, District M&E officer Member organisations at district or village level Stakeholder Focus area (TCA and OCA) Method Sample BONEPWA+ Maun volunteers in Maun FG x 2 volunteers group in Maun BONEPWA+ Ngami District Village - Sehithwa SSI x 2 volunteers at Sehithwa BOCAIP Okavango District SSI x 2 HIV Counsellor BOCAIP Okavango District FG x 3 Focus Group of volunteers from within Gumare, and from surrounding villages Annex V: Methodology for evaluation of Maatla Project, for USAID 10 BOCONGO member 1: Women Against Rape SSI x 2 Director / Maun NCONGO member 1: Sekgele Training TCA, SSI x 1 Director / Maun NCONGO member 2: Community development Trust SSI x 1 NCONGO member 2: BOFWA OCA lite Director / Gumare NCONGO member 3: Thuso Rehabilitation Centre OCA lite Director / Maun Other interviews at district or village level Stakeholder Method Sample District Aids Coordinator (supervisor) SSI x 2 1 x Gumare, 1 x Maun Other DMSAC members (Ministry of Local Government and Rural Development; DHMT) SSI x 3 2 x Gumare, 1 x Maun DHMT coordinators SSI x 2 1 x Gumare 1 x Maun Social and community development (S&CD) representatives / OVC / CDW SSI x 4 2 x Gumare, 2 x Maun 3 Data analysis 1.1.1 Qualitative data analysis In order to ensure rigor in the analysis, NVIV0 10 software will be used to analyse the qualitative data. NVIV0 10 is data analysis software which assists handling very rich information. A summary of key themes, lessons learned and recommendations that arise from the study will be presented in the generated coding reports. The software removes many of the manual tasks associated with analysis Annex V: Methodology for evaluation of Maatla Project, for USAID 11 like classifying, sorting and arranging information, so the researcher has more time to explore trends, build and test theories and ultimately arrive at answers to questions. The diagram below summarises the process that will be used in the thematic analysis of qualitative data. Codes and code book developed Data analysis team briefed to ensure inter-coder reliability Qualitative data categorised according to each relevant code Coding reports generated to summarise data for each indicator/code Data is analysed: description; comparison with quantitative & documents; inferences about the outcomes and impact of program. Report (qualitative and quantitative data) presented through narrative report and graphics Annex V: Methodology for evaluation of Maatla Project, for USAID 12 4 Southern Hemisphere’s model 4.1 An overview of the Southern Hemisphere model The Southern Hemisphere model for undertaking applied research and outcome / impact evaluations might be described as a naturalistic-pragmatic model:- naturalistic, in that the consultancy focuses upon grounded inquiry or study within the ‘real-world’ as opposed to experimental or manipulated settings, whilst acknowledging that such a reality is comprised of multiple perspectives, which are both divergent and inter-related; and pragmatic in that the evaluation aims and objectives – plus resources available for the study – will influence methodological decisions. The model is also described as being participatory; that is, it is underpinned by broad consultation and engagement with a variety of stakeholders affiliated to the programme or project under review. This approach is based upon three key considerations, endorsed by all four of the consultancy’s partners; namely: a) The ethical imperative of the evaluator to ensure that all relevant parties are offered the opportunity to give their input; that is, to ensure equality of representation amongst all relevant stakeholders, including programme management, programme implementers, target groups, and beneficiaries. b) Maximum involvement of the client organisation to ensure that its evaluation requirements, constraints and information needs, are clarified and addressed; and c) Empowerment of the client organisation, which is offered insight into evaluation processes, methodologies and benefits, as well as the skills to act upon such knowledge in future evaluations. This is in keeping with the consultancy’s belief that programme evaluations offer a valuable means of facilitating reflection, learning and organisational development and, as a result, the client organisation is encouraged to view the evaluation process not as an accountability measure only, but as a means of improving upon its work and effectiveness1 . SH’s evaluation model is underpinned by an established process, which is divided into a number of stages or activities, which collectively form a coherent, linear process or ‘mode of operation’. These are outlined in the graphic below. 1 Such an approach is referred to by Bawden and Packham (cited in Patton 2002: 179) as ‘systemic praxis’. Annex V: Methodology for evaluation of Maatla Project, for USAID 13 Diagram 1: An outline of the Southern Hemisphere Evaluation Model Step One: Planning workshop hosted with the client organisation/s and other relevant stakeholders, invited at client organisation's discretion. Following this workshop, an inception report is produced, to confirm the way forward in the evaluation process. Step Two: Following a comprehensive document review, research instruments are compiled and sent to client organisation/s for members' input and approval. Step Three: Fieldwork is conducted. Step Four: Preparation and dissemination of a draft report, outlining research findings, conclusions and recommendations. Client organisation offered opportunity to review and comment. Step Five: Feedback and recommendations workshop hosted to present findings and recommendations. Client organisation - and other attending stakeholders' - feedback incorporated into final report. Dissemination of final report at client organisation's discretion. Annex V: Methodology for evaluation of Maatla Project, for USAID 14 5 Evaluation model assumptions The assumptions underpinning the SH evaluation model may be summarised as follows: Diagram 2: A summary of the assumptions underpinning the SH evaluation model •Reality is mediated through human constructions, which may be inter-related, shared or divergent. SH's work demonstrates a commitment to recording and explaining phenomena from the perspective of those being studied. •Hence, I would argue that SH's approach and model falls within the constructivist 'camp' of evaluation research. Ontological assumptions •The SH evaluation model is fallibilistic, in that the production of knowledge is attempted in as accurate or ‘truthful’ a manner as possible, whilst acknowledging that all findings are provisional and may be amended in the course of a study as new data comes to light. •A subjectivist position is also adopted, in that the world and the researcher are perceived as inter-dependent. Whilst neutrality is a guiding ideal, it is acknowledged that research can never be entirely ‘value-free’; that is, the subjective and interpretive nature of all research is acknowledged. Epistemological assumptions •The consultancy follows a pragmatic approach, in that methods are selected on the basis of their level of appropriateness to the evaluation’s aims, objectives and available resources. Methods are often mixed and are utilised flexibly. • Whilst there is no methodological hierarchy, the consultancy firmly supports the use of qualitative methods for the collection of rich and detailed data (for provision of the how and why input of a study). • A mainly inductive - as opposed to deductive - analytical process is used, coupled with an emergent or grounded theory approach - as opposed to the use of a priori theories. Methodological and Analytical assumptions Annex V: Methodology for evaluation of Maatla Project, for USAID 15 6 Southern Hemisphere’s quality assurance mechanisms The following table outlines SH’s quality assurance mechanisms. Annex V: Methodology for evaluation of Maatla Project, for USAID 16 Key Criteria Definition Quality Indicators Credibility (this corresponds with the positivist criterion of internal validity) Evaluation evidence is perceived by stakeholders both internal and external to the study to be trustworthy, plausible, useful, and relevant to the aims and objectives of the evaluation. 1.1 Evidence of member checks / reflections on research design, process, findings and recommendation (planning workshop, submission of draft report, feedback and recommendations workshop). 1.2 Evidence of investigator and/or data source and/or methodological triangulation. 1.3 Evidence of external review of the research process and findings. 1.4 Rationale and description of sample composition and selection (for example, characteristics of sample basis for inclusions and exclusions, sample size and how sample allowed comparative data and / o negative case analyses to be undertaken). 1.5 Client feedback indicates that the SH outcome and / or impact evaluation outputs are perceived as bein trustworthy, credible and believable. 1.6 Client feedback indicates that the SH outcome and / or impact evaluation outputs are perceived as bein relevant and useful, incorporating supportive evidence; that is, examples of client utilisation of evaluatio findings and recommendations. Transferability (this corresponds with the positivist criterion of generalisability external validity) The applicability or ‘fittingness’ and value of evaluation findings, conclusions and recommendations to other contexts or programmes of a similar nature. 2.1 Evidence of ‘thick’, detailed descriptions of research context to enable readers’ assessment o transferability (background section of report). 2.2 Evidence that client organisations regard SH evaluation outputs to be of value in / transferable to othe contexts or programmes of a similar nature. Dependability / Auditability (this corresponds with the positivist criterion of reliability) The consistency, coherence, logic and stability of the study process and data analysis over time and across researchers and methods. 1.1 Dependability audit: 3.1.1. Inclusion of detailed audit trail of data collection process and procedures. 3.1.2. Inclusion of detailed audit trail of data analysis process and procedures. 3.2. Documentation of any changes to evaluation design, including reasons and any implications thereof. 3.3. Documentation of any study limitations and their possible impact upon the evaluation. 3.4. Evidence that client organisations regard the SH evaluation outputs to be stable and replicable acros researchers and methods. Confirmability (this corresponds with the positivist criterion of objectivity) Findings are drawn from available data and are independent of the perceptions and possible biases of the evaluators; that is, they are deemed to be value-neutral and representative of all stakeholders’ 4.1 Confirmability audit: 4.1.1 Evidence of use of multiple fieldworkers /interviewers. 4.1.2 Indication that fieldworker training was conducted. 4.1.3 Evidence of the use of appropriate and unbiased questioning techniques in data collectio instruments. 4.1.4 Evidence that data collection instruments were piloted. 4.1.5 Indication that quality checks were conducted on submitted transcripts. Annex V: Methodology for evaluation of Maatla Project, for USAID 17 input. 4.1.6 Indication of the use of multiple coders for inter-coder cross-checks. 4.1.7 Use of peer review of coding and data analysis. 4.2 Use of a set of pre-defined and clearly stipulated criteria against which the programme /intervention wi be assessed. 4.3 Evidence that client organisations regard the evaluation process and outputs as being sufficiently neutra and inclusive of all stakeholders’ input. 1 APPENDIX VI: MAATLA FINAL INSTRUMENTS 2 APPENDIX VII: SCHEDULE OF CAPACITY ASSESSMENTS CONDUCTED OCAs Completed Organization Round 1 Round 2 Maatla partners did not conduct any OCA assessments in year 3 Round 3 BOCAIP Q3 of FY12 Q4 of FY13 Q2 of FY15 BONEPWA+ Q3 of FY12 Q4 of FY13 Q2 of FY15 BOCONGO Q3 of FY12 Q4 of FY13 Q1 of FY15 BONASO Q4 of FY12 Q4 of FY13 Q2 of FY15 TCAs Completed Organization Round 1 Round 2 Round 3 BOCAIP Q4 of FY12 Q4 of FY13 During Q1-2 of Year IV, BOCAIP was supported to conduct a technical capacity assessment, the report and prioritization for which will be completed in Q3. BONEPWA+ Q4 of FY12 Not completed. Moved to Q1 of FY15 Not completed. Scheduled for Q3. BOCONGO Not completed. Moved to Q1 of FY15 Not completed. Not completed. BOCONGO’s OCA results will inform its technical capacity building plans for HIV/AIDS and gender mainstreaming. BONASO Not completed. Moved to Q1 of FY15 Not completed. Not completed. Scheduled for Q3. 3 APPENDIX VIII: CAPACITY ASSESSMENT RESULTS Appendix VIII Capacity Assessment Results Methodology The strengths and weaknesses of implementing organisations in the Maatla project in various competency areas were evaluated using three standardised instruments. These instruments were developed by the lead implementing organisation for the Maatla project, FHI360. The instruments have been used internally as part of a routine self-assessment exercise by programme implementers. Through using the same instruments, sub-contracted implementing partners can be evaluated on the criteria aligned with performance areas identified for the Maatla project which they have been working toward improvement on. The instruments which were used are: 1. Organizational Capacity Assessment tool (OCA) 2. Organizational Capacity Assessment tool Lite (OCA Lite) 3. Technical Capacity Assessment Tool for National NGOs and Networks (TCA) OCA was design as a self-assessment tool for building organisational capacity. The OCA instrument has been adapted for the Maatla project. It looks at the following organisational competency domains: strategic leadership, organisational structures (governance), human resources, financial management, infrastructure, programme and service management, process management and inter-organisational linkages. OCA Lite is a more focussed and shorter version of OCA aimed at being simpler for organisations to implement. OCA Lite addresses the same domains with the exception of inter￾organisational linkages. TCA is the simplest and most focussed of the three instruments, focussing on four key domains: technical planning and design, staff capacity, linkages and networks and quality improvement and assurance. The following responses were collected in face-to-face interviews with key personnel at different implementing organisations in the following locations: Instrument Organisation Location Membership OCA Lite BOFWA Maun NCONGO OCA Lite Thuso Maun NCONGO OCA Lite BOCAIP Gumare BOCAIP OCA Lite BONEPWA Maun BONEPWA OCA Lite NCONGO Maun NCONGO TCA Sekgele Gumare NCONGO TCA BOCAIP Gumare BOCAIP TCA BONEPWA Maun BONEPWA TCA BOCAIP Gaborone BOCAIP TCA BONEPWA Gaborone BONEPWA OCAT BOCONGO Gaborone BOCONGO The above table includes one response to the OCA, five responses to the OCA Lite and five response to the TCA tool. Data Analysis Each instrument’s subsections are made up of evaluation points which are rated on the following scale: • NA = Not applicable (the standard is not applicable to the organization) • 0 = No there are no systems in place • 1 = Yes partially (there are systems in place but there is room for improvement) • 2 = Yes (there are systems in place and they can be verified) In the case that an evaluation point is not applicable to that organisation, the possible score for that subsection is adjusted downward accordingly. Subsections added up together are converted into a percentage score. In terms of the OCA and OCA Lite that percentage can be classified as follows: 1. Mature: 90%-100%, “The organisation has well developed and well functioning credible systems, adequate resources and viable programmes” 2. Expanding: 80-89%, “The organisation is able to rapidly respond to change and sustain itself due to its credible systems, adequate resources and viable programs” 3. Developing: 70-79%, “All basic organisational development, systems and processes are in place, select domains have ongoing weaknesses” 4. Beginning: 69% and below, “Organisational development domains, systems and processes are at minimum” Source: http://www.fhi360.org/sites/default/files/media/documents/Organisational_Capacity_Assessm ent.pdf Organizational Capacity Assessment Tool Lite Results Discussion When considering the result of the OCA Lite assessments it is important to note that in the case of BOCAIP, BOFWA, BONEPWA and NCONGO, one of several regional offices of each organisation was assessed (see the table above in the methodology section for a breakdown). Ratings at the level of regional office may not be a reflection of the status of the organisation as a whole. In contrast, Thuso is an organisation with a single branch. It is also worth noting here that all organisations assessed using the OCA Lite tool with the exception of BOCAIP are members of NCONGO’s capacity-building network of organisations. The overall results of the OCA Lite organisational assessment is presented in the bar graph that follows. Thuso (a member of BOCONGO) was the most highly rated organisation, scoring 92% overall, achieving the status, ‘mature’. In contrast, the organisation receiving the weakest rating was BOCAIP at 65% giving it the status, ‘developing’. NCONGO and BOFWA are both classified as ‘expanding’ organisations. BONEPWA’s score between 70 and 79% earns it the intermediate status, ‘developing’. Figure: OCA Lite overall ratings The scores for each organisation across the domains that make up this total score can be comparatively assessed in the radar chart that follows. Each corner of the heptagon in the chart represents a domain of the OCA Lite score, with scores ranging from their lowest in the centre of the graph at 46% to 100% at the edge of the heptagon. Figure: OCA Lite ratings by domain Looking at domain performance in the radar graph above, BOCAIP’s overall weak rating is most significantly influenced by a low infrastructure rating, followed by a weak strategic leadership rating and a comparatively poor service management score. In contrast, BONEPWA’s weakest domain ratings are infrastructure and process management. While Thuso’s overall score is greater, NCONGO’s performance is more even across domains. On the other hand, BOFWA’s domain ratings appear skewed strategic leadership, governance and finance while other domains receive intermediate ratings. Individual domain ratings will be explored in the comparative discussion of each domain that follows. Strategic Leadership This domain looks for evidence of high-level competencies, addressing the sub-areas strategic planning, resource mobilisation and organisation visibility. Strategic planning is to do with established purpose and adherence to that purpose. Resource mobilisation addresses capacity to accrue and maintain funding. Visibility looks for evidence that an organisation maintains a public face. Strategic leadership’s scores by organisation are presented in the graph that follows. Strategic leadership in the graph is the overall score for strategic planning, resource mobilisation and visibility. Figure: OCA Lite strategic leadership Compared to overall scores, strategic leadership is a weak point particularly for Thuso (92% vs 75%) and to minor extent, NCONGO (89% vs 84%). Resource mobilisation was a weak area for all organisations with the exception of BOFWA. With the exception of BOCAIP, all organisations appear to be doing fairly well in the area of strategic leadership. Weaker ratings for strategic leadership in general are mostly substantiated by undocumented policy and procedure. Good ratings on strategic leadership points are supported in these evaluations by examples and evidence of carrying out these functions. Most significantly, resource mobilisation is an area for BOCAIP in Gumare to improve on. Organisation Structure and Governance This domain has to do with foundational documentation and decision-making structures. Organisational structure and governance has no sub-sections, so only overall scores by organisation are presented in the graph that follows. Figure: OCA Lite organisation structure and governance BONEPWA and BOFWA stand out in organisational structure and governance by meeting all documentation criteria and have regular decision-making processes up to standard. In stark contrast, some the necessary expertise required of those those in decision-making structures was lacking for BOCAIP and several decision-making intervals were not maintained, hence a much lower rating. Organisations other than BOCAIP were generally able to substantiate the required registrations, decision-making competencies and intervals. Human Resources The human resources domain is made up two sub-sections: employee management and volunteer management. Generally, the competency areas inside the human resources domain have to do with policy and procedure being in place with respect to employees and volunteers. Adherence to practice standards and the existence of working human resource administrative procedures are also addressed in this domain. You can compare the human resource ratings of the organisations assessed by OCA Lite in the graph that follows. Figure: OCA Lite human resources The best performer in human resources was Thuso at 91% overall and the worst was BONEPWA at 65%. In the case of BONEPWA, several good, or common practice competencies have lapsed due resource constraints and lack of personnel to implement them. In the case of BOFWA who also didn’t score that well on employee management, the cycle of some human resource procedures had not been followed according to policy and in other cases, policies and procedures were still to be finalised. In general, score shortfalls for the human resources domain in most cases has to do with lack of adherence to a few administrative criteria. Financial Management The financial management domain includes the following competency sub-sections: financial planning and budgeting, financial management and accounting and record-keeping. Financial planning and budgeting assesses the existence of basic good practice and links between day to day financial decisions and the general fiscal situation of the organisation. Financial management is a checklist of specific procedures to do with different procurement areas. Accounting and record-keeping is a checklist of record-keeping and compliance procedures and related practice. See the graph at the end of this section to compare scores by organisation. Thuso, BOFWA and NCONGO all have exemplary ratings for financial management. While BOCAIP have low rating for financial planning and budgeting, this competency area is made up of only five items, two of which are not applicable to BOACIP. BOCAIP’s Gumare branch is significantly supported in the financial management domain as all accounting and record￾keeping is taken care of by BOCAIP’s head office (hence the absence of this score) who also assist in financial planning and budgeting, so this does not necessarily mean a poor outlook for BOCAIP’s Gumare branch. Similarly, with BONEPWA’s Maun branch, accounting and record-keeping are almost entirely taken care of by their head office but the received good rating on the few areas they take care of internally. The weak rating for financial management has to do with a lack of rigor and procedure, or tasks only partly being adhered. As in the case of BOCAIP, support with head office means this is not necessarily a sign of great problems. Figure: OCA Lite financial management Infrastructure The Infrastructure domain has no sub-sections and consists of a checklist of office infrastructural provisions (physical and electronic). Competencies include specifics like operational IT equipment and provisions that print material is properly filed. The infrastructure domain is generally an indicator that the office environment of the of is conducive for work. The overall ratings for BOCAIP and BONEPWA is low mostly because of partial satisfaction of criteria but there are also a few areas where criteria was not met at all (for example, neither have a data backup system). NCONGO’s infrastructural challenges result from partially met criteria that rather reflect inconvenience to office staff (possibly creating inefficiencies) rather than significant challenges (with the exception of the absence of fire safety procedures). BOFWA’s only notable shortcoming is the absence of a data backup system. See the graph that follows for a comparison of infrastructure ratings. Figure: OCA Lite infrastructure Programme and Service Management This domain is comprised of two sub-sections: programme planning and programming implementation and documentation. Programme planning is a checklist of good practice for programme planning and touches on criteria like incorporation of past experience and staff input. Programming implementation and documentation reflects on practical examples of the organisation’s ability implement programmes and improve future implementation based on the past. Thuso and NCONGO are exemplary with maximum ratings for programme and service management. BOCAIP and BOFWA comparative scores come from the fact that they do not meet with criteria that substantiate reflexivity in the programmes they deliver. BOCAIP also only partly satisfies related criteria in a domain which has only a few evaluation criteria. See the graph below to compare programme and service management ratings. Figure: OCA Lite programme and services management Process Management The process management domain is comprised of three sub-sections: monitoring & evaluation management, data collection and reporting and performance monitoring. Assessment of monitoring & evaluation management looks at the presence of policy, practice and responsible person fulfilling that role. Data collection and reporting examines the timeliness and good practice across the reporting process. Performance monitoring is a checklist of regular milestones, covering managerial oversight, reflexivity and regularity in reporting in general. NCONGO is the best performing organisation on process management at 94% and BONEPWA was the most weakly rated at 58% overall. The poor rating for BONEPWA’s Maun branch has to do with absence of monitoring & evaluation function and personnel and related reporting procedures being behind the schedule established in OCA Lite. BOFWA and BOCAIP received identical sub-scores and overall domain scores (69%) for process management. In both cases, there is a system in place but it misses full ratings do to partially satisfied criteria, substantiated by both organisations being a bit behind with paperwork and that they do not adhere to a schedule when it comes to process management procedures. Similarly, Thuso’s process management rating (81%) falls short on a few points to do regularity of related reporting and slightly non-optimal integration with partners, government and donors. Figure: OCA Lite process management OCA Lite Discussion The OCA Lite organisational assessments for this evaluation offer the ability to compare organisational performance by OCA Lite performance areas. Without a longitudinal dimension, how these organisations might be trying to improve performance on these criteria is not apparent. As above, the standout organisation on the positive side is definitely BOFWA. In contrast, the weakest rated organisation in this evaluation using the OCA Lite is BOCAIP’s Gumare branch. Organizational Capacity Assessment Tool Results Discussion BOCONGO was the sole organisation assessed using the OCA tool. The results of BOCONGO’s assessment will be discussed in comparison to the last round of internal assessment, earlier in 2015. OCA and OCA Lite are very similar with OCA simply having a slightly longer checklist of competencies per domain as well as one additional domain: inter-organisational linkages. Inter￾organisational linkages generally assess an organisation’s level of connection and participation in relevant resource sharing, integration and collaboration with other organisations with similar functions and objectives. The overall scores between internal (88%) and external (82%) evaluations are similar. A score sitting between 80 and 89% qualifies BOCONGO as an ‘expanding’ organisation which is “able to rapidly respond to change and sustain itself due to its credible systems, adequate resources and viable programs” which is a generally positive. However, notable differences exist between internal and external evaluation domain ratings for BOCONGO. These differences are illustrated in the radar graph that follows. Each point of the octagon in the radar graph represents one of OCA’s domains with scores at their lowest at the centre of the graph (50%) and at their highest in the octagon’s corners. The three strongest domain discrepancies between internal and external OCA assessments from in order from largest are in strategic leadership (81% internal vs 55% external), inter￾organisational linkages (95% internal vs 69% external) and process management (90% internal vs 69% external). Of all notable discrepancies the external evaluation’s rating was lower in all cases except in the case of human resources where the external evaluation rated BOCONGO more highly on human resource competencies (73% internal vs 89% external). These differences will be discussed in more detail in the subsections to follow. Figure: BOCONGO internal & external domain ratings compared Comment [AC1]: We need the date confirmed. Strategic leadership BOCONGO was rated lower in the external assessment on each strategic leadership domain sub-section by a comparable proportion (26% lower). Resource mobilisation especially has many individual competencies rated 0 externally where the internal assessment had rated these with 2 as the rating. Strategic leadership is one of the domains with starker contrast between evaluations. Figure: Internal vs external BOCONGO OCA strategic leadership ratings Organisational Structure and governance BOCONGO’s organisational structure and governance scores from both evaluations were both excellent at 97% with no meaningful differences in scoring. BOCONGO is a well-established and well-governed organisation by all appearances. Human resources Human resources ratings is the one domain where the external evaluation rated BOCONGO higher than in the internal evaluation (receiving 16% more). Volunteer management was fully rated in both cases and there is little difference between professional development ratings. Several employee management and employee retention competencies were indicated as 0 in the internal assessment where the external evaluation found BOCONGO to be performing better on these points in contrast. The difference between internal and external ratings for human resources is directly comparable in the graph that follows. Figure: Internal vs external BOCONGO OCA human resources ratings Financial management The financial management domain is made up of three sub-sections: financial planning, financial management and accounting. Financial management is one of four domains where there is no remarkable disagreement between evaluations to discuss. Both evaluations conferred an excellent rating on BOCONGO in this area. The only notable difference is that the internal evaluation from earlier in the year found some minor fault at the time with accounting practices. Good financial management ratings are corroborated by good organisational structure and governance ratings. Figure: Internal vs external BOCONGO OCA financial management ratings Infrastructure and information technology The infrastructure and information technology domain is also source of little disagreement between internal and external evaluations, both having conferred positive assessments (86% internal vs 84% external). According to the internal evaluation, BOCONGO misses a few points on asset management, computer password security, backup systems as well as some other minor points. Programme service management The programme service management domain is one of four domains where internal and external OCA ratings are in agreement. While the overall score for both internal and external in this case are the same (88%), both evaluations arrive at this conclusion differently when considering the ratings for programme service management’s sub-sections. Programme planning was given a near-perfect rating internally where the external evaluation found that BOCONGO’s capacity for reflection and learning from programme implementation experience was intermediate rather than excellent (95% internal vs 75% external). Programme implementation scores are very similar in practice, the difference in ratings being inflated by double-weighting of programme implementation in the OCA instrument specification. See the difference between internal and external programme service management scores for BOCONGO in the graph at the end of this subsection. Figure: Internal vs external BOCONGO OCA programme & service ratings Process management Process management is broken up into three sub-sections: monitoring & evaluation system management, data collection and reporting and evaluation and data analysis. As with the broader trend of the discrepancies between internal and external OCA assessment for BOCONGO, the internal rating is notably higher for the process management domain (90% internal vs 69% external). The biggest difference between internal and external assessment of BOCONGO’s process management capacities were to do with reporting and evaluation and data analysis, where the external evaluation found that BOCONGO’s evaluative practices did not employ external evaluators, did not work beyond donor requirement and showed no evidence of being reflexive with programme management through evaluation. The internal evaluation did not identify the same faults. Figure: Internal vs external BOCONGO OCA process management ratings Inter-organisational linkages Inter-organisational linkages is one of the domains where this is some significant disagreement between assessments (95% internal vs 69% external). A variety of criteria were only partly satisfied according to the external assessment. BOCONGO OCA Discussion BOCONGO appears to be a solidly performing organisation on the whole. Differences in assessments difficult to account for, especially with only two observations of a single organisation. Technical Capacity Assessment Tool Results Discussion Compared to both the OCA and OCA Lite, the TCA tool is a shorter, simpler and more targeted instrument, focused more on output and delivery of organisations with a focus on HIV & AIDS prevention and testing, tuberculosis treatment, PHDP, OVC, gender and GBV. TCA was used in this evaluation to evaluate BONEPWA and BOCAIP at regional level as well at their head offices. Sekgele Training Home Society’s sole (and therefore national) office in Gumare was also assessed using the TCA. Historical data for the head offices of BONEPWA and BOCAIP from internal assessment using data from an internal TCA from September 2014 will be used for comparison. Similarly, recent historical data for BONEPWA’s head office using the TCA assessment from July 2015 has been used for comparison in this evaluation’s findings for validation. The overall TCA results by organisation and office is compared in the graph that follows. Both of BONEPWA and BOCAIP’s national offices were rate very highly at 97% with their regional offices trailing slight with BOCAIP Gumare at 94% and BONEPWA Maun a 91%. Segkele is clearly the weakest in terms of TCA criteria at 63%. In summary, Sekgele’s TCA rating is attributable to poor performance in the quality improvement and assurance domain but this will be explored in the discussion that follows. Figure: TCA overall ratings Figure: external overall scores TCA for BONEPWA and BOCAIP national offices Figure: external overall scores TCA for district offices of BOCAIP, BONEPWA & Sekgele Domains ‘Technical planning and design’ is the first domain of the TCA. Technical planning and design is a checklist of of numerous competencies in a few general areas including an organisation’s level of alignment with current policy and standard practice in their areas of focus (HIV & AIDS prevention and testing, tuberculosis treatment, PHDP, OVC, gender and GBV). Other areas are capacity for client referral (according to focus area), good practice with HIV prevention activities and incorporation of strategic input from affiliates and stakeholders. ‘Staff capacity’ is the second domain. Staff capacity covers recent training of both staff and managerial staff in their functions relevant to their organisation’s areas of focus. The balance of staff capacity competencies are the availability of document resources, policies, the organisation’s agenda, job roles and update to date literature on focus areas. TCA’s third and forth domain are notably shorter in number of competencies. The third TCA domain is called ‘linkages and networks’. Linkage and networks broadly addresses an organisations level of participation and contribution to national policy, NGO networks and stakeholders. The fourth and final domain is called ‘quality improvement and assurance’ which broadly addresses technical supervisory capacity and whether that internal technical supervisory function is achieving goals like: maintaining up to date materials based on best practice, systematic mentorship, capacity to seek out external expertise and the capacity to document and report on technical supervisory functions for the organisation. As there are only four domains, they will be discussed together in the sections that follow, comparing organisation and the data that was collected as part of this evaluation with historical data. 2015 External Evaluation The radar graph that follows comparatively displays scores for all for TCA domains by organisation from the data collect for this evaluation. Technical planning & design is abbreviated as “technical planning” and quality improvement and assurance is abbreviated as “quality assurance” in the graph. The radar graph shows that all organisations with the exception of Sekgele have good scores of 80% and above and have a good balance between domains. In contrast, the graph line representing Sekgele cuts through the middle of the graph from linkages and networks to staff capacity, indicating low compliance with the criteria set out in quality improvement and assurance. Figure: TCA ratings by domain Figure: external TCA ratings by domain for BONEPWA and BOCAIP national offices Figure: external TCA ratings by domain for district offices of BOCAIP, BONEPWA & Sekgele Sekgele’s poor ratings in the are of quality improvement and assurance seem to reflect their status as a smaller, regional organisation as compared to BOCAIP or BONEPWA (either nationally or regionally). Their overall rating in this domain is due to partial competency ratings to do with their mentorship system. Related functions involving documented procedures, reporting and reaching out for external expertise were all non-existent. Aside from from Sekgele’s low rating, other organisations evaluated with TCA had good to excellent ratings. BONEPWA’s Maun branch did not meet with a few competency criteria under technical planning & design, notably having no response to tuberculosis or GBV and only partially satisfying a few other competencies. BOCAIP’s Gumare branch’s only truly detracting competency is that HIV training and counselling training had not been refreshed for the past two years. National organisation branches had only a few partial criteria ratings as the reason for a their TCA ratings being just short of perfect. See individual domain scores by organisation from this evaluation in the following table for more clarity on individual ratings. Organisation Technical planning and design Staff capacity Linkages and networks Quality improvement and assurance Sekgele 75% 75% 92% 31% BONEPWA National 98% 100% 100% 83% BONEPWA Maun 87% 97% 79% 100% BOCAIP National 100% 97% 86% 94% BOCAIP Gumare 97% 91% 86% 94% Change Between 2014 to 2015 for BONEPWA and BOCAIP national offices Internal measurement points exist for the head offices of both BOCAIP and BONEPWA from September 2014, roughly a year prior to this evaluation. Comparisons to the scores from this internal evaluation point for both organisations seem to show general improvement along TCA rating criteria. The ratings for each domain from September 2014’s internal evaluation and this evaluation are plotted by organisation for comparison on the two radar graphs that follow. The first is BOCAIP. In this comparison it would seem that BOCAIP improve in all domains with the exception of linkages and networks which remained the same at 86% (still a good score). Technical planning seems to have made a significant leap too 100% followed in order by staff capacity and quality assurance. Figure: Comparing national BOCAIP internal 2014 TCA domain ratings with external 2015 Similarly to BOCAIP’s national office, BONEPWA’s national office is unchanged in one domain (quality improvement and assurance at 83%) but improved dramatically in all other domains since September 2014’ internal evaluation. The biggest improvement was in staff capacity by more than 20% (raised to 100%), followed in order by linkages and networks and technical planning. Figure: Comparing national BONEPWA internal 2014 TCA domain ratings with external 2015 ratings TCA Validation Having a recent internal measurement for comparison can be a useful validation exercise but in this case comparison raises some questions of validity. An internal review point of BONEPWA’s head office happened in July 2015 which was about a month prior to this evaluation. The internal and external ratings for each domain have been plotted on the radar graph that follows and a fairly different pictures emerge. The two are only in alignment in terms of quality improvement and assurance but for other domains, there is a disparity whereby the external evaluator’s rating is higher by around 20% per domain. This comparison does not offer any corroboration to either assessment. Figure: Comparing internal and external national BONEPWA TCA 2015 domain ratings Further corroboration of some significant difference between internal and external execution of TCA can be see through comparing internal assessments for BONEPWA’s head office from September 2014 and July 2015. The radar graph that follows shows both years of internal assessment for BONEPWA’s head office with very similar results. Figure: Comparing internal 2014 and 2015 national BONEPWA TCA ratings TCA Discussion All organisations assessed using TCA rating criteria did well with the exception of Sekgele. In practice it would seem that there may not be a basis for fair comparison between Sekgele and BOCAIP or BONEPWA given different scales of organisation. Sekgele does however have numerous organisational competencies that it could improve on by TCA or other standards. The weakness of this comparison is evidenced in that the majority of the competency criteria for the domains, technical planning and design and staff capacity were deemed not applicable in relation to Sekgele. Figure: Comparing internal 2014 and external 2015 national BONEPWA TCA domain ratings Figure: Comparing 2014 and 2015 internal national BONEPWA TCA domain ratings 4 APPENDIX IX: LIST OF TRAINING CONDUCTED FY Training courses Who participated Length of course Facilitator/ Core Sub-Partner FY12 Result 1: Strengthened technical competence and organizational capacity of NGOs and networks to support field offices, affiliates and member organizations with expanded and strengthened delivery of strategic and high quality HIV/AIDS services FY12 The purpose of the workshop was to prepare the mentors to work with CS. Mentors. A total of 12 participants from 8 companies. 2 days BOCCIM FY12 Microsoft Outlook training for BONEPWA+ staff, 10 participants. 1 day A BOCCIM mentor FY12 OD training: visibility and communications Gaborone and Maun for all Maatla partners 2 days WUSC FY12 OD training: results￾based management (M&E) workshop All partners 5 days WUSC with the Institute for Development Management (IDM) FY12 OD training: financial management workshop National NGOs Unknown BOCCIM FY12 OD training: industrial relations and Botswana labour laws National NGOs 2 days BOCCIM FY12 Gender training: how National NGOs, 26 people attended 4 days WUSC and PCI 5 FY Training courses Who participated Length of course Facilitator/ Core Sub-Partner to review and refine institutional policies to integrate gender considerations FY12 In-house training on MS Outlook BONEPWA+ employees 1 day A consultancy on IT assistance using WUSC sector funds. Co-facilitated by BONEPWA+ OD Advisor and Botho College. WUSC worked with FHI360 to procure equipment for partners FY12 Diffusion of training curricular on HIV prevention, care and treatment BOCAIP and BONEPWA+ for their service providers in Ngami, Okavango, Chobe and Ghanzi Unknown MoH and FHI 360 FY 12 Result 2: A model developed and implemented to strengthen district level systems to provide and coordinate HIV/AIDS services in hard to reach areas FY12 Leadership development training NCONGO members. 23 people from 12 organizations. 5 days BOCODOL FY12 Governance training CSOs in Ngami and Okavango. 20 people from 11 organizations 3 days A consultancy firm, Initiatives Inc (NCONGO developed TORs) FY12 Financial management training CSOs in Ngami and Okavango. 41 people from 23 organizations 2 days Teo consultants (NCONGO developed TORs) FY12 Two training workshops on industrial relations NCONGO members. Project management had 15 participants and 2 days each BOCCIM in collaboration with NCONGO and facilitated by 2 6 FY Training courses Who participated Length of course Facilitator/ Core Sub-Partner and project management industrial relations had 16 participants. consultants FY12 District level training workshops to prepare private sector mentors to engage with CSOs 6 mentor organizations and 6 teacher volunteers. BOCCIM FY12 USG financial rules and regulations FHI 360 staff 3 days FHI 360 FY Training courses Who participated Length of course Facilitator/Core Sub-partner FY13 Result 1: Strengthened technical competence and organizational capacity of NGOs and networks to support field offices, affiliates and member organizations with expanded and strengthened delivery of strategic and high quality HIV/AIDS services FY13 Database management, and also participated in the resource mobilization training Interns (at 4 national organizations who are supporting efforts in strengthening ICT and resource mobilization) WUSC FY13 Effective office management Receptionists 5 days WUSC FY13 Leadership Development and Succession Planning Training BOCONGO, BOCAIP, BONASO and BONEPWA+ 2 days 7 FY Training courses Who participated Length of course Facilitator/Core Sub-partner FY13 Customised governance trainings National NGOs FY13 Financial management training National NGOs, 11 participants attended FY13 Technical areas BONEPWA+ volunteers in Ghanzi FHI360 supported BONEPWA+ FY13 Refresher trainings in the Maatla technical areas/focus districts New and existing volunteers in the 4 model districts FHI360 supported BONEPWA+ and BOCAIP FY13 Result 2: A model developed and implemented to strengthen district level systems to provide and coordinate HIV/AIDS services in hard to reach areas FY13 OD success competency domains: Basic computer skills training NCONGO members NCONGO FY13 Financial Management for NGOs NCONGO members FY13 Strategic Leadership NCONGO members FY13 Governance training NCONGO members 8 FY Training courses Who participated Length of course Facilitator/Core Sub-partner FY13 Mainstreaming training Non-HIV/AIDS and health CSOs in Ngami and Okavango NCONGO in collaboration with NACA FY13 Governance trainings NCONGO Facilitated through the WUSC program FY13 Project management and industrial relations CSOs in Ghanzi and in Chobe 2 days each BOCCIM FY13 Mentor trainings: District level training workshops to prepare the new private sector mentors to engage with CSOs Mentors. 9 mentors were trained in Ghanzi, and 4 in Chobe BOCCIM FY13 Classroom-based training ICIPs 1 week Inveneo Result 3: Strengthened institution(s) able to provide sustainable capacity building to civil society FY13 Customised, context-specific, participatory resource Mobilisation training BOCONGO WUSC through the ODAs FY Training courses Who participated Length of course Facilitator/Core Sub-partner FY14 Result 1: Strengthened technical competence and organizational capacity of NGOs and networks to support field offices, 9 FY Training courses Who participated Length of course Facilitator/Core Sub-partner affiliates and member organizations with expanded and strengthened delivery of strategic and high quality HIV/AIDS services FY14 Customised leadership refresher training CEOs of the national umbrella organizations, including NCONGO WUSC, through a consultant FY14 Leadership for strategic health communication for HIV combination prevention HIV prevention champions 33 participants from different government departments and civil society organizations attended the training NACA FY14 Refresher training in HCT BOCAIP counselors With the support from MoH through the DHMTs. FY14 Refresher training in community TB care BOCAIP staff and volunteers With the support from MoH through the DHMTs. FY14 Refresher training in adult and children’s palliative care BONEPWA+ volunteers in Chobe district FY14 Result 2: A model developed and implemented to strengthen district level systems to provide and coordinate HIV/AIDS services in hard to reach areas FY14 Gender analysis training WUSC Uniterra members (co-facilitated by BOCONGO’s Gender Officer) in Ngamiland and Okavango districts, and 7 organizations were assessed. WUSC FY14 4 trainings: HR, M&E, basic computer training and gender and HIV/AIDS Ngami coalition members 10 FY Training courses Who participated Length of course Facilitator/Core Sub-partner mainstreaming training FY14 Training to strengthen DMSAC relationships, role clarification, partner management and team building activities for the DMSAC Okavango NASTAD FY Training courses Who participated Length of course Facilitator/Core Sub-partner FY15 Individualized immersion sessions (review of the previously created RM plans) Each of the national umbrella bodies FHI360 and WUSC consultants 11 APPENDIX X: LISTS OF TOOLKITS AND GUIDELINES DEVELOPED, OD Toolkits and Guidelines FHI360 • Financial Management Toolkit • Governance success story • Volunteer Management toolkit • Policies BOCAIP • Governance Manual • Resource Mobilization • HIV mainstreaming BONEPWA+ • Support group coordination and management guidelines • Governance Manual • Revised Constitution • Communication Strategy • Draft Strategic Plan BOCAIP • Board Governance manual • Strategic Plan • Staff Performance Appraisal tool • Board and National Coordinator Performance Tool • Communication Strategy Technical Capacity Guidelines and Toolkits Developed FHI360 • HIV prevention 10-14 • HIV prevention 15-19 • HIV prevention Adults • Gender Norms and GBV • OVC handbook • Technical Supervisory checklist • Technical Capacity Assessment tool • HIV care and support: – Pre-ART – Positive Living – Adherence Counseling – Disclosure 12 M&E tools and data sets developed Tools Data Sets HIV prevention Daily Register HIV prevention HIV prevention Monthly Summary Form Care and Support for people living with HIV Gender Norms and GBV Daily Register OVC Gender Norms Monthly Summary Form HCT PHDP daily Register TB PHDP Monthly Summary Form Referrals Initial Needs Assessment tool Training Care daily asessment tool Care monthly summary form OVC initial assesment tool OVC daily register OVC monthly Summary Form TB initial registration (Botswana National TB program) Referral Daily Tally Sheet Referral Monthly summary Form TB HCT individual forms HCT daily Tally Sheet HCT Training registers 3 APPENDIX XI: USAID CRITERIA TO ENSURE THE QUALITY OF THE EVALUATION REPORT (USAID EVALUATION POLICY)