EVALUATION USAID/TANZANIA POST-PERFORMANCE EVALUATION OF PAMOJA TWAJENGA March 2019 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by ME&A. USAID/TANZANIA POST-PERFORMANCE EVALUATION OF PAMOJA TWAJENGA Submitted: March 18, 2019 Prepared by: Lawrence Robertson, Team Lead, ME&A Brenda L. Pearson, Chief of Party. ME&A Bernard Kindoli, CSO Development Consultant, ME&A Shakila Mayumana, Gender Specialist, ME&A Edward Mhina, CSO Development Consultant, ME&A Gerald Usika, Survey Specialist, NORC Submitted by: Brenda L. Pearson, Chief of Party ME&A, Inc. USAID/Tanzania Data for Development Program Plot 122 Mwai Kibaki Road, Mikocheni A, Dar es Salaam, Tanzania Email: Bpearson@engl.com Cover Photo: PT Supported NACOPHA Cluster Leaders Credit: Gerald Usika DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. ABSTRACT Data for Development conducted a post-performance evaluation of the United States Agency for International Development Tanzania’s (USAID/Tanzania) Capacity Development for Partners of Accountability (CPDA) Activity, or Pamoja Twajenga (PT) (“Together we Build” in Kiswahili) in late 2018. The activity, implemented by Chemonics from July 2013 to July 2017, supported the Mission’s grants to Tanzanian institutions of accountability (IoAs) and civil society organizations (CSOs), as well as other CSOs. PT provided capacity building support to improve constructive citizen-government engagement. The evaluation team (ET) used a mixed-methods approach through document and data review and key informant interviews to answer five evaluation questions. The ET found that PT’s partners identified improved and constructive citizen/government engagement that had occurred over the course of the program. The ET found that the technical support provided to USAID’s CSO partners met the needs and priorities of the Mission in its management of CSO grant activities. Performance management data demonstrated that PT met or almost met its targets for capacity strengthening of CSOs, while IoA targets were not met as two of the three IoA partners had limited years of engagement with the project. The ET found that capacity developed with the support of PT has been sustained as long as CSOs have the financial resources. Stakeholders are largely satisfied with the performance of the contractor in training and providing TA. The evaluation makes recommendations for the Mission for any future capacity development support to CSOs or Government of Tanzania partners. CONTENTS EXECUTIVE SUMMARY................................................................................................................................................... i Evaluation Purpose and Evaluation Questions.............................................................................................. i Program Background .......................................................................................................................................... i Evaluation Methods and Limitations............................................................................................................... ii Findings, Conclusions, and Recommendations............................................................................................ ii 1.0 EVALUATION PURPOSE AND EVALUATION QUESTIONS...................................................................... 6 1.1 Evaluation Purpose .............................................................................................................................. 6 1.2 Evaluation Questions.......................................................................................................................... 6 1.3 Audience and Intended Users........................................................................................................... 6 2.0 PROGRAM BACKGROUND.................................................................................................................................. 7 3.0 EVALUATION METHODS AND LIMITATIONS .............................................................................................. 8 3.1 Evaluation Methods............................................................................................................................. 8 3.2 Limitations........................................................................................................................................... 11 4.0 FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS ..................................................................... 12 4.1 EQ 1: To what extent did the intended outcome of improved and constructive citizen/government engagement occur over the course of the program?........................... 12 4.1.1 Findings.................................................................................................................................. 12 4.1.2 Conclusions .......................................................................................................................... 15 4.1.3 Recommendations .............................................................................................................. 16 4.2 EQ 2: Did the technical support provided by CDPA to USAID’s CSO partners meet the needs and priorities of the Mission related to its management of civil society grant activities?.............................................................................................................................................. 16 4.2.1 Findings.................................................................................................................................. 16 4.2.2 Conclusions .......................................................................................................................... 17 4.2.3 Recommendations.............................................................................................................. 17 4.3 EQ 3: Did the activity meet its targets in strengthening the capacity of CSOs and GoT IOAs?.................................................................................................................................................... 18 4.3.1 Findings.................................................................................................................................. 18 4.3.2 Conclusions .......................................................................................................................... 21 4.3.3 Recommendations.............................................................................................................. 21 4.4 EQ 4: To what extent have the actions supported or encouraged through the CDPA activity been sustained by beneficiary IOAs and CSOs? .......................................................... 21 4.4.1 Findings .................................................................................................................................. 21 4.4.2 Conclusions .......................................................................................................................... 24 4.4.3 Recommendations.............................................................................................................. 24 4.5 EQ 5: Are the stakeholders satisfied with the performance of the contractor in training and providing TA to the direct CSOs and IOAs? Why and why not?.................................. 25 4.5.1 Findings.................................................................................................................................. 25 4.5.2 Conclusions .......................................................................................................................... 31 4.5.3 Recommendations.............................................................................................................. 31 ANNEXES ......................................................................................................................................................................... 32 Annex I: Evaluation Statement of Work ..................................................................................................... 33 Annex II: Inception Report............................................................................................................................. 53 Annex III: Evaluation Design With Data Collection Instruments.......................................................... 63 Annex IV: List of Interviews........................................................................................................................... 86 Annex V: Disclosure of Any Conflicts of Interest.................................................................................... 90 LIST OF TABLES Table 1: Data Collected in PT Post Evaluation Field Work .................................................................................. 10 Table 2: Satisfaction of Stakeholders with Training and TA Provided by Pamoja Twajenga ......................... 30 LIST OF FIGURES Figure 1: Score of Combined Key Areas of Organization Capacity Among USG Direct and Indirect Local IPs in Percent.................................................................................................................................................................... 19 Figure 2: Proportion of Total Targeted PT-Supported Organizations Meeting Their Annual Proposed Benchmarks for Specific Categories of Institutional Strength Development .................................................... 19 Figure 3: Percent Change in Overall Scores of Advocacy Skills of Targeted CSOs........................................ 20 ACRONYMS Acronym Description AAC Authorized Association Consortium AFNET Anti Female Genital Mutilation Network AIMS Achievement, Innovate, Measure, and Share AJAT Afya ya Jamii Tanzania AOR Agreement Officer Representative CDCS Country Development Cooperation Strategy CDP Capacity Development Plan CDPA Capacity Development for Partners of Accountability CEGO Citizens Engaging in Government Oversight CHEDO Community Health Education and Development Organization Chemonics Chemonics International COP Chief of Party COR Contracting Officer Representative CSO Civil Society Organization CWMAC Community Wildlife Management Areas Consortium DC District Council DO Development Objective DRG Democracy, Human Rights, and Governance E&D E&D Readership and Development Agency EQ Evaluation Question ES Ethics Secretariat ESRF Economic and Social Research Foundation ET Evaluation Team FY Fiscal Year G2G Government-to-Government GESI Gender Empowerment and Social Inclusion GI Group Interview GICA Government Institutional Capacity Assessment GoT Government of Tanzania HDT Human Development Trust IoA Institute of Accountability IP Implementing Partner IPRS Implementing Partner Reporting System KII Key Informant Interview LEAT Lawyers’ Environmental Action Team LGA Local Government Authority LOE Level of Effort LOP Life of Program MBOMIPA Matumizi Bora Wildlife Management Area M&E Monitoring and Evaluation MAT Medical Association of Tanzania ME&A ME&A, Inc. MMDEA Mahenge Mazombe Development Association MUVIMA Mufindi Vijana kwa Maendeleo Acronym Description N/A Not Applicable NACOPHA National Council of People Living with HIV and AIDS in Tanzania NAOT National Audit Office of Tanzania NGO Non-Governmental Organization OCA Organizational Capacity Assessment PCCB Preventing and Combating Corruption Bureau PELUM Participatory Ecological Land Use Management Tanzania PLHIV People Living with HIV PPR Performance Plan and Report PPRA Public Procurement Regulatory Authority PT Pamoja Twajenga RECLAIM (Re)claim Women’s Space in World Heritage RUDI Rural Urban Development Initiatives SAM Social Accountability Monitoring SATC Supporting Accountability to Tanzania Citizens SI Standard Indicator SOW Scope of Work TA Technical Assistance TACOSODE Tanzania Council for Social Development TAGRODE Tanzania Grassroots Oriented Development TAMWA Tanzania Media Women’s Association TANLAP Tanzania Network of Legal Aid Providers UMADEP Uluguru Mountain Agricultural Development Project USAID United States Agency for International Development USG United States Government WCSA Women Creative Solutions Association WiLDAF Women in Law and Development in Africa ZLSC Zanzibar Legal Service Center i EXECUTIVE SUMMARY EVALUATION PURPOSE AND EVALUATION QUESTIONS The following evaluation report on the Capacity Development for Partners of Accountability (CPDA) Activity, or Pamoja Twajenga (PT) (“Together we Build” in Kiswahili), was commissioned by the United States Agency for International Development Tanzania (USAID/Tanzania) in September 2018. The evaluation was conducted as a retrospective post-performance evaluation of the Activity from its inception in July 2013 to its completion in July 2017. USAID requested in the Scope of Work (see Annex I) that the evaluation: • Document the rationale for the approach; • Understand what worked well and what did not; • Examine what has been sustained; and • Consider where to improve if the Agency would like to design a similar activity in the future. The evaluation answered five evaluation questions (EQs): 1. To what extent did the intended outcome of improved and constructive citizen/government engagement occur over the course of the program? 2. Did the technical support provided by CDPA to USAID’s civil society organization (CSO) partners meet the needs and priorities of the Mission related to its management of civil society grant activities? 3. Did the activity meet its targets in strengthening the capacity of CSOs and Government of Tanzania (GoT) institutes of accountability (IoAs)? 4. To what extent have the actions supported or encouraged through the CDPA activity been sustained by beneficiary IoAs and CSOs? 5. Are the stakeholders satisfied with the performance of the contractor in training and providing technical assistance to the direct CSOs and IoAs? The results of this evaluation will be used by USAID/Tanzania to inform decisions and design of the forthcoming Country Development Cooperation Strategy (CDCS) and future programming. The intended users of the evaluation include USAID/Tanzania Development Objective 3 Team, the Democracy, Human Rights, and Governance office in particular, and the Agency at large. PROGRAM BACKGROUND PT was implemented by Chemonics International (Chemonics). The stated purpose of the Activity was to build the capacity of targeted CSOs and GoT IoAs to engage government entities with citizens in government oversight. PT formed part of USAID/Tanzania’s larger Supporting Accountability to Tanzanian Citizens (SATC) program, which was developed in 2012 and began implementation in early 2013, and aimed to: • Strengthen targeted CSOs’ capacity to act as advocates and watchdogs; and • Strengthen GoT IoAs’ capacity to inform the public and to respond to civil society accountability concerns. SATC was developed to fill gaps in relationships that limited connectivity between citizens and their local governments, between CSOs and citizens, between CSOs, and between civil society and government, particularly IoAs and local level government. SATC was comprised of eight other awards plus PT. PT was developed as part of this broader initiative because USAID anticipated that the CSO and IoA partners would need capacity building support. The five USAID direct grantee CSOs were: ii • Lawyers’ Environmental Action Team (LEAT); • National Council of People Living with HIV and AIDS in Tanzania (NACOPHA); • Participatory Ecological Land Use Management Tanzania (PELUM); • Tanzania Council for Social Development (TACOSODE); and • Women in Law and Development in Africa (WiLDAF). PT also worked with 14 sub-grantee partners of the five direct grantee CSOs. Ten (10) other non-funded CSOs identified by USAID as potential future partners for PT also received capacity development support. USAID also provided government-to-government (G2G) grants to three GoT IoAs to help them strengthen their working relationships with civil society and improve the accessibility and utility of information they intended to make public. The three USAID-supported IoAs were: • Ethics Secretariat (ES); • National Audit Office of Tanzania (NAOT); and • Public Procurement Regulatory Authority (PPRA). EVALUATION METHODS AND LIMITATIONS The evaluation team (ET) employed a mixed-methods approach to conduct the evaluation. The ET gathered, analyzed, and triangulated primary source data from key informant interviews (KIIs) and group interviews (GIs) with stakeholders involved in the program or beneficiaries of program assistance activities. The ET also did a desk review of program materials and other written products relevant to the activity. The review was used as part of developing the Inception Report (Annex II) and the Evaluation Design (Annex III), including all evaluation instruments. The team also analyzed trends from performance monitoring data reported during the activity into the implementing partner reporting system (IPRS). Data was collected from 35 organizations: USAID, Chemonics, PT subcontractor Economic and Social Research Foundation (ESRF), all thee IoA PT partners, all five USAID direct grantee CSOs, a sample of their sub-grantees (14 CSOs), and all 10 non-grantee PT CSO partners. Fieldwork was conducted in four PT sites in Dodoma, Iringa, and Dar es Salaam in Mainland Tanzania and Unguja in Zanzibar. In all, the ET conducted 54 KIIs and GIs. The ET worked with USAID, Chemonics, and PT partners to purposively select relevant staff from the organizations. All qualitative data from KIIs and GIs were recorded and transcribed (apart from USAID KIIs, which were not recorded). The ET used a Microsoft Excel matrix for qualitative coding of transcripts and tallied recurrent themes. Limitations identified included data availability and data quality, selection bias, recall bias, and halo bias. The ET took steps during the design and implementation of the evaluation to manage and minimize the effects of these limitations on data quality. FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS EQ1: To what extent did the intended outcome of improved and constructive citizen/government engagement occur over the course of the program? Findings: The ET’s interviews with CSO and government partners indicated improved constructive citizen/government engagement during the activity through public awareness, advocacy, issue-based dialogues, and capacity building. PT partners asserted that their work in implementing their USAID awards under SATC with PT’s assistance contributed to this outcome. The PT component of SATC focused on organizational capacity building to support their ability to engage citizens, CSOs, and GoT IoAs and provided training, technical assistance (TA), mentoring, and coaching to support public awareness raising from IoAs, advocacy from CSOs, and issue-based dialogues that linked CSOs, communities, and government institutions. iii Conclusions: Activity partners noted improved citizen/government engagement during the period of PT; however, the retrospective nature of the post-performance evaluation methods does not give a comprehensive picture of the activity’s performance in improving citizen/government engagement in Tanzania over 2013-2017. Findings across multiple stakeholders suggest contributions from PT’s work in terms of building the organizational capacity of CSOs and IoAs, building capacity in public awareness and advocacy, and improving constructive citizen-government engagement through issue-based dialogues. Recommendations: USAID should continue to look for opportunities to support improved and constructive citizen/government engagement as important tracks in supporting civic engagement and government accountability in Tanzania. • USAID should consider continuing to support issue-based dialogues as a mechanism to support constructive citizen-government engagement. • As part of planning for the next CDCS, USAID should consider commissioning an assessment of opportunities and constraints to citizen participation in governance. • USAID should consider continued engagement in monitoring and assessing the organizational capacity of CSOs and IoAs—and building their capacity as needed. • USAID should consider continued engagement in monitoring public awareness and advocacy by CSOs and IoAs and supporting further capacity building in public awareness raising and civic advocacy. EQ2: Did the technical support provided by CDPA to USAID’s CSO partners meet the needs and priorities of the Mission related to its management of civil society grant activities? Findings: USAID staff interviewed asserted that the TA provided by the activity to USAID’s CSO partners met the needs and priorities of the Mission. All of the six USAID staff with direct engagement with PT interviewed noted that activity TA provided to USAID’s CSO partners met the needs and priorities of the Mission in its CSO grants management. USAID staff appreciated the roles that PT played in supporting direct grantee CSOs. Notwithstanding, USAID staff that supervised direct grantees noted that issues remained in PT’s direct grantees’ work under their awards with USAID. Conclusions: The technical support provided by CPDA to USAID’s CSO partners met the needs and priorities of the Mission related to its management of civil society grant activities. USAID and former PT staff saw support from PT as important in helping USAID’s direct grantees meet the demanding USAID rules and regulations required to implement their awards—a prerequisite for working towards Citizens Engaging in Government Oversight (CEGO) program outcomes. PT’s training and TA to the direct grant CSOs met USAID’s needs related to its management of civil society grant activities by supporting the overall development of these organizations as well as building their capacity and supporting them in adhering to guidance and best practices. Recommendations: USAID should continue to work with Tanzanian CSOs towards the Mission’s objectives and Tanzania’s development goals. • USAID should continue to engage with current and former direct grantees, other potential CSO partners, and sub-grantees of former direct grantees to maintain the interest of these partners in working with USAID. • USAID should develop ways to keep these capacitated former and/or potential partners abreast with new developments regarding USAID grants/program management requirements to keep them ready to potentially partner with the Mission. • USAID staff should continue to monitor and assess the organizational and advocacy capacity of past PT partner CSOs that might serve as useful partners for USAID in the future. iv EQ3: Did the activity meet its targets in strengthening the capacity of CSOs and GoT IoAs? Findings: PT reported on progress towards its capacity building objectives through 10 indicators, including one standard indicator and nine custom indicators. The activity exceeded targets for only one indicator, while falling slightly below targets for the other nine indicators. Challenges in reaching targets included under-reporting of performance data in Fiscal Year (FY) 2017 and that some IoAs did not continue their CEGO awards with USAID. Conclusions: PT did not quite reach its targets. Setting targets for capacity building assistance when the contractor can only provide support but does not have any way of enforcing the use of stronger practices, may make targets difficult to attain. Recommendations: USAID should consider more appropriate targets in circumstances where the contractor does not have managerial control rather than using targets that call for 100 percent achievement by all partners under the contract. USAID should also develop ways to encourage consistent reporting to the IPRS. EQ4: To what extent have the actions supported or encouraged through the CDPA activity been sustained by beneficiary IoAs and CSOs? Findings: PT staff did assessments and CSOs did self-assessments after training with PT support. The way Chemonics staff and CSO partner staff discussed PT activities and answered interview questions focused on how PT helped them meet USAID requirements. USAID staff noted that awards to the ES and two direct grantees continued after PT closed; staff that continued to manage these awards noted that these partners continued to use PT-supported tools and practices. CSOs interviewed also noted that they continue to use PT-supported skills and have maintained their relationships with their sub-grantees and partners, as did IoAs. Fourteen (14) CSOs noted they were not able to continue with some PT-developed practices based on resource constraints. Conclusions: Some direct grantees, sub-grantees, and other CSOs that worked with PT were able to sustain practices and skills promoted by PT. The four-year time frame for PT appears to be short in terms of supporting the development of Tanzanian CSOs to meet higher standards of organizational and financial capacity identified in the organizational capacity assessment process. Four years is a short period to successfully achieve sustainable development of organizational capacity. CSO partners that PT worked with depend heavily on grant resources, which further makes it difficult to sustain capacity building achievements. Recommendations: USAID should consider making sustainability and sustainability planning a more explicit part of CSO capacity-building support awards like PT. • USAID should consider developing an approach to supporting the development of CSO capacity focused on sustainability that works through a contractor or grantee who has a contractual responsibility and incentive to encourage and plan for sustainability. • USAID should consider maintaining relationships with direct grantees and other implementing partners (IPs), not necessarily funding relationships, to encourage continued practice of PT￾promoted practices and skills by CSOs. • USAID should consider using its staff to continue to conduct assessments of CSOs’ capacity in a participatory manner under PT along with the staff of these organizations. • USAID should consider gathering CSO partners and prospective partners together as part of building a community of capable Tanzanian CSO partners that could work directly with USAID or as subcontractors to other USAID partners, particularly in challenging priority areas for USAID, such as civic engagement and government accountability. v • USAID should consider providing information sessions and refresher training to former direct grantees, other CSO partners of PT, and former sub-grantees to keep them abreast of developments in the civil society sector and USAID. This could be done through follow-up and under other USAID mechanisms that work through and support CSOs. EQ5: Are the stakeholders satisfied with the performance of the contractor in training and providing TA to the direct CSOs and IoAs? Findings: PT stakeholders were broadly satisfied with the activity’s performance and appreciated the support of PT for their organizational capacity development and advocacy/dialogues. Direct USAID grantee, sub-grantees, other CSOs that worked with PT, and IoAs were, on average, satisfied with the training and TA they received in all areas. Where partners were less satisfied was with the steps PT could not take to address the capacity gaps identified though PT assessment processes given the limited time and resources available in the program. To a great extent, PT beneficiaries were satisfied with the content of training, the relevance of training to the growth of their organizations, the quality of after training services, and its applicability to their day-to￾day operations. All five of PT’s direct grantee partners felt that the training and TA were tailored to address specific organization capacity gaps and practical skills needed for them to successfully implement their CEGO program. Other PT CSO partners and sub-grantees also largely expressed satisfaction with PT training and TA. USAID interviewees that compared the PT model to a Grants under Contract model did not identify unique benefits to USAID from the PT model. CSOs that were less satisfied had unmet or unrealistic expectations. Although most CSOs reported that they were satisfied with PT training and TA, four other CSOs that worked with PT noted that they were disappointed that their expectations of getting a grant award from USAID once they improved their organizational capacity was not met. They felt that their involvement in the PT training and TA processes would help them be competitive in winning a grant award from USAID, but these ambitions were not realized. Conclusions: USAID staff were largely satisfied with PT’s performance; former PT staff were proud of their contributions to capacitating CSOs and IoAs, and beneficiary CSOs and IoAs were largely satisfied with the support they had received. However, the latter also noted that some areas of assistance were more useful than others and provided a more critical review of some assistance. Recommendations: USAID should continue to consider including capacity building assistance in awards to CSO and government partners that may face challenges implementing USAID awards. • USAID can consider programming CSO awards and capacity building support through mechanisms like PT in the future or through more conventional Grants under Contract modalities. • USAID should manage expectations with CSO partners and potential partners to avoid building unrealistic or unmet expectations about the prospects for USAID funding for CSOs. Several CSOs mentioned that after the activity, they expected an opportunity to work more directly with USAID and that they were still waiting post-program for that time to come. 6 1.0 EVALUATION PURPOSE AND EVALUATION QUESTIONS 1.1 EVALUATION PURPOSE The United States Agency for International Development Tanzania (USAID/Tanzania) commissioned Data for Development, implemented by ME&A, Inc. (ME&A), to conduct a post-performance evaluation of its Capacity Development for Partners of Accountability (CPDA) Activity—more commonly known as Pamoja Twajenga (PT) or “Together We Build” in Kiswahili—in Fall 2018. USAID asked Data for Development to develop an approach to gather and analyze valid and reliable data to assess the adequacy and comparative advantage of CPDA’s approach. The evaluation is a post-performance evaluation since PT ended in July 2017. As per the Scope of Work (SOW), the evaluation sought to: • Document the rationale for the approach; • Understand what worked well and what did not; • Examine what has been sustained; and • Consider where to improve if the Agency would like to design a similar activity in the future. The evaluation may inform future programming at the USAID/Tanzania Mission and suggest approaches/strategies for USAID. 1.2 EVALUATION QUESTIONS According to the SOW, the evaluation sought to answer the following five evaluation questions (EQs): 1. To what extent did the intended outcome of improved and constructive citizen/government engagement occur over the course of the program? 2. Did the technical support provided by CDPA to USAID’s civil society organization (CSO) partners meet the needs and priorities of the Mission related to its management of civil society grant activities? 3. Did the activity meet its targets in strengthening the capacity of CSOs and Government of Tanzania (GoT) institutions of accountability (IoAs)? 4. To what extent have the actions supported or encouraged through the CDPA activity been sustained by beneficiary IoAs and CSOs? 5. Are the stakeholders satisfied with the performance of the contractor in training and providing technical assistance (TA) to the direct CSOs and IoAs? 1.3 AUDIENCE AND INTENDED USERS The results of this evaluation activity will be used by USAID/Tanzania across sectors to inform decisions and design of the forthcoming CDCS and future development programming. The intended users of the evaluation include USAID/Tanzania Development Objective (DO) 3 team; the Democracy, Human Rights, and Governance (DRG) office in particular; and the Agency at large.1 1 Audience and intended use from USAID Tanzania DRG Office’s Evaluation SOW. 7 2.0 PROGRAM BACKGROUND PT was implemented by Chemonics International (Chemonics) from July 2013 through early July 2017. The purpose of the activity was to build the capacity of targeted CSOs and GoT IoAs to engage citizens in government oversight. PT formed part of USAID/Tanzania’s larger Supporting Accountability to Tanzania Citizens (SATC) program that was developed in 2012 and began implementation in early 2013, which aimed to: • Strengthen targeted CSO’s capacity to act as advocates and watchdogs; and • Strengthen GoT IoA’s capacity to inform the public and to respond to civil society accountability concerns. The SATC program was developed to fill gaps in relationships that limited connectivity between citizens and their local governments; between CSOs and citizens; between CSOs; and between civil society and government, particularly IoAs and local-level government. SATC complimented other elements of USAID’s Democracy and Governance portfolio under DO3 of the Country Development Cooperation Strategy (CDCS): “Improved Accountability and Oversight of Public Resources through Increased Citizen Participation.” The SATC program was comprised of eight other awards in addition to PT. PT was developed as part of this broader initiative because USAID anticipated that the CSO and IoA partners would need capacity building support in order to work effectively with USAID in implementing their awards. USAID issued five direct grants to CSOs to support their endeavors in improving accountability and governance in agriculture and food security, health, and natural resources management and to improve their linkages to smaller local CSOs and their citizen constituents, monitor government accountability, and improve flows of information with IoAs. This part of the program was referred to as Citizens Engaging in Government Oversight (CEGO). Awards issued at the beginning of 2013 supported recipients’ advocacy and work with sub-grantees at the grassroots level to: • Share information with citizens and gather evidence about both government performance and citizen priorities; and • Develop effective advocacy messages for appropriate decision-making levels and strengthen their own general understanding of IoAs. These five USAID direct grantee CSOs were: • Lawyers’ Environmental Action Team (LEAT); • National Council of People Living with HIV and AIDS in Tanzania (NACOPHA); • Participatory Ecological Land Use Management Tanzania (PELUM); • Tanzania Council for Social Development (TACOSODE); and • Women in Law and Development in Africa (WiLDAF). USAID also provided government-to-government (G2G) grants to three GoT IoAs to help strengthen their working relationships with civil society and to improve the accessibility and utility of information they intend to make public. Increased access to public information is important because it allows citizens to play their role in ensuring representation and accountability in policy-making and service delivery. These three USAID-supported IoAs were: • Ethics Secretariat (ES); • National Audit Office of Tanzania (NAOT); and • Public Procurement Regulatory Authority (PPRA). In addition, USAID/Tanzania identified 10 other CSOs and asked PT to support their capacity 8 development. These other CSOs were seen as being of interest to USAID and as potential partners in the future. They were not recipients of USAID grants at that time. The 10 non-grantee CSOs PT supported were: • Authorized Association Consortium (AAC)/Community Wildlife Management Areas Consortium (CWMAC); • Human Development Trust (HDT); • Medical Association of Tanzania (MAT); • (Re)claim Women’s Space in World Heritage (RECLAIM); • Rural Urban Development Initiatives (RUDI); • Tanzania Media Women’s Association – Mainland (TAMWA Mainland); • Tanzania Media Women’s Association – Zanzibar (TAMWA Zanzibar); • Women Creative Solutions Association (WCSA); • Zanzibar Legal Service Center (ZLSC); and • E&D Readership and Development Agency (E&D)/Soma. PT has four objectives: 1. Strengthen the capacity of USAID/Tanzania partner CSOs to serve as strong advocacy organizations, well-networked at the grassroots level and able to inform and equip citizens to engage with government; 2. Provide technical support to USAID’s CSO partners to support USAID in its management of civil society grant activities; 3. Strengthen the capacity of GoT IoAs to package information; employ a cohort of civil servants with a common understanding of ethics, procurement, auditing, and good governance rules; and, generally, perform activities under their assistance agreement with USAID/Tanzania; and 4. Facilitate constructive engagement between CSOs and IoAs. In addition, PT’s staff and consultants worked with sub-grantees of direct grantee CSOs they were supporting. Chemonics provided specialized TA in the form of training, coaching, mentorship, and on-the￾job training. Specific areas of interventions were determined according to the particular needs of the CSO or IoA through Organizational Capacity Assessments (OCAs) or Government Institutional Capacity Assessments (GICAs). PT then worked with these organizations on capacity development plans (CDPs) to meet their identified needs and provided training, mentoring, and coaching focused on these needs at both the individual/team and organizational levels. PT’s support helped build the knowledge and skills of individuals and teams, as well as strengthen structures, tools, policies, procedures, and practices to enhance the capacity of partner CSOs and IoAs to be more efficient, cost-effective, transparent, and technically proficient organizations that are accountable to their respective constituents. PT also supported advocacy assessments of these organizations and helped develop their skills and experience in advocacy and working with citizens and government. 3.0 EVALUATION METHODS AND LIMITATIONS 3.1 EVALUATION METHODS To address the EQs, the evaluation team (ET) employed a mixed-methods approach, which gathered and analyzed program reporting through a desk review, primary data collected through key informant 9 interviews (KIIs) and group interviews (GIs) with stakeholders involved in the activity or beneficiaries of activity interventions, and activity performance monitoring data. This approach allowed the ET to triangulate information, ensuring that findings and conclusions are robust and that recommendations are sound. Desk Review The ET reviewed key CDPA/PT materials and other documents that have information to answer the research questions and to provide background information to support the evaluation process, particularly for the Inception Report. Documents reviewed included: 1) the SOW, 2) Annual Work Plans for the duration of the activity, 3) Monitoring & Evaluation (M&E) Plan, 4) Quarterly Reports, 5) Annual Reports, 6) Final Report, 7) USAID/Tanzania CDCS 2014/19, and 8) PT Achievement, Innovate, Measure, and Share (AIMS) reports. In addition, USAID provided the team with a sample of PT-produced and PT-partner produced reports, which were also reviewed and used as data for the evaluation. These included: 1) OCA Reports, 2) GICAs, 3) CDPs, 4) Annual and Quarterly Reports of CEGO grantees, 5) Annual and Quarterly Work Plans of CEGO grantees, 6) Annual Work Plans of IoAs, and 7) Annual and Quarterly of Reports of IoAs. Key Informant and Group Interviews The ET conducted a total of 54 semi-structured interviews. KIIs were conducted one-on-one with an informant; when an interview had more than one participant, the team conducted the interview as a GI instead. KIIs and GIs followed the same questions and discussion guide (see Annex III for the instruments used and Annex IV for the list of individuals interviewed). The ET worked with the USAID/Tanzania Mission DRG office to identify the relevant individuals from institutional partners of PT. Key informants interviewed included senior-level staff from Chemonics, national level grantees, non-grantee staff, sub-grantee staff, IoAs (ES, NAOT, and PPRA), Economic and Social Research Foundation (ESRF), and USAID staff. The ET purposively selected the staff with the right information about PT. Snowball sampling was used to select 1-6 key informants from PT partner organizations to participate in KIIs or GIs. Chemonics and/or USAID/Tanzania were consulted to ensure that the most relevant staff at CSOs and IoAs were selected as key informants for interviews. Based on the number of stakeholders of PT, the ET planned and conducted interviews with a total of 35 organizations. A total of 54 KIIs and GIs were completed in all four selected regions plus Zanzibar. The ET developed a KII/GI guide that was tailored to the different types of interviewees. The guide included 15-20 open-ended questions. All KIIs/GIs, with the exception of interviews with USAID staff, were recorded. Informed consent was obtained prior to the start of each interview. Interview transcripts were coded thematically to produce a summary of recurrent themes that emerged in response to each topic. Review and Analysis of Performance Monitoring Data Performance monitoring data produced by PT was obtained from the secondary data provided by the Mission and Chemonics. These data were essential specifically to answer EQ 3.2 They were analyzed using descriptive statistics in Microsoft Excel. Quarterly and annual reporting on program indicators and trends over time were observed from the start to the end of program implementation. Performance data were drawn from the Implementing Partner Reporting System (IPRS) to assess the time series of actual versus target performance on indicators reported to the Mission. The ET assessed performance on pre- 2 EQ3: Did the activity meet its targets in strengthening the capacity of CSOs and GoT IoAs? 10 determined, valid criteria (objectives and expected results), which can be consistently measured over the activity’s life cycle. Target Areas and Sampling Data for the evaluation were collected from 35 organizations. The ET held KIIs with USAID staff, former Chemonics PT staff, and staff from Chemonics’ subcontractor, ESRF. The ET interviewed key staff from all three IoA PT partners and all five USAID direct grantee CSOs. A minimum of two sub-grantees were selected from the set of partners working in districts from each of the five direct grantees. Overall, the ET interviewed 14 sub-grantees. The ET also held KIIs with all 10 non-grantees CSO partners of PT. (See Annex IV for the list of all interviews). The particular individuals interviewed from these organizations were selected on a case-by-case basis as the ET identified interviewees with the most knowledge and engagement with PT. Purposive sampling was used to select the regions and districts for KIIs and GIs. It was not feasible for the ET to visit all regions where PT activities took place. The ET selected regions and districts for KIIs and GIs to achieve a geographic balance and to cover the main areas where PT-supported CSOs, direct grantee CSOs, and IoAs worked. PT focused on Dodoma, Iringa, Morogoro, Mtwara, Dar es Salaam, and Zanzibar of which the ET conducted fieldwork in in Dodoma, Iringa, Morogoro, Dar es Salaam, and Zanzibar. Unstructured informal interviews were held to better understand stakeholders’ engagement with PT. The bulk of CSO direct grantees and sub-grantees reached were in the following districts: Bahi, Chamwino, and Kongwa in Dodoma region; Mvomero and Morogoro in Morogoro; Iringa, Kilolo, and Mufindi in Iringa; Rungwe in Mbeya; and Temeke in Dar es Salaam. Seven districts were selected for field work. WiLDAF, NACOPHA, PELUM, and TACOSODE overlapped in some selected districts, which were prioritized. Two districts in each region—Morogoro (Mvomero and Morogoro District Center), Iringa (Kilolo and Mufindi), and Dodoma (Bahi and Chamwino)—were selected to ensure maximum coverage of the CSO activities. Temeke was selected in Dar es Salaam region as the district with the most CSO coverage. (See Table 1 for a summary of data collected by PT stakeholder group and location.) Table 1: Data Collected in PT Post Evaluation Field Work PT Stakeholder National (Dar es Salaam) Morogoro Region Iringa Region Dodoma Region Zanzibar Total USAID technical staff 9 - - - - 9 PT Chemonics staff & partners 8 - - - - 8 IoAs – ES, NAOT, PPRA 1 - - 3 - 4 Direct USAID CSO Grantees 6 1 - - - 7 CSO Subgrantees 1 1 8 4 0 14 Other CSOs, PT partners3 9 0 0 0 3 12 Total KIIs & GIs 34 2 8 7 3 54 3 PT worked to support 10 CSOs that did not have direct grants with USAID. These are listed on page 8 above: AAC/CWMAC, HDT, MAT, RECLAIM, RUDI, TAMWA Mainland, TAMWA Zanzibar, WCSA, ZLSC, and E&D/Soma. 11 Data Analysis Methods The ET conducted both qualitative and quantitative analyses of the achievements and shortcomings of the activity in relation to its objective and targets and the EQs. The ET used an Excel matrix for qualitative coding of transcripts and tallied recurrent themes across respondent groups by organization and organization type (USAID, Chemonics, ESRF, IoA, direct grantee, sub-grantee to direct grantee, and other CSO partner). Excel was also used to analyze and visualize quantitative performance monitoring data from the activity using descriptive statistics. The analyses covered the output and outcome indicators for the activity, as well as the KII and GI data on improved and constructive citizen/government engagement, the approaches used in the PT, the sustainability of actions supported or encouraged through PT, and stakeholder satisfaction with the performance of Chemonics in PT. All information and data were disaggregated by the types of relationships these organizations had with PT, types of cooperation with PT, and geographic location at the national, regional, or local level. Important findings below are illustrated with quotes. 3.2 LIMITATIONS Limitations with potential impact on the evaluation, and how the ET minimized these limits, included: • Data availability and data quality: Data to assess whether PT met its objectives were drawn from PT reporting and PT inputs into the IPRS. IPRS data is assumed to be accurate, as is PT-reported performance data. However, some of these data seem to imperfectly measure key concepts in capacity building and advocacy/citizen-government engagement. These issues are discussed in findings. • Selection bias: The ET did not have time and resources to conduct KIIs in all districts where PT’s CSO direct grantees and sub-grantees worked, introducing possible biases in the selection of these sub-grantees. Fieldwork in five target districts reached all five CSOs direct grantees and 14 CSOs sub￾grantees. All of these stakeholders were reached in the five sites selected. Sampling five of seven districts reduces the risks of bias as the team sampled most districts and sub-grantees. • Recall bias: Since a number of questions raised during the interviews addressed issues that took place from 2013-2017, and given that the activity is already phased out, informants may not have been able to provide accurate and complete responses. They may also have conflated training or support received from other programs or donors as having been from PT. The ET managed this potential bias by following up open-ended questions about PT by asking about specific details of the project to prompt memories. Risks of conflating other assistance with PT were not found to be a problem. • Halo bias: There is a known tendency among respondents to under-report socially undesirable answers and alter their responses to approximate what they perceive as the social norm. The extent to which respondents were prepared to reveal their true opinions may have varied for some questions that call upon the respondents to share their attitudes and perceptions in front of their peers or colleagues. To mitigate this limitation, the ET used confidentiality and anonymity statements to all who participated in KIIs and GIs. The team also conducted the interviews in a neutral setting where respondents felt comfortable. • Reliance on second-hand perceptions to assess citizen outcomes: Due to timing of the evaluation, limited resources, and the retrospective nature of the methodology, there were no direct interviews with citizens that benefitted from PT. The ET managed this limitation by asking USAID, former PT staff, the IoAs, USAID’s direct grantee CSOs and their sub-grantees, and PT’s other CSO partners about improved and constructive citizen-government engagement and having them discuss PT’s contributions to these outcomes. 12 4.0 FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS 4.1 EQ 1: TO WHAT EXTENT DID THE INTENDED OUTCOME OF IMPROVED AND CONSTRUCTIVE CITIZEN/GOVERNMENT ENGAGEMENT OCCUR OVER THE COURSE OF THE PROGRAM? Data relevant to answering EQ1 come from PT reporting and interviews with USAID staff, Chemonics’ former PT staff, direct grantee staff, sub-grantee staff, other CSO partner staff, and IoA staff. PT reports and ET interviews noted some improved and constructive citizen/government engagement occurred over the course of PT from July 2013 through the end of the program in early July 2017. Note that neither this question nor the data to answer this question address the contributions of PT to this intended outcome. Nevertheless, in answering this question, many interviewees spoke to the contributions of their organizations towards some improved and constructive citizen/government engagement, and some volunteered that PT had helped their organization contribute to this outcome. 4.1.1 Findings Citizen and Government Engagement All five USAID staff interviewed that had direct knowledge or experience with PT noted improved citizen/government engagement. Two other USAID interviewees declined to answer as they had no information. Interviewees noted both general improvements and more constructive engagement both overall and between the CSOs and IoAs with the awards that they managed in addition to sub-grantees and PT partners. Former PT staff interviewed said that they provided basic management skill training and mentoring to CSOs and IoAs. They also said that PT support contributed to strengthened capacity of CSOs to monitor issues related to delivery of social services, public expenditures, and corruption. Former PT staff further noted that reports from CSOs and IoAs suggested that training and TA provided by PT to its beneficiaries contributed to a great extent to improved access to information and constructive citizen-government engagement. Former PT staff were of the opinion that the activity had been very successful in supporting the capacity building that contributed to these achievements. All five direct grantee CSOs, all 14 of their sub-grantees, all 10 CSOs assisted by PT, and all three IoAs noted improved and constructive citizen and government engagement in interviews. Several specific categories where improvements were noted are mentioned below after the initial discussion of broader improvements. Direct grantee WiLDAF noted improved and constructive citizen/government engagement in issues concerning law reforms, inheritance laws, changing child laws as well as marriage laws. LEAT and NACOPHA noted improved capacity of village-based institutions in engaging with government in issues related to local resource allocation and health and other services, making for more constructive citizen/government engagement. Other CSOs noted positive examples as well. One direct grantee mentioned that issues of discrimination against people living with HIV (PLHIV) are decreasing as they are now as accepted as any other group in society. The ES emphasized that, during this period, it reached out to citizens and changed how they engaged with citizens so that people knew about the mission of the ES and could contribute to the administrative methods the ES uses to monitor and enforce ethics by public servants. These changes were supported by 13 the ES’s G2G award and TA from PT. The ET noted that this improved constructive citizen/government engagement. NAOT emphasized that the office improved citizen/government engagement by increasing its engagement with citizens through increased outreach by NAOT staff and through the development of a “popular” version of their annual report in Kiswahili. Both of these efforts were designed to and did increase knowledge among citizens of the work of the NAOT, thus improving citizen/government engagement. PPRA noted improvement in constructive citizen/government engagement with citizens knowing their rights and laws on public procurement and gaining awareness and understanding of public procurement procedures: “Yes, there is great improvement in constructive citizen engagement as follows: citizens are well informed and aware of PPRA core obligations of public procurement to the society, to an extent that these citizens can differentiate information and complaints that need to be taken and sorted by PPRA, and which information and complaints should be taken to PCCB,4 unlike before, citizens were not well engaged so they did not have the right information about PPRA.” However, PPRA also noted that the legal framework for the agency did not allow for some kinds of public engagement in procurement and review that were envisioned in their G2G award. Some limits on transparency with citizens were seen to be required to ensure fairness in the contract award and procurement process. USAID and former PT staff interviewed also mentioned limits to the overall USAID engagement with PPRA deriving from the legislation on what citizens could do with PPRA—and how this affected PT engagement with PPRA and CSOs that were originally expected to work with PPRA. PPRA appreciated the support for venues, per diems, and all other costs for the benchmarking exercises that were done with CSOs. PPRA saw these as improving the information from citizens to PPRA, which was useful for investigations on citizen complaints on misuse of funds, corruption, and other issues. Improved Advocacy Work Four of the five direct grantees noted improved advocacy work in communities, as did three of their 14 sub-grantees interviewed. Key informants from direct grantee CSOs that worked with PT asserted that their participation in the implementation of CEGO program improved their and their sub-grantees’ capacity in both making citizens responsible for their development and making government more accountable and responsive to citizens’ demands. A total of 16 out of 24 CSOs interviewed (both direct grantee, sub-grantees, and other CSOs) said that they noted some improvement in citizen-government engagement during the course of implementing the CEGO program. Sixteen (16) out of 24 interviewed CSOs noted that citizens were able to access quality services in the areas of health, legal assistance, gender equality, natural resources management, and land ownership and tenure. For example, one direct grantee discussed citizens benefiting from natural resources proceedings and obtaining land ownership certificates. As he/she stated, “… about 650 people were enabled to obtain land ownership certificates through PELUM and CSO partners facilitation.” Its sub-grantee, on the other hand, said that “… we noticed reduced incidences of land conflicts as a result of enhancing land dispute resolutions right from [the] village level.” Direct grantee CSOs stated they did not know if the outcome of citizen engagement would lead to resolution of these disputes but believed it could guide them to seek legal services, which could potentially make a difference. Some notable examples of constructive citizen-government engagement highlighted by CSOs included increased understanding of natural resources policies, laws, and regulations among citizens and government officials (at district, ward, and village levels) for improved participation in monitoring natural resources management and conservation [the cases of Matumizi Bora Wildlife Management Area (MBOMIPA), Mufindi Youth for Development, and CWMAC] and a greater sense of ownership of 4 The Preventing and Combating Corruption Bureau. 14 development processes by citizens. Other examples highlighted by CSOs included demand for improved service delivery to PLHIV (led by NACOPHA clusters in Iringa and Mafinga), improved provision of health services at the facility level [cases of the Iringa Mercy Organization, Community Health Education and Development Organization (CHEDO), the Anti-Female Genital Mutilation Network, and the Mazombe Mahenge Association for Development], and improved citizen participation in wildlife and natural resources management (led by MBOMIPA and CWMAC). A MBOMIPA key informant cited one example of citizen ownership and participation in natural resources management. He said that “…in Mfyome village of Iringa, region villagers are now conducting patrols to prevent illegal extraction of natural resources in their localities. Through these patrols, Mfyome villagers, in collaboration with Wildlife Management Authority paramilitary, were able to follow up and catch culprits who were then fined [for illegally harvesting logs].” A total of 10 out of 14 interviewed sub-grantees shared the similar opinion that citizen-government engagement had been constructive. According to a CHEDO key informant “…they noted a lot of changes, including establishment of functioning health committees, establishment of service provision monitoring committees, and follow-ups on their accountability to citizens.” They (sub-grantees) mentioned a number of examples of constructive citizen government engagement. All three IoAs noted improved advocacy from communities over the 2013-2017 period. PPRA noted improved and constructive citizen and government engagement in “CSOs taking responsibility of being ambassadors to special groups, communities in order to teach them about laws that protect value for money in public programs….” ES reported that citizens often did not know the Secretariat prior to PT. With PT’s support, ES used its grant with USAID to raise awareness on how to engage with the roles and regulatory functions of the ES. PT supported the ES on how to reach the citizens “in a participatory approach.” One ES respondent stated that “Before Pamoja Twajenga, when it came to the engagement of citizens, we were blind. With their presence, however, they opened up our eyes.” Issue-Based Dialogues Ten (10) out of 14 interviewed sub-grantees noted that the formation of platforms for community/citizens and local government authorities (LGAs) to discuss and find solutions to developmental problems contributed to increased attendance at village meetings. Citizens’ understanding of the protocols improved, advancing their agendas to different levels of the LGAs in search for solutions to their problems; some key informants also reported reduced conflicts between communities and authorities. IoA interviews noted improved and constructive citizen and government engagement from issue-based dialogues. Dialogues were used by the IoAs and district authorities to spread information to citizens (including women, disabled people, and PLHIV) to contribute to increased awareness, transparency, and understanding of rights as well as directly connect citizens to the government. IoAs noted that issue-based dialogues have grown into a core interactive platform for sharing ideas and connecting with citizens and media in improving constructive citizen engagement. One IoA interviewee stated that they now increasingly view CSOs as “ambassadors of their communities.” Social Accountability Monitoring (SAM) Eight out of 11 interviewed sub-grantees noted that formation and capacity building of SAM committees at the village, ward, and council levels has made it possible for citizens to access key information on their development activities, including plans, income, and expenditure, as well as promoting citizens’ participation in village-level planning meetings and statutory decision-making meetings, such as village assembly. One direct grantee noted that they had created SAM teams to get access to information and foster accountability and transparency over main national resources. One sub-grantee key informant said, “CHF contributions increased following the introduction of making health facility information publicly available and are still at high levels…citizens’ attendance to public meetings is still active to the point village leaders say the program has kindled fire!” Non-grantee CSOs noted that citizens are more confident and aware of issues concerning their villages, and the leaders are being held accountable. 15 Citizen Awareness Activities Eight out of 11 interviewed sub-grantees stated that provision of information by LGAs and service providers to citizens promoted trust between citizens’ service providers and LGAs as evidenced by decreased complaints on services delivered and increased participation of citizens in planning, implementation, and monitoring of development activities through established platforms. Sub-grantees further noted that “the seeming antagonism that was growing between service providers and citizens in the program area has been reduced. Before the program, citizens felt that health service providers were personally benefiting from the sale of drugs that were meant to be availed though public health facilities. The situation has now changed due to the fact that all received supplies and subsequent usage are posted on notice boards for public consumption. So, cooperation has increased, and service provision has improved.” Direct grantees noted improvements in constructive engagement through public outreach and radio programs, village assembly meetings, and participation of citizens in holding the government accountable, especially in providing health services. Sub-grantees noted that sensitization, self-awareness, recognition by the society, openness, and education on sexually transmitted diseases and HIV/AIDS as well as the impact of early marriages has increased. IoAs mentioned that whistleblowers, and citizens, are informed and are more aware of where to take their complaints, according to type of complaints. Constructive engagement was observed by direct grantees through increased awareness and, in villages, enabling women to take over and complete the construction and ownership of health facilities, for example. The Tanzania Network of Legal Aid Providers (TANLAP) mentioned improved and constructive citizen/government engagement through various seminars, extended public hearings, meetings with ministers, and sharing of experiences from East Africa. Cascading Training/TA All 11 interviewed sub-grantees noted that the partners that had direct USAID grants (in this case LEAT, PELUM, TACOSODE, and NACOPHA) had worked with them to cascade PT training and TA to their organizations.5 NAOT stated that there was increased citizen understanding of the performance of public institutions through the publication and dissemination of the citizens’ report [a popular version of the Controller and Auditor General (CAG) report]. The report was made available to Parliamentarians, CSOs, and ordinary “wananchi.”6 4.1.2 Conclusions Key informants’ responses asserted that the intended outcome of improved and constructive citizen/government engagement occurred over the course of the PT program and the larger SATC initiative. USAID direct grantee CSOs and IoAs with G2G grants asserted that their work in implementing their CEGO awards contributed to this outcome. This post-evaluation of PT does not give a comprehensive picture of the evolution of improved and constructive citizen/government engagement in Tanzania over 2013-2017 or serve as an assessment of the challenges and opportunities in citizen/government engagement at present in the country. However, from findings across multiple stakeholders (including USAID, former PT staff, CSOs, IoA partners), PT’s work to build the organizational capacity of CSOs and IoAs and capacity in public awareness and advocacy 5 IMO, Mahenge Mazombe Development Association (MMDEA), MBOMIPA, Mufindi Vijana kwa Maendeleo (MUVIMA), NACOPHA Iringa Cluster, NACOPHA Mafinga Cluster, Tanzania Grassroots Oriented Development (TAGRODE), CHEDO, Afya ya Jamii Tanzania (AJAT), Anti Female Genital Mutilation Network (AFNET), and Uluguru Mountain Agricultural Development Project (UMADEP). 6 Wananchi means citizens. 16 and its support for issue-based dialogues have, in turn, supported improved and constructive citizen and government engagement. 4.1.3 Recommendations USAID should continue to look for opportunities to support improved and constructive citizen/government engagement, civic engagement, and government accountability in Tanzania. USAID should consider continuing to support issue-based dialogues as a mechanism to support constructive citizen-government engagement. As part of planning for the next CDCS, USAID should consider commissioning an assessment of opportunities and constraints in citizen participation in governance in Tanzania that could provide broader data on the opportunities and constraints that exist for the Mission to encourage citizen engagement in governance. USAID should consider continued engagement in monitoring and evaluating the organizational capacity of CSOs and IoAs—and supporting capacity building as needed. This monitoring and capacity development support can contribute to maintaining CSO/IoA activism and their potential as possible partners for USAID (or other donors) on shared goals and objectives. USAID should consider continued engagement in monitoring and evaluating public awareness and advocacy by CSOs and IoAs and supporting further capacity building in public awareness raising and civic advocacy. Tanzanian citizens also need to know and understand the capacity and activism of CSOs and IoAs in Tanzania for them to be effective overall (and effective potential partners for USAID and other donors). 4.2 EQ 2: DID THE TECHNICAL SUPPORT PROVIDED BY CDPA TO USAID’S CSO PARTNERS MEET THE NEEDS AND PRIORITIES OF THE MISSION RELATED TO ITS MANAGEMENT OF CIVIL SOCIETY GRANT ACTIVITIES? Data to answer this EQ come directly from interviews with USAID staff and former PT staff/consultants. Direct grantees, their sub-grantees, other CSOs, and IoA staff were not asked questions for this EQ as they are not the best informants to know and understand whether PT’s support met the Mission’s needs and priorities. 4.2.1 Findings Organizational Capacity Development (General) The six USAID staff interviewed that had direct engagement with PT all asserted that the TA provided to USAID’s CSO partners met the needs and priorities of the Mission in its CSO grants management. USAID staff in particular appreciated the roles that PT played in supporting direct grantee CSOs (and IoAs with G2G awards) to plan, implement, and report on the results of their CEGO awards from USAID (financial reporting and results/success stories). USAID staff emphasized how the quality of CSO narrative and financial reporting of direct grantees improved with PT’s training and mentoring, as well as its role as a quality control check reviewing and having CSOs strengthen their quarterly and annual submissions prior to their submission by these grantees to USAID. As one key informant noted, “PT acted as a bridge that enforced standards.” While noting satisfaction, USAID staff that supervised direct grantees further noted that issues remained in PT’s work with direct grantees under their USAID awards. This led three USAID staff to question whether PT was meeting USAID’s needs while asking about the source of the continued problems. As one staff member questioned, “Was the problem that the [PT] TA was not good, or were the direct grantees not able to use the TA that they were given?” This critical perspective on PT’s support, however, followed their overall assessment that the program met USAID’s needs. 17 Three USAID staff noted challenges for Chemonics with high levels of staff turnover at PT; this was considered as “problematic and confusing” to USAID and its partners. High levels of staff turnover impeded PT’s work as new staff had to learn about PT’s processes and past work as well as build knowledge of and relationships with PT’s partners in order to work with them effectively. Turnover at PT was also a problem for CSO and IoA partners who had to identify and build shared understanding with shifts in the people that were their main PT counterparts. Former PT staff noted several reasons for the high turnover. Tanzanians with the skills, qualifications, and experience in organizational development in a number of specialties where PT hired were seen to be in high demand. The competitive market for their services led hired staff to resign in favor of other opportunities. Several key PT staff also resigned over management style and personality differences internal to Chemonics and the PT team. PT staff interviewed noted that CSO partner organizations were able to improve their management systems through PT trainings, TA, and coaching and that doing this helped grantees meet USAID’s requirements for CSO partners. One former PT staff asserted that “TA and trainings were meant to meet needs and priorities of both USAID and the grantees.” Another PT staff stated that “the fact that direct grantees did very well in audits and that they were able to adequately account for financial resources granted to them and deliver expected CEGO program outcomes” to the satisfaction of USAID suggested that USAID’s needs and priorities were met. A third former PT staff was of the opinion that PT adequately addressed the CSOs’ capacity gaps identified in the pre-award capacity assessments that were conducted, calling these services “crucial for management of USAID grants and delivering CEGO program.” All five USAID direct grantees affirmed that PT’s support was instrumental in their ability to comply with the complicated and challenging USAID requirement for implementing their awards with the Mission. They particularly praised PT’s TA on grants management, financial management, and organizational development. 4.2.2 Conclusions The technical support provided by PT to USAID’s CSO partners met the needs and priorities of the Mission related to its management of civil society grant activities. Support from PT was seen by USAID and former PT staff as important in helping USAID’s direct grantees meet the demanding USAID rules and regulations required to implement their awards—a prerequisite towards working towards CEGO program outcomes. While PT’s activity with CSOs was said to have been useful to Mission staff, there is no evidence that there was any training or other TA directly for Mission staff participants on CSO engagement to facilitate USAID’s work with CSOs across Mission portfolios. Through its work on capacity building for USAID, PT helped build relationships between USAID and direct grantee CSO partners and made these partners more capable. Both direct grantee and other CSO partners and USAID staff noted that capacity strengthening and their relationships were useful for successfully implementing activities towards shared objectives. CSOs sought to have this capacity and relationships continue and win new or more awards with USAID. USAID staff noted how CSOs were valuable partners in their sectors. PT’s role as a provider of training and TA to the direct grantee CSOs met USAID’s needs related to its management of civil society grant activities by supporting the overall development of these organizations as well as building their capacity and supporting them in adhering to USAID rules, regulations, and expectations in grant execution and reporting. 4.2.3 Recommendations USAID should continue to engage with current and former direct grantees, other potential CSO partners, and sub-grantees of former direct grantees to keep the interest of these partners in working with USAID high because capable CSO partners are central to implementing activities towards USAID’s objectives in many sectors. 18 USAID should develop ways to keep these capacitated former and/or potential partners abreast of new developments regarding USAID grants/program management requirements to maintain their readiness to potentially partner with USAID. USAID staff should continue to monitor and assess the organizational and advocacy capacity of past PT partner CSOs that might serve as useful partners for USAID in the future. Continued engagement of USAID staff with these CSOs—including continued monitoring and evaluating CSO organizational and advocacy capacity—may help limit future capacity building needs of these CSOs should they win other USAID awards directly or serve as subcontractors to other USAID partners. Continued demonstrations that USAID staff care about these partners—even if they do not have direct grants from USAID—may help incentivize these CSOs to sustain the achievements made in organizational and advocacy capacity development through PT. PT’s measures of organizational capacity date from mid-2017; PT’s former partners report that their organizational capacity has changed since that time. The capacity of prospective partners needs to be assessed as part of evaluating potential grantees at the time of grants competitions. 4.3 EQ 3: DID THE ACTIVITY MEET ITS TARGETS IN STRENGTHENING THE CAPACITY OF CSOS AND GOT IOAS? Data on whether PT met its targets in civil society and IoA strengthening are drawn from PT’s performance monitoring and reporting through the IPRS and the activity’s Final Report. Interviews with PT’s CSO partners, whether USAID direct grantees and their sub-grantees or other CSOs that worked with PT, did not ask about targets as PT partners had no reason to know the activity’s targets; indeed, discussion indicated that CSO partners did not know or recall PT’s targets. Direct grantee CSOs focused on the goals, objectives, and targets in their own direct grants with USAID. The IoAs also focused on their own awards and had no reason to and/or did not know PT’s targets. The EQ does not ask whether PT met all of its targets, but only targets in capacity building. Thus, the analysis does not cover all PT indicators; seven indicators that measured activity are not used in the evaluation.7 The final measurement of indicators by PT was challenging because the activity ended in July, not at the end of the fiscal year (FY). Reporting for the IPRS was thus not done comprehensively by PT partners in Year 4. The shortened reporting period led some CSOs that were working on indicators not to report by the close of PT. 4.3.1 Findings PT reported on progress towards its capacity building objectives through 10 indicators: one standard indicator (SI) and nine custom indicators. Indicator 1 was an SI: the score, in percent, of combined key areas of organization capacity among United States Government (USG) direct and indirect local implementing partners (IPs). PT initially set targets that combined CSO partners and its IoA partners. The indicator was measured by OCA scores for CSOs and GICA scores for IoAs. PT measurements were done by repeating OCAs and GICAs with partners. A baseline was set in Year 1, and a life of program (LOP) goal of 50 percent improvement was made by FY 2017. According to the IPRS data, Indicator 1 surpassed its target in all years except for Year 4 (see Figure 1). The Final Report indicated that the final scores of 44 percent in FY 2017 were slightly under the LOP target of 51 percent. By the end of FY 2017, the CSO partners had achieved a score of 44.7 percent compared to a score of 20.5 percent for ES, the 7 Indicator 8, the number of selected CSOs accessing mentorship services in targeted regions; Indicator 11, the number of PT￾supported organizations using Government of Tanzania-produced information for advocacy; Indicator 12, the number of PT￾supported CSOs and IoAs reporting constructive engagement during each quarter; Indicator 13, the number of publishable stories per year that support qualitative outcome data provided to USAID; Indicator 14, the number of activities conducted by IoAs to address gender concerns; Indicator 15, the percentage of citizen-identified issues in the complaint register acted upon by partner IoAs; and Indicator 16, the number of joint meetings of targeted IoAs held. 19 sole remaining IoA partner (compared to a combined IoA partner score of 17.7 percent in FY 2016), which helps explain why the final score was below the LOP target. Figure 1: Score of Combined Key Areas of Organization Capacity Among USG Direct and Indirect Local IPs in Percent The ET also considered and used IPRS data to measure a similar indicator that tracked the achievements of 13 CDP benchmarks. IPRS data showed PT’s performance was above targets in FY 2015, slightly below targets for FY 2016, and far below the final LOP target, as organizations did not input data into the IPRS by the time PT closed. The IPRS data were thus not useful for measuring LOP achievement, only progress in the middle of implementation, which, on average, was about at the target. Indicator 2 was the proportion of total targeted PT-supported organizations meeting their annual proposed benchmarks for specific categories of institutional strength development. Figure 2 presents IPRS data on achievement for the first three years of PT. PT-supported organizations met or nearly met their annual proposed benchmarks between FY 2014 and FY 2016; there was a marked change in reported institutional strength between FY 2014 to FY 2015 when PT exceeded the target at 50 percent, this was only slightly improved in FY 2016. IPRS data were not inputted by all CSO and IoA partners after PT closeout. Figure 2 thus does not include and characterize PT’s FY 2017 achievements. The LOP target set was 14 of 18 organizations (78 percent). The PT Final Report noted that this total target was met, but the breakdown of CSO attainment (14 of 14) and IoA attainment (0 of 3) adds up to 14 of 17 (76.5 percent), so LOP performance actually fell slightly short of target in percentage terms. However, this also indicates that the target was set higher than possible achievement. Figure 2: Proportion of Total Targeted PT-Supported Organizations Meeting Their Annual Proposed Benchmarks for Specific Categories of Institutional Strength Development Indicator 3 was the proportion of PT-supported grantees in substantial compliance with USAID procurement requirements. This indicator was only measured for organizations with direct awards with 20 USAID: the five direct grantee CSOs and the three IoAs. The LOP target for the indicator was eight of eight (100 percent); Year 3 achievements were seven of eight (87.5 percent). For the five direct grantee CSOs and one remaining IoA, the LOP achievement was five of six (83.3 percent); all five CSOs performed consistently across all indicator components while the ES, the sole remaining IoA, did not. Indicator 4, the proportion of PT-supported grantees in substantial compliance with USAID’s financial management requirements, achieved 66.6 percent of its LOP targets. Four of the six CSOs met their target, with NACOPHA being the sole exception, while ES, the sole IoA, did not. Indicator 5, the proportion of PT-supported grantees in substantial compliance with USAID reporting requirements, achieved 83.3 percent of its LOP target. All five CSOs met their LOP targets while the ES, once again, did not. Under Indicator 6, the number of PT-supported organizations demonstrating improvement on gender action sections in their CDPs, Chemonics set a LOP target of 18. LOP achievement was 12, or 75 percent of target. The Final Report explains “Despite program emphasis through training and coaching, gender was not given priority in implementation by partners.” Indicator 7, the percent change in overall scores of advocacy skills of PT-supported organizations, had a LOP target of 75 percent. PT achieved a 65 percent change, short of the 75 percent LOP target (see Figure 3). PT noted some organizations performed poorly, which brought down the average despite high performance by others. IPRS data was identical to that reported by PT. Figure 3: Percent Change in Overall Scores of Advocacy Skills of Targeted CSOs Indicator 9, the percentage of PT-supported organizations that have stronger outreach activities to men and women, had a LOP target of 70 percent. Fifty (50) percent (7 of 14) of supported organizations reported stronger outreach activities, again short of the LOP target. Indicator 10, the number of selected CSOs that received targeted technical support for capacity development and institutional strengthening, had an LOP target of 14. LOP achievement was 15, above the target. Indicator 17, the percentage of IoA communication strategy benchmarks met, had a LOP target of 75 percent. LOP achievement was 10 of 14 (71.4 percent), slightly under the target. The Final Report notes PT's engagement with IoAs shifted in Year 4 such that 71.4 percent represents the LOP achievement of all planned communication benchmarks. 21 4.3.2 Conclusions Strictly measured, only one of PT’s 10 capacity building targets for CSOs and IoAs was exceeded, while targets for the remaining nine indicators in this area were not met. However, PT was close to meeting most of these nine other targets. PT thus did not quite reach its LOP targets. Challenges in reaching targets were affected by two IoAs not continuing their CEGO awards with USAID. Setting targets for capacity building assistance when it was limited to providing without any way of enforcing the use of stronger practices may make targets difficult to attain. 4.3.3 Recommendations USAID should consider more realistic targets in circumstances where the contractor does not have managerial control rather than using targets that call for 100 percent achievement by all partners under the contract. A recognition that not all diverse partners will meet targets would better represent reality and be more attainable (e.g., 8 of 10 CSO partners, or 80 percent achievement). USAID should also develop ways to encourage consistent reporting to the IPRS. 4.4 EQ 4: TO WHAT EXTENT HAVE THE ACTIONS SUPPORTED OR ENCOURAGED THROUGH THE CDPA ACTIVITY BEEN SUSTAINED BY BENEFICIARY IOAS AND CSOS? PT staff did assessments and CSOs did self-assessments after training with PT support. Some notable examples of constructive citizen-government engagement and data on the sustainability of PT actions come from interviews with USAID staff, former PT staff, USAID direct grantee and sub-grantees staff, other CSOs that worked with PT, and the IoAs who continued to engage with citizens and LGAs and/or sustained PT practices since PT’s close on July 7, 2017. Their knowledge of practices that continued to be used is direct evidence of actions that have been sustained. Data were not collected from USAID staff and former PT staff if they did not continue to engage with these partners as they thus did not have direct knowledge of sustainability. Data on sustainability were solicited in KIIs both by asking about whether organizations had PT-supported processes that had been sustained (and why these processes had been sustained) and whether organizations had PT-supported processes that had not been sustained (and why these processes had not been sustained). 4.4.1 Findings USAID staff noted that awards to the ES and two direct grantees (TACOSODE and NACOPHA) continued after PT closed. The three staff that continued to manage these awards noted that these partners continued to produce high quality reporting documents as they had with PT, suggesting that the achievements in stronger reporting from PT carried over and had been sustained. Two of these USAID staff noted that achievements in financial management also carried over and were sustained. However, one USAID staff interviewee noted that the audit report conducted on the direct grantee for its CEGO award, a year after PT ended, showed problems that had not yet been addressed by this direct grantee. This fact led to “doubts about what PT had been doing if the organization had this many problems with the audit.” Some former PT staff interviewed have continued to engage with the CSO partners for technical advice and, in some cases, on a consultancy basis after the close of PT. Both former PT staff that were still engaged with these partners noted that the former PT partners continue to use the skills they had developed with PT. These staff felt that continued use by CSOs indicated an appreciation of training and TA provided by PT. Former PT staff and consultants that shared examples of sustained activities explained that the activities were sustained because they were in the interests of these organizations and fit their programs. 22 Direct grantees (2) and sub-grantees (11) interviewed noted that they have maintained contact with each other. Direct grantees noted that their sub-grantees continue to consult with them for advice, while sub￾grantees noted that they continue to engage with their former direct grantee partners. The way Chemonics staff and CSO partner staff discussed PT activities and answered interview questions focused on how PT helped them meet USAID requirements. Key informants from all five USAID direct grantees noted however that their organizations continue to use skills and knowledge acquired through PT training and TA in their work after PT closeout, as did subgrantees and PT’s other CSO partners. All 29 CSOs (direct grantees, subgrantees, and other CSO partners) interviewed reported continuing all or some practices. These CSOs noted that the basic requirements of running an organization, meeting the mandates of their organizations, and furthering their own organizational goals were contributing factors to sustained PT-promoted practices and skills. CSO interviewees noted that they were able to sustain PT￾promoted practices/skills because they were within their organizational mandate and goals, and they had adequate finance to do so. IoAs also noted that they continue to use PT-improved systems and practices. All five direct grantees, 11 of 14 sub-grantees, and all 10 other CSO partners interviewed noted sustaining PT-promoted practices. Seven CSOs asserted that they continued to practice all the skills and knowledge they gained from PT in several cases illustrating this assertion with examples from areas of TA. Other CSOs noted specific areas where they sustained the results of PT assistance. Specific areas noted where PT-promoted practices were sustained included: • Workplan development: Four of the five direct grantees and all 11 sub-grantees asserted that they have continued to use the PT workplan framework. Six other CSO partners also said that they continued to use workplan processes from PT. • Program implementation: Three direct grantees, all 11 sub-grantees, and seven other CSO partners asserted that they continue to use skills developed in PT to implement their programs. • Communication and reporting: One direct grantee and four other CSO partners reported that they are continuing to prepare leaflets and newsletters with success stories and accompanying pictures as a result of PT TA. All 11 sub-grantees interviewed said that they have maintained a culture of preparing and sharing performance reports with stakeholders. • Financial resources: The five direct grantees, nine other CSO partners, and all 11 interviewed sub-grantees said that they are using PT-supported proposal writing skills to prepare and submit proposals to potential donors, although most applications are not successful. • Institutional or organizational management: All direct grantees, seven other CSO partners, and all 11 interviewed sub-grantees said that organizational manuals/guidelines that they developed when working with PT are still being used and have helped their organizations to maintain their performance standards. • Financial Management: Four direct grantees, nine subgrantees, and three other CSO partners noted continuing to use financial management tools, including QuickBooks applications, in which PT had provided training. CSOs said that acquired financial management skills are relevant for ensuring accountability to donors and demonstrating trustworthiness to resource partners, stakeholders, and general public. They also said that they now use audits as a management tool as all managers and officers are involved in responding to and implementing audit findings and implementation in order to prevent anomalies from recurring in future audits. • Governance and Board: Four direct grantees noted improved boards, as did five other CSO partners. Eight of 11 interviewed sub-grantees said that their boards’ performance has improved following PT assessment and TA. 23 • Gender Empowerment and Social Inclusion (GESI): Two direct grantees, six other CSO partners, and eight of 11 interviewed sub-grantees said that GESI is mainstreamed in all their programs and that they have gender guidelines/policy in place to guide them. • M&E Skills: CSO partners not only noted M&E as a challenge for their organizations prior to PT assistance but noted support for M&E through PT had led to continued improvement in this area. Two direct grantees and three other CSO partners noted continued stronger M&E after PT. Six of 11 interviewed sub-grantees said that they maintained the practice of reviewing and updating their M&E framework and plans even after the CEGO program closure. • Sustainability Plans: Sustainability plans developed under PT were said to exist (and thus continue) for all direct grantees and were noted by 10 other CSO partners. One direct grantee noted that it had updated its plan post-PT. Five of 11 sub-grantees said that they have documented sustainability plans that guide their resources mobilization and fundraising initiatives. Factors volunteered that supported sustaining PT-promoted practices and skills among CSO partners were: • The existence of financial resources, noted by one direct grantee, 12 sub-grantees, and four other CSOs; • Continuing to use organizational manuals/guidelines developed with PT, which was noted by the five direct grantees, 11 sub-grantees, and four other CSOs; • Ten (10) CSOs (one direct grantee, six sub-grantees, and three other CSO partners) attributed the sustainability of PT-promoted practices, skills, and activities to effectiveness of their governance systems; and • Among the IoAs, PPRA emphasized the presence of adequate human resources as a key to the sustainability of PT-promoted practices and skills. Inhibiting factors to sustaining PT-promoted practices and skills noted by CSOs included the inability to maintain quality staff after the end of program-provided resources. This challenge was noted across CSOs, including USAID direct grantees and those who received funding from other sources. The lack of financial resources was also noted by CSOs as a reason for not continuing to facilitate citizen-government dialogue platforms. Fourteen (14) CSOs provided examples of unsustained PT-promoted practices/skills due to financial constraints, including: • Human Resources: One direct grantee could not retain its Iringa office team as its staff contracts ended with the close of its CEGO program. • M&E: Six of 17 CSOs that reported strengthened M&E noted that they could not sustain M&E functions due to inability to retain trained staff at the closure of the CEGO program; and • Issue-Based Dialogues: One direct grantee partner said it could not sustain issue-based dialogues due to financial constraints. Six out of 11 sub-grantees interviewed could not sustain issue-based dialogues due to similar reasons. CSO explanations for not sustaining PT-supported changes centered on the lack of financial resources, a lack of institutional or organizational management, the lack of human resources, and not having strong enough organizational governance. IoAs noted that they did not sustain some PT-supported achievements as GoT policies changed (NAOT) or their own priorities changed (ES). PPRA did not state any explanations for not sustaining practices. PPRA explained that the organization now conducts issue-based dialogues in a different way. While PPRA 24 does not practice the way PT trained, it uses sensitization as an approach, which was created from ideas aligned to the PT program. PPRA’s sensitization targets the public in general, plus makes efforts to increase the interest of entrepreneurs in participating in public tenders. PPRA noted that government procurement laws and regulations “don’t allow citizens to be engaged in the public procurement process.” By law, some issues in public procurement have to remain confidential until after the tender has been awarded. By law, PPRA can only inform citizens of the details after the cooling off period. IoAs noted that they continue to use some PT-supported processes. NAOT emphasized that it maintains the culture of training its auditors on client-focused auditing practices as promoted by PT due to the fact that it formed a team of local trainers within the institution who serve as NAOT resource persons entrusted to roll out auditor trainings beyond the NAOT’s USAID-funded program. Another practice sustained by NAOT is continued publication and dissemination of Citizen Audit Reports. PPRA interviewees noted that the organization continues to produce simpler and shorter annual reports in both Kiswahili and English, known as Popular Versions. At the time of field work, ES was continuing to implement its USAID award. It asserted that the Secretariat continues to use PT-supported processes and things the ES had developed and learned to use through PT training and TA. As the ES noted, it continues to use these processes in the program management cycle to keep things going, “If you sleep, you don’t eat.” ES noted that it continues to hold issue-based dialogues, a technique it learned through PT. 4.4.2 Conclusions Direct grantees, sub-grantees, and other CSOs that worked with PT were to some extent able to sustain practices and skills promoted and provided by PT. The four-year time frame for PT appears to be short in terms of supporting the development of Tanzanian CSOs to meet exacting USAID organizational and financial capacity requirements. Four years is a short period to successfully emphasize sustainable development of this capacity. CSO partners that PT worked with depend heavily on grant resources; grant resources are problematic in terms of supporting sustainability of CSOs. Limited financial resources, particularly as CEGO grants ended, impeded the maintenance of PT-achievements as CSOs, in some cases, were not able to keep trained staff without the grant funds. IoA PT partners continue to use some processes and products developed with PT support as they value and appreciate these changes to their operations. 4.4.3 Recommendations USAID should consider making sustainability and sustainability planning a more explicit part of CSO capacity-building support awards like PT. USAID should consider developing an approach to supporting the development of CSO capacity focused on sustainability that works through a contractor or grantee who has both positive and negative incentives to offer partners to encourage sustainability. USAID should consider maintaining networks and collaborative relationships with CSOs that are former direct grantees, work with other USAID IPs, or were PT beneficiaries to encourage and strengthen CSOs with practices and skills promoted by PT. USAID should consider continuing to conduct assessments of CSO capacity in a participatory manner, like under PT, along with the staff of these organizations. USAID can continue to motivate PT beneficiaries to continue and potentially build on PT-supported capacity by demonstrating USAID’s continued interest in the strength of these CSOs. USAID should consider gathering CSO partners and prospective partners together as part of building a community of capable Tanzanian CSO partners that could work directly with USAID or as subcontractors to other USAID partners, particularly in challenging priority areas for USAID such as civic engagement and government accountability. Networks and coalitions around shared interests can help sustain and further the development of the sector and CSOs within it. 25 USAID should consider providing information sessions and even refresher training to former direct grantees, other CSO partners of PT, and former sub-grantees to keep them abreast of developments in the civil society sector and USAID. USAID direct grantees and other IPs should also be encouraged to develop and adopt citizen-government engagement methodologies and techniques that are less costly and can easily be owned and managed by citizens themselves with minimal support from CSOs. 4.5 EQ 5: ARE THE STAKEHOLDERS SATISFIED WITH THE PERFORMANCE OF THE CONTRACTOR IN TRAINING AND PROVIDING TA TO THE DIRECT CSOS AND IOAS? WHY AND WHY NOT? Data for answering EQ5 on satisfaction with the performance of PT training and TA come from interviews with stakeholders. Stakeholders interviewed included USAID staff, former PT staff, USAID direct grantees and their sub-grantees, other CSOs that worked with PT, and IoAs. The ET asked questions about aspects of satisfaction directly, as well as more general questions about the usefulness of PT support. Questions asked what types of PT assistance were the most useful and whether other assistance was more or less useful. CSO and IoA partners were asked to rate 16 different aspects of PT training and TA on a scale from very satisfied, to satisfied, to neutral, to dissatisfied, and, finally, to very dissatisfied. At USAID’s request, the ET also asked whether there was support from PT that would have been more useful for these CSOs based on their current situation. Stakeholders also volunteered additional information about their satisfaction, current needs and priorities, and hopes for the future relevant to answering this EQ. Stakeholders’ satisfaction is discussed below in specific areas emphasized by stakeholders: quality of training/TA, relevance of topics and their applicable to learning needs, ability to use skills on the job, follow-up received, materials or job aids, and cascade training/TA. 4.5.1 Findings Overall Satisfaction USAID staff interviewed were largely satisfied with PT’s performance. Two of the seven USAID interviewees felt they were not qualified to assess satisfaction based on their limited engagement with PT. The other five USAID interviewees noted the value of PT’s capacity building assistance to USAID, USAID CSO partners, and the IoAs in a variety of ways. Their satisfaction with PT’s performance was mainly in how PT helped the CSO direct grantees better fulfill their CEGO grants and how engaging with PT helped build the capacity of these CSOs, their sub-grantees, and the other CSOs with whom PT worked. Former PT staff noted the contributions of PT’s training and TA to building the capacity of CSOs and IoAs and to improved constructive engagements between citizens and government. They attributed improved engagement to the provision of PT support, which helped give them the capacity needed to successfully implement CEGO awards. Some former PT staff and consultants noted challenges in CSO/IoA satisfaction during the early phase of PT implementation. One noted, “At the beginning, beneficiaries didn’t appreciate TA support. They felt like it was a burden to them, but with time they came to realize the importance of the TA to the overall performance of their respective organizations.” All five CSO direct grantees asserted that they were generally satisfied with the support delivered by PT. Direct grantees stated they were satisfied with support for training of trainers, which they used to train their sub-grantees. While PT partners noted areas of PT support as particularly useful, they also offered critical comments on other areas of PT assistance. These areas are discussed in more detail below. Sub-grantees of direct grantees expressed satisfaction with the emphasis on compliance and follow-up from PT and their direct grantee partners (with PT support). Other CSO partners interviewed also identified areas where they were particularly satisfied with PT support. 26 All three IoAs noted satisfaction with PT. An ES key informant asserted that “Pamoja Twajenga helped us to a great extent…Pamoja Twajenga came to act as a mediator and at the same time a capacity builder to us.” The informant noted that “the program [grant with USAID] went on very smoothly because of Pamoja Twajenga.” In referring to the overall contribution to the ES, the key informant stated: “Pamoja Twajenga has been of significant help to us and to issues on which we might have spent a lot of resources. They have made it easier. For instance, conducting needs assessment and the like are the issues that need a lot of money because they would need commission of consultants. Our engagement, however, has capacitated the secretariat to conduct such kind of programs in a cheap way or using fewer resources than commissioning of consultants. Tools come when they are already developed to be used.” Specific areas of satisfaction with PT training and TA cited by key informants are described below. Areas of Satisfaction with PT Training and TA OCA Seventeen (17) interviewed CSOs (two direct grantees, 11 sub-grantees, and four other CSOs) ascribed their current organizational capacity to the OCA tool and process facilitated by PT. All other CSO partners (three direct grantees, four subgrantees, and six other CSO partners) noted PT’s contributions to their organizational development. CSO partners said that the OCA helped them to critically reflect on their situation and set benchmarks for organizational growth. They further said that training and TA on organizational development helped them improve their organizational systems and practices. They further appreciated the support given by PT in either reviewing or developing organizational manuals/guidelines. Interviewed CSOs attributed their improved financial accountability, accounting literacy skills, and compliance to internal control systems to PT’s TA and training on financial management. One sub-grantee CSO praised the value of the OCA to the CSO stating that “… the process helped us to identify capacity gaps that are hindering the growth of our organization. Through the facilitation of our partner, we prepared a plan to address identified gaps and developed key policies, guidelines, and strategies for the organization. We were able to develop financial manual/guidelines, a human resources manual, and a five-year strategic plan.” Reporting The two interviewed USAID direct grantees noted that training and TA on report writing skills enabled them to meet USAID reporting requirements, and the skills acquired are helping them beyond the CEGO program. All 11 interviewed sub-grantees noted that report writing skills have helped them to be in harmony with direct grantees by meeting set reporting standards. These skills have enabled them to prepare quality report to various stakeholders. All other CSOs (four) that worked with PT attributed their current ability to prepare good reports to PT training and TA. M&E Skills Two direct USAID grantees noted that their capacity and ability to prepare M&E plans has improved following their participation in M&E training and TA provided by PT. They further said that their capacity to collect and use data for reporting has also improved. All 11 interviewed sub-grantees similarly asserted that their current M&E capacity had grown based on cascaded M&E training by USAID direct grantees. Four other CSOs that worked with PT also attributed their ability to prepare M&E frameworks and plans to PT training. At the same time, however, five sub-grantees and one other CSO that worked with PT noted that their M&E skills were challenged by their inability to retain personnel trained by PT, particularly M&E staff. 27 Work Planning Two direct USAID grantees said that they are now able to prepare sound work plans and ensure that they are implemented in the wake of PT’s assistance. One key informant drew a contrast by saying, “Prior to PT training and TA, we used to have ad hoc plans, but now we are capable of preparing comprehensive plans for all our organization’s programs.” Seven out of 11 sub-grantees interviewed noted that their capacity to prepare work plans and ensure that implementation adheres to plan have also improved following the cascading of M&E training, mentoring, and coaching by direct grantees. All four other CSOs that worked with PT said that the skills acquired from PT training and TA are helping them to meet the requirements of their donors. Board Governance Six out of 11 interviewed sub-grantees said that TA on board governance helped make their boards more effective and independent from management. PT trained board members on their roles and responsibilities. It encouraged and supported CSOs to revise their boards and to separate organizational management functions from strategic leadership, which was also seen as ensuring board independence. Four other CSOs that worked with PT implemented similar board reforms. One direct USAID grantee voiced its appreciation to PT’s efforts to develop its board capacity noting that with PT’s support, the organization “has prepared a board charter, which stipulates, among other things, tenure of board members and performance appraisal of the board.” Strategic Planning One direct grantee praised assistance from PT to develop/review its strategic plans. Its view was this assistance made it more focused in its efforts to look for financial resources for its programs. Six out of 11 sub-grantees also noted that they were assisted to prepare strategic plans for their organizations. Two of four other CSOs that worked with PT likewise noted that having a strategic plan helped them to be more focused with their programs. Mentoring and Coaching Two direct grantees appreciated the coaching and mentorship they received from PT in the areas of financial management and report writing. They attributed their current capacity in financial management and report writing to PT training and TA. Ten (10) out of 11 interviewed sub-grantees appreciated coaching they received from USAID direct grantees. They said that the handholding they received from direct grantees, and occasionally from PT staff, has increased their competency and confidence. One sub-grantee now claimed, “We can confidently manage a USAID grant if by chance this happens in the future due to the capacity building we received in the course of implementing CEGO program.” Each of the 11 sub-grantees highlighted additional programmatic areas/skills that were impacted by PT training and TA. One of the other CSOs that worked with PT asserted that “the value of mentoring and coaching received was far important than any grant.” Another CSO said that “assistance in developing a resources mobilization strategy has made it possible for us to acquire funding from other donors.” The CSO saw this achievement as an extremely valuable contribution. Advocacy Training Five out of 11 interviewed sub-grantees noted that using new PT-taught advocacy skills helped them to obtain results while maintaining good working relationship with LGAs. Sub-grantees attributed their ability to transform their way of doing advocacy (from a confrontational approach to evidence-based dialogues) and empowering citizens to demand accountability from LGAs and services providers to PT training and TA on advocacy. 28 Issue-Based Dialogues Ten (10) out of 11 interviewed sub-grantees claimed that issue-based dialogues were instrumental in bringing desired changes in service provision in, for example, health, land, and natural resources management. Social Accountability Monitoring (SAM) Eight of 11 interviewed sub-grantees noted that trained and functioning SAM teams were helpful in improving accountability on the use of public resources and in making citizens more responsible for their development. One sub-grantee, for example, claimed, “Social accountability monitoring was useful at our organization as it made the villagers understand better their development programs and be able to demand accountability from their leaders.” NAOT noted that it improved its capacity to prepare effective communication materials for dissemination in the form of Citizen Reports to its beneficiaries and the general public as a result of activity TA on how to write success stories and integrate pictures into their reports. NAOT claimed that the new Citizen Report is the most sought after NAOT publication with high demand by the ordinary citizens and CSOs. Quality of Training/TA Two USAID direct grantees noted that they were generally satisfied with quality of training and TA delivered by PT. All 11 interviewed sub-grantees that addressed this question noted their satisfaction with how direct USAID grantees cascaded training to their organizations. They said that cascaded training was expertly delivered and noted that now sub-grantees were also able to practice these skills and even transfer these skills to others. Relevance of the Topic and Applicability to Learning Needs Key informants from two direct grantee partners noted that PT training and TA were relevant then and are still relevant today in that they were tailored to meet their specific organizational capacity needs such that they are now using the acquired knowledge and skills in executing their day-to-day organizational functions. Each of the 11 sub-grantees interviewed were of a similar opinion; these partners felt that PT training and TA were highly relevant and useful. They also noted that cascaded training and TA were relevant to their day-to-day operations and in delivering CEGO program outcomes. Five of the other CSOs that worked with PT also expressed satisfaction with the quality of training. Cascading of PT Training/TA Eleven (11) out of 11 sub-grantees noted that their partners with direct USAID grants (e.g., WiLDAF, LEAT, PELUM, TACOSODE, and NACOPHA) had worked with them to cascade PT training and TA to their organizations as well as through them to the community level. All were satisfied with this cascade training. One sub-grantee reported the areas in which it was particularly satisfied saying that “we received technical and financial support from PELUM ranging from financial assistance, work plan development, and organizational capacity assessment to identify areas of growth, capacity development plans to address weakness areas, coaching in those areas that had shortcomings, advocacy training, and communications—that is, training on how to write success stories and picture taking.” NAOT noted that it was satisfied with PT training and TA to the extent that it has decided to form a team of trainers to facilitate similar training to other auditors countrywide. Specific areas of dissatisfaction with PT training and TA identified by key informants are described below. 29 Areas of Dissatisfaction with PT Training and TA KIIs noted some cases of dissatisfaction which were related directly to PT training and TA delivery, while other KIIs found that dissatisfaction was more closely related to expectations or problems with their own organization’s internal capacity. Quality of Training/TA Two direct grantees and two sub-grantee CSOs were dissatisfied with the way PT delivered advocacy training, and two sub-grantees expressed dissatisfaction with how GESI training was delivered. These two CSOs attributed the problems to PT providing facilitators who were not conversant with the subject matter. Unmet Unrealistic Expectations Non-grantee CSO partners appear to have had unrealistic expectations about future grants that they thought would follow their participation in PT. Although most CSOs reported that they were satisfied with PT training and TA, four other CSOs that worked with PT noted that they were disappointed that their expectations of getting a grant award from USAID after they improved their organizational capacity through PT were not met. They felt that their involvement in PT training and TA processes would help them be competitive in winning a grant award from USAID, but these ambitions were not realized in their case. Attentiveness to Funding and Fundraising Needs One CSO partner noted that gaps persist in its organization in areas such as fundraising, how to manage resources, how to mobilize resources, and how to effectively engage with citizens with the government, even after PT support. For this CSO, the extent of PT support was not enough to meet their needs. Follow-up and Timing of Implementation Another CSO partner noted that it was not satisfied with the timing and adequacy of follow up visits by PT. It considered follow-up coaching and mentoring on training too limited and too late to support the training it received. This, in turn, contributed to its failure to implement issue-based dialogues effectively. Turnover/Loss of Institutional Knowledge One direct grantee noted it could not retain the whole of its Iringa based staff after CEGO program; without the grant, they could not meet payroll expenses. At the time of field work, the only remaining CSO staff in Iringa were working on a voluntary basis hoping that they might secure “some funding to continue the good work that CEGO program facilitated” in the region. Four of 11 sub-grantees also noted staff turnover as a challenge to sustainability and hence a source of dissatisfaction. They said that staff turnover affected them adversely by losing resource persons who left their organizations in search of “greener pastures.” Other CSO partners also noted staff turnover as a problem. One noted that it could not stop its trained M&E person from leaving for other opportunities. Extent of Time in PT NAOT was not satisfied with PT, as the time the organization partnered with PT was seen as too short. That NAOT “implemented only one-year activities—meaning that (they) did not complete the program” meant it was not able to benefit from capacity building to the extent expected. NAOT’s G2G award was correspondingly cut. Some CSO partners also felt that program duration was too short to be as effective as they needed it to be in capacity development. Satisfaction Levels by Scale During KIIs, key informants were asked to rate different aspects of PT training and TA on a scale from very satisfied, to satisfied, to neutral, to dissatisfied, and, finally, to very dissatisfied. Based on their 30 responses, the ET calculated the average level satisfaction with 19 different aspects of PT support. The results are presented in Table 2 by PT’s four different types of partners. Not all CSOs or IoAs had the experience and knowledge to rate each aspect of PT support, thus the total number of responses is less than the total number of PT partners for most of the categories. As seen in Table 2, key informants were satisfied with almost all areas of PT training and TA; in almost all areas, responses were satisfied, on average. There are four instances in which key informants were neutral about PT training and TA and 10 instances in which there were very satisfied. In all remaining cases, key informants were satisfied with the PT training and TA. Table 2: Satisfaction of Stakeholders with Training and TA Provided by Pamoja Twajenga Training/TA type IoAs Direct Grantees Other CSOs Sub￾Grantees 1. Advocacy Satisfied Satisfied Satisfied Satisfied 2. Audit preparedness Satisfied Neutral Satisfied Very satisfied 3. Board governance Neutral Satisfied Satisfied Satisfied 4. CDPs Satisfied Satisfied Very satisfied Very satisfied 5. Communications Satisfied Satisfied Satisfied Satisfied 6. Financial management Neutral Satisfied Satisfied Very satisfied 7. Fundraising support Satisfied Satisfied Satisfied Satisfied 8. Generation of action plans Satisfied Satisfied Satisfied Satisfied 9. GESI Satisfied Satisfied Satisfied Satisfied 10. Grants compliance checklist Satisfied Satisfied Satisfied Satisfied 11. Grants management compliance assessments Very satisfied Satisfied Very satisfied Satisfied 12. Grants management handbook Very satisfied Satisfied Satisfied Satisfied 13. M&E Satisfied Satisfied Satisfied Satisfied 14. OCAs Satisfied Satisfied Very satisfied Satisfied 15. Peer-to-peer engagement Satisfied Satisfied Satisfied Very satisfied 16. Quarterly and annual reporting Satisfied Satisfied Satisfied Satisfied 17. SAM Neutral Satisfied Satisfied Satisfied 18. Succession plans Satisfied Satisfied Satisfied Satisfied 19. Workplan development Satisfied Very satisfied Satisfied Satisfied Overall Satisfied Satisfied Satisfied Satisfied IoAs were particularly satisfied with PT’s support on grants compliance through the checklist and handbook. IoAs were especially interested in meeting the compliance needs of USAID to continue implementing their awards (which NAOT was not able to do). IoAs were less satisfied with PT’s support for board governance—as not relevant to them—and with PT’s support for financial monitoring and SAM. Direct grantees were less satisfied with PT’s support for audit preparedness and very satisfied with PT’s support for workplan development. Other CSO partners were very satisfied with three types of TA/training: CDPs, grants management compliance assessments, and OCAs. 31 Finally, sub-grantees were very satisfied with three types of training and TA provided by the direct grantees that they partner with, sometimes with the direct support of PT: audit preparedness, CDPs, financial management, and peer-to-peer engagement. 4.5.2 Conclusions PT stakeholders were broadly satisfied with PT’s performance and appreciated PT’s support for their organizational capacity development and advocacy/dialogues. Direct USAID grantee, sub-grantees, other CSOs that worked with PT, and IoAs were, on average, satisfied with the PT training and TA they received in all areas. Partners were less satisfied with capacity gaps identified though PT’s assessment processes that were left unaddressed given the limited time and resources available in the PT program. To a great extent, PT beneficiaries were satisfied with the content of training, the relevance of training to their organizational growth, the quality of after-training services, and its applicability to their day-to-day operations. PT’s USAID direct grantee partners felt that the training and TA were tailored to address the specific organization capacity gaps and practical skills needed for them to successfully implement their CEGO awards. Other PT CSO partners and sub-grantees were also largely satisfied with PT training and TA. Based on where they are now, CSO partners sought additional training and TA in a number of areas to further their organizational development. With hindsight, looking back on the assistance PT provided, CSO partners did not identify other aspects of capacity development support that they would have preferred to have received through PT. CSO responses to this question instead noted preferences for additional assistance in some of the same areas provided by PT or for continued assistance today. 4.5.3 Recommendations USAID should continue to consider adding capacity building assistance to awards with CSO and government partners that may face challenges implementing USAID awards. USAID can consider programming CSO awards and capacity building support through mechanisms like PT in the future or through more conventional Grants under Contract modalities. Although the ET did not attempt to compare and contrast different assistance modalities for CSO capacity building, the evaluation did not identify any benefits that accrued to CSO partners that came from the modality used by PT— being supported by a partner who, in turn, was under contract to USAID. USAID can consider future support through either type of mechanism. USAID should be clear in its communications with CSO partners and potential partners to avoid building unrealistic or unmet expectations about the prospects for CSO funding from USAID. 32 ANNEXES 33 ANNEX I: EVALUATION STATEMENT OF WORK Post-Performance Evaluation of USAID/Tanzania’s Capacity Development for Partners of Accountability (Pamoja Twajenga) activity implemented by Chemonics International 1. BACKGROUND INFORMATION This SOW is drafted to carry-out a post-performance evaluation of the Capacity Development for Partners of Accountability (Pamoja Twajenga) activity implemented by Chemonics International during the period 2013-2017. A. Identifying Information Strategy/Project/Activity Name Capacity Development for Partners of Accountability (Pamoja Twajenga) Implementer Chemonics International Contract Number 2013-2017: 621-C-13-00002 Total Estimated Ceiling of the Evaluated Project/Activity (TEC) 2013-2017: with modification $7,837,551 Life of Strategy, Project, or Activity 07-08-2013 to 07-07-2017 Active Geographic Regions Dodoma, Iringa, Morogoro, Mtwara, and Zanzibar Development Objective(s) (DOs) Primary: DO3: Effective Democratic Governance Improved Secondary: DO1: Tanzania Women and Youth Empowered Secondary: DO2: Inclusive broad-based economic growth sustained USAID Office Democracy, Human Rights, and Governance (DRG) COR Florida Henjewele B. Development Context Problem or Opportunity Addressed by the Activity Being Evaluated The Capacity Development for Partners of Accountability (CDPA) or Pamoja Twajenga activity implemented by Chemonics International started in July 2013 and sought to develop and enhance the capacity of partner organizations (both state and non-state) in support of the Mission’s Development Objective (DO) 3, which focuses on improving democratic governance, under the Country Development Cooperative Strategy (CDCS) approved in 2014. The purpose of the activity was to build the capacity of targeted civil society organizations (CSOs) and Government of Tanzania (GoT) Institutions of Accountability (IoAs) to engage citizen in government oversight. The CDPA formed part of USAID/Tanzania’s overall Supporting Accountability to Tanzania Citizens (SATC) Project that aimed to: • Strengthen targeted CSOs’ capacity to act as advocates and watchdogs; and • Strengthen GoT IoAs’ capacity to inform the public and to respond to civil society accountability concerns. The major relational gaps targeted were connectivity between citizens and their local governments; CSOs and citizens; civil society among itself; and between civil society and government-with a particular focus on IoAs and local-level government. The SATC Project complimented other elements of USAID’s 34 Democracy and Governance portfolio under DO3: “Improved Accountability and Oversight of Public Resources through Increased Citizen Participation.” Theory of Change The activity’s approach focused on both the demand and supply side of accountability; supporting CSOs on the demand side and IoAs on the supply side. The theory of change that underpinned the activity purports that international development partner pressure on the GoT alone is unlikely to result in major changes. But that if citizens are more aware, motivated, and empowered to hold their government more accountable at the community level on a consistent long-term basis, this may lead to demand-driven improvements in governance, increased pressure for more open political competition, and a healthier, more transparent exchange between citizens and their government. Problem the Activity Is Addressing According to statistics, there were an estimated 5,300 non-governmental organizations (NGOs) in Tanzania by 2010, most of which were classified as CSOs and almost half of them were registered under the NGO Act. CSOs were viewed by stakeholders as beneficial, indicating the positive role CSOs already play in Tanzanian communities. The GoT, which at times has interpreted advocacy as opposition politics, in March 2012 renewed its commitment to supporting civil society under the Open Government Partnership (OGP) plan 2012-2013. The positive development provided an opportune space to support increased advocacy and other activities advancing accountable government through the civil society sector. According to the 2015 USAID Civil Society Sustainability Index for Sub-Saharan Africa report, Tanzania has more than 20,000 CSOs focused on advocating for democratic reforms, human rights, and accountable governance, but the majority of them are characterized by weak internal governance structures, weak governing boards, poor financial management and accounting systems, lack of clear constituencies, and weak accountability to their beneficiaries, and, at best, have weak monitoring and evaluation frameworks in place. In addition, poor or non-existent strategic plans, a failure to implement activities in line with plans, and weak networks at the grassroots level are additional challenges CSOs in Tanzania face. Other gaps and weaknesses within the civil society sector include inconsistent funding, which forces CSOs to heavily depend on foreign donors. Critical areas where civil society would benefit from enhanced capacity include improved ability to effectively demand and undertake the oversight of public resources and relate to overall organizational effectiveness. The latter includes facets of organizational capacity including vision and purpose, planning and strategic management, monitoring and evaluation, activity design and implementation, constituency outreach, constructive engagement with the GoT, social accountability monitoring, external evidence￾based advocacy, human resources development, fundraising and local resource mobilization, financial management, and general CSO governance. Given these challenges and others, Tanzanian citizens’ civic participation in governance matters is too weak to provide significant and sustained public oversight of the GoT’s use of public resources, much less ensure accountability for how those resources are utilized. Overall, the achievement of good governance in Tanzania is constrained by a lack of political competition, a weak civil society, limited government capacity, minimal intra-government accountability and constraints to accessing information.8 There is a persistent disconnection between national-level democratic reforms and the day-to-day experience of Tanzanian citizens who report low level of satisfaction with governance and service delivery.9 Government accountability structures also do not always account for informal systems of service delivery and governance that sometimes parallel and even undermine the formal. In response, the GoT’s National Strategy for Growth and Reduction of Poverty (NSGRP II), known in Kiswahili acronym as MKUKUTA II, 8 ARD Inc, and Democracy International Inc, Democracy, and Governance Assessment of Tanzania. Final Report, 2010. 9 ARD Inc, and Democracy International Inc, Democracy, and Governance Assessment of Tanzania. Final Report, 2010. 35 set local partnership, people’s participation, and community engagement as guiding principles in sustaining good governance and fighting corruption. To coordinate implementation of all government reforms, the GoT established the Good Governance Coordination Unit (GGCU) in the Office of the President. While these bodies and strategies have laudable mandates, they are typically under-funded and their ability to be effective and independent is limited. USAID’s overall SATC Project will target the supply of public resource oversight with direct support mechanisms to GoT IoAs, some of which were established by the GoT under the MKUKUTA framework. With regards to IoAs, USAID also has invested directly in the Public Procurement Regulatory Authority (PPRA), National Audit Office of Tanzania (NAOT), and the Ethics Secretariat (ES). The partnership with the Ministry of Public Service and Good Governance (MPSGG) in Zanzibar did not materialize as the Ministry did not meet the set criteria. USAID/Tanzania seeks to support a strong civil society sector that empowers citizens to have oversight on their government. Links to USAID/Tanzania’s Country Development Cooperation Strategy (CDCS) Although the design of the Capacity Development for Partners of Accountability (CDPA) activity and USAID’s support to Chemonics precedes the development and approval of the current CDCS, the activity is a critical component of the Mission’s approach to achieving DO 3: Effective Democratic Governance Improved, Intermediate Result (IR) 3.1 Citizen engagement made more effective; sub-intermediate results (sub-IR) 3.1.1 Citizen access to information increased; sub-IR 3.1.2 Institutional capacity of CSOs increased; and sub-IR 3.1.3 CSOs successful oversight at national and sub-national levels advocated and conducted. Citizen engagement is made more effective when citizens are able to access information produced by IoAs (IR 3.3). By promoting individual civic rights, including women and youth rights, IR 3.1 Citizen engagement made more effective, also supports results achievement under IR 1.1 Gender equality and youth inclusion; and IR 1.2 Health status improved. In supporting citizen oversight of local government, citizen engagement helps to improve service delivery, in line with IR 3.2 Government delivery of services improved, and IR 3.3 Government accountability increased. In addition, CSOs funded under IR 3.1 worked in priority USAID sectors of education, agriculture, and natural resource management in view of advancing achievement under IR 1.3 Lifelong learning skills improved, IR 2.2 Agricultural productivity and profitability increased in targeted value chains, and IR 2.3 stewardship of natural resources improved. Capacity Development for Partners of Accountability Scope The CDPA activity was designed to assist citizens in understanding the way governance affects them, what role they can play in improving governance, and be equipped with easily accessible and user-friendly information on their government’s performance. To achieve these aims, USAID issued five direct awards to CSOs to support their endeavors in improving accountable governance in the sectors of agriculture and food security, health, and natural resources management and to improve their linkages to smaller local CSOs and their citizen constituents, monitor government accountability, and improve flows of information with IoAs. Specifically, these awards to CSOs were issued at beginning of 2013 to support them in their advocacy efforts, and their work with sub-grantees at the grassroots level to: a. Share information with citizens and gather evidence about both government performance and citizen priorities; and b. Develop effective advocacy messages for appropriate decision-making levels and strengthen their own general understanding of IoAs. Concurrently, USAID provided support to three GoT institutions of accountability to help them strengthen their working relationships with civil society and to improve the accessibility and utility of information they intended to make public. Increased access to public information is important because it allows citizens to play their role in ensuring that they are being represented in the policy-making and service delivery of their government. Even more importantly, it permits citizens to learn about government plans or actions that might be critical to their own personal interests. 36 The purpose of the CDPA Activity (also known as Pamoja Twajenga) was to provide selected Tanzania CSOs and GoT IoAs with an array of support (such as technical assistance, training, logistics, etc.) in order for these targeted organizations to: achieve sustainable and significant impact as advocates in their respective sectors; to be accountable to citizens as their constituents; to manage resources (donor or other); and to perform meaningful watchdog functions regarding use of public resources. CDPA’s primary objectives included: 1. Strengthen the capacity of USAID/Tanzania partner CSOs to serve as strong advocacy organizations, well-networked at the grassroots level and able to inform and equip citizens to engage with government; 2. Provide technical support to USAID in its management of civil society grant activities; 3. Strengthen the capacity of GoT IoAs to package information; employ a cohort of civil servants with a common understanding of ethics, procurement, auditing, and good governance rules; and, generally, perform activities under their assistance agreement with USAID/Tanzania; and 4. Facilitate constructive engagement between CSOs and IoAs. The CDPA activity was designed within the context of USAID Forward; an ambitious reform effort to change the way USAID does business with partnerships, an emphasis on innovation and a relentless focus on results. It is a comprehensive package of reforming seven key areas: reforming implementation and procurement, strengthening talent management, rebuilding policy capacity, strengthening monitoring and evaluation, rebuilding budget management, promoting science and technology, and encouraging innovation. USAID/Tanzania seeks to further strengthen the capacity of local civil society and private sector actors to become more effective and influential development partners in their communities and in Tanzania. The Mission also acknowledges that well developed capacities will help local CSOs to become eligible and more effectively compete for resources, as well as be more accountable in the programmatic and financial management of projects and programs they will implement. USAID defines capacity development as any action that improves effectiveness of individuals, organizations, networks, or systems-including organizational and financial stability, program service delivery, program quality and growth. C. Intended Results of CDPA Activity • Result 1.1: Organizational development of selected CSOs and non-DRG organizations supported; • Result 1.2: Development/strengthening of issue-based dialogue of selected CSOs and non-DRG organizations supported; • Result 2.1: Grants management aspects of organizational development in USAID grantees supported; • Result 2.2: USAID grantees demonstrated impact and success supported; • Result 3.1: Organizational Development of targeted IoAs supported; and • Result 3.2: Issue-based dialogue between IoAs, CSOs, and citizens supported. 37 Figure 1: Envisioned Structure of USAID’s Overall Accountable Government Activity (Supporting Accountability to Tanzania Citizens) Through the Contractor ILLUSTRATION OF PAMOJA TWAJENGA RESULTS FRAMEWORK D. Approach and Implementation Under the CDPA activity, Chemonics International supported three IoAs that were receiving direct funding from USAID as well as five CSOs receiving direct funding, and 10 CSOs that were not receiving funds from USAID. In addition, Chemonics worked with sub-grantees of the CSOs they were supporting. Chemonics’ support targeted specialized technical assistance in the form of training, coaching, mentorship, and on-the-job training. Chemonics International used participatory approaches that positioned local partners as an integral part of activities. This included mentoring them and pairing them to the extent possible with organizations and companies with higher capacity levels to ensure sharing of knowledge and best practices, ownership of interventions, and an overall leadership role in activities. While specific areas of interventions were to be determined by the particular needs of the CSO or IoA, through Organizational Capacity Assessments (OCA), capacity building in general focused at the individual/team and at the organizational level. These include building the knowledge and skills of individuals and teams, as well as strengthening structures, tools, policies, procedures, and practices. The contractor was expected to enhance the capacity of these CSOs and IoAs in promoting more efficient, cost-effective, accountable, transparent, and technically proficient organizations that are accountable to their respective constituents. Additionally, the contractor was expected to work on improving linkage between the direct grantees and their sub-grantees (see Figure 1 below), as well as provide management support to USAID/Tanzania’s Democracy, Human Rights, and Governance (DRG) team with grants management (which varied in terms of size, length, and programmatic areas of focus), and build the capacity of recipient CSOs to issue and be accountable for grants issued to sub-grantees. In general, CDPA employed a mix of traditional and innovative capacity building approaches. The activity engaged in strategic partnerships with established Tanzanian organizations through indefinite quantity subcontracts (IQSs) to leverage expertise in communications, strategic planning, and financial and audit best practices. CDPA also engaged experts to provide advocacy and organizational lessons learned, 38 advocacy expertise, and best practices through quarterly awareness-raising sessions and training (list of grantees and sub-grantees is annexed). In carrying out its capacity building efforts, CDPA employed a wide range of participatory methods, including document management system and a Pamoja Twajenga quality certification, one-on-one mentoring and coaching, peer-to-peer learning, roundtable discussion, an “achieve, innovate, measure, and share assessment,” and use of pool of skilled trainees and service providers. E. Existing Documents Several documents are available for review for this post performance evaluation of CDPA. These include but not limited to (i) CDPA SOW, (ii) CDPA annual work plans for the duration of the activity, (iii) CDPA M&E Plan, (iv) CDPA quarterly progress reports, (v) CDPA annual reports (vi) CDPA Final Report, (vii) CDCS 2014/19, (viii) SATC PAD, (ix) SOL, (x) CDPA Achieve, Innovate, Measure, and Share (AIMS) reports, (xi) Organizational Capacity Assessment Reports, (xii) Government Institutions Capacity Assessment (GiCA), (xiii) Capacity Development Plans, (xiv) Annual Report of CEGO grantees, (xv) Annual Work Plans of CEGO Grantees, (xvi) Annual Work Plans of IoAs, (xvii) Annual Reports of IoAs, (xviii) IoAs Grant Agreements, (xix) OCA, (xx) Re-OCA reports, and (xxi) CEGOs Grant Agreements. II. EVALUATION RATIONALE A. Evaluation Purpose CDPA was a unique mechanism that had a standalone capacity building/technical assistance expertise to support USAID’s Tanzanian implementing partners, including actors of both demand and supply side of accountability. While some of USAID’s direct support to CSOs and IoAs ended in 2017, the impact or performance of these investments has not been evaluated. Moreover, the unique nature of CDPA activity design and technical assistance delivery approach calls for post-performance evaluation of CDPA activity in general and Chemonics in particular. Recognizing the unique nature of this activity, USAID/Tanzania determined the importance of evaluating this approach and documenting lessons learned and best practices in order to assess the adequacy and comparative advantage of the approach in contributing to achievement of the DO. The evaluation will help to document the rationale for the approach, what worked well and what did not, what has been sustained, and where to improve if the Agency would like to design a similar activity. It is expected that findings and recommendations will inform future programming at the Mission and suggest approaches/strategies that could inform the Agency at large. B. Audience and Intended Uses The results of this evaluation activity will be used by USAID/Tanzania across sectors to inform decisions and design of the forthcoming CDCS and future programming. The intended users of the evaluation include USAID/Tanzania Development Objective (DO3) Team, DRG office in particular, and the Agency at large. C. Evaluation Questions The intended scope of the CDPA activity with USAID covered four objectives: a) to strengthen the capacity of USAID/Tanzania partner CSOs to serve as advocacy organizations, well-networked at the grassroots level and able to inform and equip citizens to engage with government; b) provide technical support to USAID in its management of civil society grant activities; c) strengthen the capacity of government of Tanzania IoAs to package information, employ a cohort of civil servants with a common understanding of ethics, procurement, auditing, and good governance rules; and d) perform activities under their assistance agreement with USAID/Tanzania and facilitate constructive engagement between CSOs and IoAs. 39 Within this context, the following evaluation questions seek to understand performance related to activity objectives: 1. To what extent did the intended outcome of improved and constructive citizen/government engagement occur over the course of the program? 2. Did the technical support provided to USAID staff meet the needs and priorities of the Mission related to the management of civil society grant activities? USAID is interested in understanding what worked well and what didn’t.10 The evaluation team should focus on the contractor’s support in the management of civil society grants activity. 3. Did the activity meet its targets in strengthening the capacity of CSOs and GoT IoAs? USAID is interested in understanding the relationship between capacity development inputs and how this might be related to relevant CSOs and GoT performance outcomes. In addition, USAID is interested in any insight into the sustainability of activity investments in terms of CSOs and GoT performance. Capacity includes but is not limited to IoAs ability to package information, employ a cohort of civil servants with common understanding on ethics, procurement, audit, and good governance rules, work planning, monitoring and evaluation, and performance of activities under the assistance agreement with USAID/Tanzania. CSOs capacity relates to governance, administration, human resources management, financial management, organizational management, program management, and project performance. 4. To what extent have the actions supported or encouraged through the CDPA activity been sustained by beneficiary IoAs and CSOs? USAID is interested in understanding how peer-to-peer learning, om-the-job trainings, communities of practices between CSOs, mentoring, coaching in work planning, gender and social inclusion, Social Accountability Monitoring (SAM) are working. 5. Are the stakeholders satisfied with the performance of the contractor in training and providing technical assistance to CSOs and IoAs? Why and why Not? III. EVALUATION DESIGN AND METHODOLOGY A. Evaluation Design This Post-Performance Evaluation is the only external evaluation being conducted for the life span of the contract since its commencement in July 2013. Unlike mid-term performance evaluation or end-line performance evaluation, post-performance evaluation documents lessons learned and how interventions results are sustained after the activity has come to an end. Findings and recommendations of this post-performance evaluation will therefore be vital in informing USAID of the appropriateness and relevance of CDPA approach in the implementation of the activity, documenting lessons learned, what works and what did not work, future design and programming as the Mission envisions developing a new country development strategy. The evaluation team will design and use appropriate methods of data collection to gather both qualitative and quantitative data. The evaluation team must ensure that data collection methodology and the corresponding data sources will generate the highest-quality and most credible evidence corresponding to the evaluation purpose and questions. As part of capacity building, the team will design and conduct an evaluation that maximizes participation by USAID staff, IoAs staff, and any Chemonics staff that is still in country and CEGO grantees staff, namely TACOSODE, LEAT, NACOPHA, and PELUM. However, efforts should be made to ensure objectivity and 10 Some of the support provided included certification and approval process such as annual work plan approval check list, quarterly reports review checklist, annual report approval checklist, accrual approval checklist, data quality assurance by IoAs and CSOs, data entry into PIRS, field visits, and reporting during PPR. 40 validity of inputs and outputs. USAID Mission staff should be included as participants given the complexity of CDPA while assuring maximum objectivity and neutrality. B. Data Collection Methods The evaluation team is required to consider a range of data collection methods and approaches for collecting information that is required to achieve the evaluation purpose. Proposed data collection methodologies will be discussed with and approved by USAID/Tanzania prior to the start of the evaluation. USAID proposes the following as some of the methods: Desk review of primary and secondary data, face￾to-face interviews, key informant interviews, focus group discussions, questionnaires, direct observation, telephone interview with staff/organizations whose agreements have ended, and, if possible, surveys. C. Data Analysis Methods The report should include both qualitative and quantitative analyses of the achievements and shortcomings of the activity in relation to the objectives and targets. The analyses should also cover the output and outcome indicators for the contract, as well as the achievement of the approach CDPA activity operationalized. All information and data shall be disaggregated, as much as possible, by the type of organization- IoA versus civil society organization, type of capacity building offered, location – national, regional, or local level, beneficiaries of capacity development disaggregated by gender, and age. This suggested evaluation design identifies potential sources and approaches to collecting and analyzing. It is offered as a basis for future discussion by the evaluation team in the deliverables listed in Section IV. Questions Suggested Data Sources (*) Suggested Data Collection Methods Data Analysis Methods 1. To what extent did the intended outcome of improved and constructive citizen/ government engagement occur over the course of the program? Documents (including: annual reports, quarterly progress reports, annual evaluation reports, AIMS reports, performance monitoring data, third party surveys, national statistics, end of project performance report), project staff, LGAs, ministries, stakeholders, expert knowledge, beneficiaries, etc. Desk Review, key informant interviews, focus group discussion, phone calls, survey/ questionnaire. Statistical analysis of performance monitoring data, third party surveys, and national statistics, Recurrence analysis of themes in collected qualitative information. Requested level of disaggregation – gender, age, level of engagement – district, regional or national level, and grantee type, etc. 2. Did the technical support provided to USAID staff meet the needs and priorities of the Mission related to the management of civil society grant activities? USAID is interested in understanding what worked well and what didn’t. 11 The evaluation team should focus on contractor’s support in the management of civil society grants activity. Documents, including annual Contractors performance assessment reports, AIMS reports, COR’s responses to AIMS, grantees AORs, CO, Mission G2G Technical team, third party evaluations. Desk review, face-to-face interviews/key informant interviews, focus group discussion, survey/questionnaires, phone calls. Recurrence analysis of themes in collected qualitative information. Requested level of disaggregation – services provided, type of grantee, location (district, region), etc.… 11 Some of the support provided included assistance on the certification and approval process such as annual work plan approval check list, quarterly reports review checklist, annual report approval checklist, accrual approval checklist, data quality assurance by IoAs and CSOs, data entry into PIRS, field visits, and reporting during PPR. 41 Questions Suggested Data Sources (*) Suggested Data Collection Methods Data Analysis Methods 3. Did the activity meet its targets in strengthening the capacity of CSOs and GoT IoAs? USAID is interested in understanding the relationship between capacity development inputs and how this might be related to relevant CSOs and GoT performance outcomes. In addition, USAID is interested in any insight into the sustainability of activity investments in terms of CSOs and GoT performance. Capacity includes but is not limited to IoAs’ ability to package information, employ a cohort of civil servants, work planning, monitoring and evaluation, and performance of activities under the assistance agreement with USAID/Tanzania. CSOs’ capacity relates to governance, administration, human resources management, financial management, organizational management, program management, and project performance. Documents, (including quarterly and annual progress reports, third party evaluation, third party survey, national statistics, AIMS reports, CORs responses to AIMS, grantees AORs, Mission G2G technical team, payroll), project staff, LGAs, zonal offices, stakeholders, beneficiaries, website, citizens, etc. Desk review, face-to-face interviews with IoAs, grantees AORs, CSOs, physical verification of manuals, guidelines/ systems/tools, and staff payroll. Statistical analysis of performance monitoring data, third party surveys, third party evaluations, and national statistics. Recurrence analysis of themes in collected qualitative information. Requested level of disaggregation – type of capacity building, type of information, gender, age, location (district, region), etc. 4. To what extent have the actions supported or encouraged through the CDPA activity been sustained by beneficiary IoAs and CSOs? USAID is interested in understanding how peer-to￾peer learning, communities of practices between CSOs, mentoring, coaching in work planning, gender and social inclusion, Social Accountability Monitoring (SAM) are working. Documents, including annual Contractors quarterly and annual reports, AIMS reports, COR’s responses to AIMS, grantees AORs, , third party evaluations Desk review, face-to-face interviews/key informants’ interview, focus group discussion, survey/ questionnaires, phone calls. Recurrence analysis of themes in collected qualitative information. Requested level of disaggregation – type of organization, type of action, location (district or regional). 42 Questions Suggested Data Sources (*) Suggested Data Collection Methods Data Analysis Methods 5. Are the stakeholders satisfied with the performance of the contractor in training and providing technical assistance to CSOs and IoAs? Why and why Not? Documents including AIMS reports, AORs responses to AIMS, quarterly and annual progress reports, third party evaluation, third party survey, national statistics, monitoring and evaluation reports), project staff, LGAs, zonal offices, stakeholders, beneficiaries, citizens, etc. Surveys, desk review, phone calls, physical verification of information. Statistical analysis of a, third party surveys, third party evaluations, and national statistics. Recurrence analysis of themes in collected qualitative information. Requested level of disaggregation – type of capacity building, type of information, gender, age, location (district, region), etc. Notes: (*) It is acceptable to include data sources that do not need to be collected but may be analyzed by the evaluation team. In planning for and preparing the Evaluation SOW it is best practice to examine available data sources especially performance monitoring data D. Methodological Strengths and Limitations Limitations: language constraints as some beneficiaries may have minimal English language capacity. This will require a Kiswahili language speaker in the team or a translator. There are five target regions for this activity, and 10 Districts, with three IoAs, five CSOs direct grantees and 10 CSOs non-grantees. All three IoAs, five direct grantees, and 10 non-grantees should be reached, and all five regions covered. However, due to time constraints, a manageable number of districts, upon approval by USAID, should be selected for evaluation. IV. DELIVERABLE AND REPORTING REQUIREMENTS A. Deliverable 1. Inception report: The inception report with a draft evaluation work plan/schedule indicating which steps/activities must occur and when to ensure that the evaluation team fully understands the assignment approach. The work plan to be completed by the lead evaluator is expected to be received by USAID’s Contracting Officer Representative (COR) within five business days after signing the contract. The work plan and schedule should be approved by USAID/Tanzania prior to implementation. The work plan must include: 1) the anticipated schedule for design and implementation of the evaluation and logistical arrangements; and 2) a list of the members of the evaluation team, delineated by roles and responsibilities. 2. Evaluation design and methodology: Within two weeks after approval of the work plan, the evaluation team must submit to the COR an evaluation design (which will become an annex to the Evaluation report). The evaluation design will include: 1) a detailed evaluation design matrix that links the Evaluation Questions in the SOW to data sources, methods, and the data analysis plan; 2) draft questionnaires and other data collection instruments or their main features; 3) the list of potential interviewees and sites to be visited and proposed selection criteria and/or sampling plan (must include calculations and a justification of sample size, plans as to how the sampling frame will be developed, and the sampling methodology); 4) known limitations to the evaluation design; and 5) a dissemination plan. USAID offices and relevant stakeholders will take up to five business days to review this submission and consolidate comments through the COR. Once the evaluation team receives the consolidated comments on the initial evaluation design and work plan, they are expected to return with a revised evaluation design and work plan within five business days. 43 3. An in-brief meeting at the Mission to present tentative work plan/schedule and methodology and get to know the evaluation team. Within three days of arrival in Dar es Salaam, Tanzania, the evaluation team will have an in-briefing with the DRG, EG, Health, OFM, OAA, and Strategic Planning and Activity Support Office for introductions and to discuss the team’s understanding of the assignment, initial assumptions, evaluation questions, methodology, and work plan, and/or to adjust the Statement of Work (SOW), if necessary. The in-briefing should take place after the evaluation team has conducted a desk review or examined secondary data and before commencement of in country data collection/interviews. 4. An out-brief meeting: The out-brief meeting will be held approximately three days before departure of the evaluation team lead to discuss the status of data collection and preliminary findings. A well-developed power point summarizing key findings, issues, and recommendations is expected. The debrief date and time will be scheduled as agreed upon during the in-briefing meeting. 5. The draft evaluation report should be consistent with the guidance provided below and must be completed prior to the Team Leader’s departure from Tanzania. The written report must clearly describe findings, conclusions and recommendations (using the report format provided in “Reporting Requirements” below) and must address each of the questions identified in the SOW and any other issues the team considers having a bearing on the objectives of the evaluation. Any such issues can be included in the report only after consultation with USAID. The submission date for the draft evaluation report will be determined in the evaluation work plan. 6. Once the initial draft evaluation report is submitted, the Mission including DRG team, PO and CCIR members will review and provide comments on the initial draft report within 10 working days of submission, after which point the COR will submit the consolidated comments to the evaluation team. The evaluation team will then be asked to submit a revised final draft report 10 business days after receipt of Mission comments, of which will be reviewed by USAID. The evaluation team lead will submit the Final draft report within five business days of its submission. 7. A final report: The final report that incorporates all responses to Mission’s comments and suggestions should be completed within five working days after USAID/Tanzania provides its feedback on the draft report. The evaluation team leader will then submit the final report to the COR. All project data and records will be submitted in full and should be in electronic form in easily readable format, organized and documented for use by those not fully familiar with the intervention or evaluation, and owned by USAID. The report format should be restricted to Microsoft products and 12-font type must be used throughout the body of the report. The report must not exceed 30 pages, excluding both references and annexes. B. Reporting Requirements 1. Final Report Format: The findings from the evaluation will be presented in a draft report at a full briefing with USAID/Tanzania and possibly at a follow-up meeting with key stakeholders. The executive summary should be 2-5 pages in length and summarizes the purpose, background of the project being evaluated, main evaluation questions, methods, findings, conclusions, and recommendations and lessons learned (if applicable). In short, the executive summary concisely states the most salient findings, conclusions and recommendations. The evaluation final report should include: • An abstract (1 page); • Executive summary – concisely states the most salient findings, conclusions, and recommendations (not less than 2 pages and not more than 5 pages); • List of acronyms (1 page); 44 • Table of contents (2 pages); • Introduction – purpose, audience, and synopsis of task (2 page); • Background of the local context and brief overview of CDPA activity in Tanzania, USAID CDCS and other related strategies, priorities, projects/activities being evaluated; the evaluation purpose and main evaluation questions; the methodology or methodologies; the limitations to the evaluation; findings, conclusions, and recommendations; issue; future directions (3 pages); • Design and methodology – The evaluation methodology shall be explained in the report in detail. Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (e.g., selection bias, recall bias, unobservable differences between comparator groups, etc.) (3 pages); • Findings, conclusions, and recommendations in separate sections: for each objective area (20- 30 pages); • Issues: provide a list of key technical and/or administrative issues identified (2-3 pages); • Future Directions/Recommendations based on gaps or innovation model to be considered when designing a similar activity (3-4 pages); • References (including bibliographical documentation, stakeholders’ meetings, key informant interviews, and focus group discussions) (2 page); • Annexes, which shall include: o The Evaluation SOW; o Any statements of difference regarding significant unresolved differences of opinion by funders, implementers, and/or members of the evaluation team; o All data collection and analysis tools used in conducting the evaluation, such as questionnaires, checklists, and discussion guides; o All sources of information, properly identified and listed; o Signed disclosure of conflict of interest forms for all evaluation team members, either attesting to a lack of conflicts of interest or describing existing conflicts of; and o Summary information about evaluation team members, including qualifications, experience, and role on the team. For more detail, see “How-To Note: Preparing Evaluation Reports” and ADS 201maa, USAID Evaluation Report Requirements. An optional evaluation report template is available in the Evaluation Toolkit. In accordance with ADS 201, the contractor will make the final evaluation reports publicly available through the Development Experience Clearinghouse within three months of the evaluation’s conclusion. 2. Criteria to ensure quality of the Evaluation Report: Per ADS 201maa, Criteria to Ensure the Quality of the Evaluation Report, draft and final evaluation reports will be evaluated against the following criteria to ensure the quality of the evaluation report.12 • Evaluation reports should represent a thoughtful, well-researched, and well-organized effort to objectively evaluate the strategy, project, or activity. • Evaluation reports should be readily understood and should identify key points clearly, distinctly, and succinctly. 12 See ADS 201mah, USAID Evaluation Report Requirements and the Evaluation Report Review Checklist from the Evaluation Toolkit for additional guidance. 45 • The Executive Summary of an evaluation report should present a concise and accurate statement of the most critical elements of the report. • Evaluation reports should adequately address all evaluation questions included in the SOW, or the evaluation questions subsequently revised and documented in consultation and agreement with USAID. • The evaluation report should include the scope of work as an annex. All modifications to the scope of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology or timeline need to be agreed upon in writing by the technical officer. • Evaluation methodology should be explained in detail and sources of information properly identified and will be included as an Annex in the final report. • Limitations to the evaluation should be adequately disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). • Findings, conclusions, and recommendations: Should make up the main body of the report, synthesizing what was learned during the evaluation and presenting it in an easy to understand and logical fashion. • Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or simply the compilation of people’s opinions. • Conclusions synthesize and interpret findings and make judgments supported by one or more specific findings. • Findings and conclusions should be specific, concise, and supported by strong quantitative or qualitative evidence. • If evaluation findings assess person-level outcomes or impact, they should also be separately assessed for both males and females. • Recommendations, if applicable, are specific actions the evaluation team proposes be taken by program management that are based on findings and conclusions. The reader should be able to discern what evidence supports the conclusions and recommendations. Whenever possible, data should be presented visually in easy to read charts, tables, graphs, and maps to demonstrate the evidence that supports conclusions and recommendations. All graphics must have a title, be clearly labeled, and include a caption. 3. Other Requirements: All quantitative data collected by the evaluation team must be provided in machine-readable, non-proprietary formats as required by USAID’s Open Data policy (see ADS 579). The data should be organized and fully documented for use by those not fully familiar with the project or the evaluation. USAID will retain ownership of the survey and all datasets developed. All modifications to the required elements of the SOW of the contract/agreement, whether in technical requirements, evaluation questions, evaluation team composition, methodology, or timeline, need to be agreed upon in writing by the COR. Any revisions should be updated in the SOW that is included as an annex to the Evaluation Report. V. TEAM COMPOSITION The evaluation team should be constituted by four members, with a combination of Tanzanian/East African and international consultants. It is expected that the team will include experts on capacity building; 46 advocacy and networking; monitoring and evaluation; business management and finances; and gender inclusion with significant evaluation experience with USAID. The team leader should be an expatriate and have a minimum of 10 years working in international development programs—preferably governance sector, and work experience on capacity building in the areas of advocacy, organizational development, monitoring and evaluation, financial management, and CSOs engagement. He/she must demonstrate substantial experience in evaluating complex capacity building interventions of governance nature. He/she should also be expert on gender and development. He/she should demonstrate experience and capacity in writing succinct and elaborate evaluation reports. Prior work experience in Africa region is an added advantage. Tanzania and/or East Africa experience and Kiswahili speaking ability are preferred. The Monitoring and Evaluation expert should be a Tanzanian national and have five to 10 years of experience in international development evaluation methodology and practice. He/she must be a monitoring and evaluation expert, focusing on governance-related issues, coupled with strong knowledge, skills, and experience in qualitative and quantitative evaluation designs and tools as well as skills in capacity building program monitoring and evaluation. The candidate would have a clear understanding of USAID's ADS 200 and 201, especially the sections on evaluation as he/she would also ensure evaluation team adherence to USAID’s evaluation policies and best practices. He/she should have work experience and evaluated a capacity building program targeting civil society organizations and government institutions of accountability. Tanzania and/or East Africa experience and Kiswahili speaking ability are preferred. Capacity development expert should be a Tanzanian/East African with five to 10 years of experience on capacity building programs focusing on advocacy skills, organizational development, planning and strategy management, program design and management, monitoring and evaluation, financial management, fundraising and local resources mobilization, civil society governance, human resources development, communication and outreach, and networking for civil society organizations and government institutions of accountability. Citizen Engagement/ Participation Expert should be a development expert with five years of experience on citizen-government engagement, with a practical knowledge of working with government’s institutions of accountability and civil society organizations. Knowledge of citizens’ engagement approaches is desired. The expert would contribute to the team’s better understanding of the context of citizen engagement with the government’s institutions of accountability and be fluent in Kiswahili. Logistician: A national experienced in logistics and events’ planning is preferred. She/he will be responsible and working closely with the Team Lead, under his/her guidance. He/she will be responsible for setting meetings, appointments, book tickets, and arranging transport and travels to and from Tanzania and in-country travels. All team members will be required to provide a signed statement attesting to a lack of conflict of interest or describing any existing conflict of interest. The evaluation team shall demonstrate familiarity with USAID’s evaluation policies and guidance included in the USAID Automated Directive System (ADS) in Chapter 200. VI. Evaluation Schedule This is a post-performance evaluation, in which case four out of five direct civil society grantees, including CDPA contract and two of three IoAs awards came to completion in September 2017. While the most activities have come to a completion, they are yet to be closed as such, it is critically important to prioritize the evaluation. USAID recommends the evaluation commences in beginning September 2018. Below are samples of illustrative schedule and estimated level of efforts (LOE) in days by activity for a team of five. 47 Table 1: Illustrative Schedule* Timing (Anticipated Months or Duration) Proposed Activities Important Considerations/Constraints June 2018 Finalization of SOW DRG Office July 3-10, 2018 CCIR team review SOW USAID CCIR July 25, 2018 Approval SOW PO July 25, 2018 SOW shared with Data for Development USAID PO July 25-August 8, 2018 Preparation of the work plan and evaluation design Data for Development August 9-15, 2018 USAID review of the work plan and evaluation design Take into account availability in the Mission or Washington OU August 15-24, 2018 Travel, evaluation design, and preparations for data collection, including local approvals by authorities Take into account visa requirements (if an expatriate team is being mobilized) August 27, 2018 In-Briefing - August 27-September 14, 2018 Field data collection and initial data analysis Take into account the number of sites, methods, sectors, etc. September 17, 2018 Out-brief Take into account the number of sites, methods, sectors, etc. September 17-28, 2018 Report writing Take into account the number of sites, methods, sectors, etc. October 1-12, 2018 USAID review of Draft Report and comments Take into account availability in the Mission or Washington OU October 12-26, 2018 Incorporate USAID comments and prepare Final Report - October 26, 2018 Submission of Final Report to USAID - *To be reviewed for Data for Development delivery 48 Table 2: Roles Estimated LOE in Days by Activity for a Team of Five (to be modified by contractor as necessary) Task LOE for Expat Team Lead LOE for Local M&E Specialist LOE for Local TZ/East African Capacity Building Specialist LOE for Local Citizen Engagement Specialist LOE for Local Logistician Total LOE in Days Document review/desk review/work planning (evaluation design remote or in-country) 5 days 4 days 3 days 3 days - 15 Preparations for travel and organizing data collection (contracting translators, vehicles, etc.) 0 days 0 days 0 days 0 days 18 days 18 In-brief, Evaluation Design (including meetings with USAID) 3 days 2 days 2 days 2 days 2 days 11 Preparations for data collection (scheduling) 2 days 1 days 1 days 1 days 2 days 7 Data collection days by method by site 10 days 10 days 10 days 10 days 10 days 50 Data analysis 4 days 4 days 4 days 4 days 0 16 Out-Briefing 1 day 1 day 1 day 1 day 1 day 5 Draft report to USAID 10 days 10 days 10 days 10 days 2 days 42 USAID review and comments 0 0 0 0 0 0 Final report after USAID comments 10 days 10 days 10 days 10 days 10 days 50 Total 45 42 41 41 45 214 VII. EVALUATION MANAGEMENT Given the scope of this work and the importance of the findings and recommendations, ME&A, Inc. is expected to undertake quality reviews of all documents prior to submission to USAID. A. Logistics ME&A, Inc. will be responsible for all off-shore and in-country logistical support including cost for transport for international and in-country travel (including vehicle rentals), hotel bookings, working/office space, computers, printing and photocopying. However, a few implementing partners, who are still in operations and USAID, will facilitate some logistics. This includes arranging and scheduling meetings with GoT’s institutions of accountability and initial introductory meetings at USAID. In addition, the evaluation team Leader is responsible for draft and final report development, as well as other eligible expenses associated with the completion of the assignment. B. Relationships and Responsibilities ME&A, Inc. through NORC will coordinate and manage the evaluation team and will undertake the following specific responsibilities throughout the assignment: 49 • Recruit and hire the evaluation team; and • Make logistical arrangements for the consultants, including travel and transportation, country travel clearance and necessary approvals, lodging, and communications. USAID will provide overall technical leadership and direction for the evaluation team throughout the assignment and will provide assistance with the following tasks: C. Before Field Work • SOW: Respond to queries about the SOW and/or the assignment at large; • Consultant Conflict of Interest (COI): To avoid conflicts of interest or the appearance of a COI, review previous employers listed on the CVs for proposed consultants and provide additional information regarding potential COI with the project contractors evaluated/assessed and information regarding their affiliates; • Documents: Identify and prioritize background materials for the consultants and provide them to ME&A, preferably in electronic form, well in advance of the assignment; • Local Consultants: Assist with identification of potential local consultants, including contact information; and • Site Visit Preparations: Provide a list of site visit locations, key contacts, and suggested length of visit for use in planning in-country travel and accurate estimation of country travel line items costs. D. During Field Work • Mission Point of Contact: Throughout the in-country work, ensure constant availability of the Point of Contact person and provide technical leadership and direction for the team’s work; • Meeting Space: Provide guidance on the team’s selection of a meeting space for interviews and/or focus group discussions (i.e., USAID space if available, or other known office/hotel meeting space); • Participation in the evaluation exercise: Given the nature of the subject evaluation, USAID staff from DRG, Office of Financial Management responsible for government-to-government instruments, and Office of Acquisition and Assistance in-charge of contractual matters regarding G2G will take active role in the assessment; and • Facilitate Contact with Implementing Partners: Inform and where appropriate introduce the evaluation team to implementing partners and other stakeholders, and where applicable and appropriate prepare and send out an introduction letter for team’s arrival and/or anticipated meetings. E. After Field Work Timely Review: Provide timely reviews of draft/final reports and approval of deliverables. MISSION CONTACT PERSON Chemonics Capacity Development for Partners of Accountability Primary Florida Henjewele, Senior DRG Specialist and COR DRG Office, USAID/Tanzania Plot 686 Old Bagamoyo Road, Msasani P.O. Box 9130 Dar es Salaam, Tanzania Email: fhenjewele@usaid.gov 50 Tel: +255 22 2294490 Ext. 4468 Cell: + 255 754 371368 Alternate Adam Lingson, Project Management Specialist DRG Office, USAID/Tanzania Plot 686 Old Bagamoyo Road, Msasani P.O. Box 9130 Dar es Salaam, Tanzania Email: alingson@usaid.gov Tel: +255 22 2294490 Ext. Cell: +255-659 269 188 / 0764 269 188 VIII. COST ESTIMATE ME&A, Inc. will provide a cost estimate for this evaluation. IX. List of Annexes A. List of Resources/Documents 1. USAID Evaluation Policy https://www.usaid.gov/sites/default/files/documents/1870/USAIDEvaluationPolicy.pdf 2. USAID ADS 201 https://www.usaid.gov/sites/default/files/documents/1870/201.pdf 3. USAID Evaluation Report Requirements https://www.usaid.gov/sites/default/files/documents/1868/201mah.pdf 4. How-to Note. Preparing Evaluation Reports https://www.usaid.gov/sites/default/files/documents/1870/How-to-Note_Preparing-Evaluation￾Reports.pdf 5. USAID Learning Lab Evaluation Toolkit https://usaidlearninglab.org/evaluation 6. USAID/Tanzania CDCS 2014-2019 https://www.usaid.gov/sites/default/files/documents/1860/CDCS%20Tanzania%20FINAL.pdf 7. USAID/Tanzania Gender Analysis for Project Planning and Activity Design, June 2013 http://pdf.usaid.gov/pdf_docs/pa00k747.pdf 8. Chemonics’ activity documents (to be provided by USAID) 9. Contract and five modifications, including modifications to the activity description 10. Beneficiaries agreements ( LEAT, TACOSODE, PELUM, WiLDAF, NAOT, PPRA and Ethics Secretariat) 11. Annual Workplans 12. M&E Plans, including PMP 13. Quarterly Reports 14. Annual Reports 15. Success Stories 51 16. AIMS reports 17. Internal Annual evaluation reports, 2015 & 2016 18. Capacity Assessment Results (OCA and Re- OCA) by Pamoja Twajenga B. Other potentially relevant USAID documents available online such as USAID Forward Guidelines. C. List of Grantees and Non-Grantees: Direct-grantee CSOs: • Lawyers’ Environmental Action Team (LEAT) • National Council of People Living with HIV and AIDS in Tanzania (NACOPHA) • Participatory Ecological Land Use Management Tanzania (PELUM) • Tanzania Council for Social Development (TACOSODE) • Women in Law and Development in Africa (WiLDAF) Non-grantee CSOs: • Authorized Association Consortium (AAC) • Human Development Trust (HDT) • Medical Association of Tanzania (MAT) • (Re)claim Women’s Space in World Heritage (RECLAIM) • Rural Urban Development Initiatives (RUDI) • Tanzania Media Women’s Association – Mainland (TAMWA Mainland) • Tanzania Media Women’s Association – Zanzibar (TAMWA Zanzibar) • Women Creative Solutions Association (WCSA) • Zanzibar Legal Service Centre (ZLSC) • E&D Readership and Development Agency (E&D)/Soma IOAs: • Ethics Secretariat (ES) • National Audit Office of Tanzania (NAOT) • Public Procurement Regulatory Authority (PPRA) List of Subgrantees Grantee Sub Offices/Subgrantees Institutions of Accountability (IOAs) Sub-Offices 1 Ethics Secretariat (ES) 1 ES Dar es Salaam HQ 2 ES Southern Zonal Office (Mtwara) 3 ES Central Zonal Office Dodoma 4 ES Morogoro Office 5 ES Iringa Office – Southern Highland Zone Civil Society Organizations (CSOs) Direct Grantees Subgrantees 1 LEAT 1 MUVIMA 2 ASHT 3 MBOMIPA 52 Grantee Sub Offices/Subgrantees 4 MJUMIKK 2 PELUM 1 UMADEP 2 IFTZ 3 WiLDAF 1 Mobile Legal Clinic – Dar es Salaam 2 Tanzania Women and Children Welfare Center Temeke Dar 3 Center for Widow and Children’s Assistance – Mara 4 Rungwe Women and Orphans Centre-Mbeya 5 Tanzania Network for Legal Aid Providers 4 NACOPHA 1 HIV/AIDS Clusters in 46 districts 5 TACOSODE 1 Iringa Mercy Organization (IMO) 2. Mahenge Mazombe Development Association (MMDEA) 3 Community Health and Education Organization (CEDO) 4 AFNET 5 Afya ya Jamii Tanzania-AJAT-Foundation 6 UMWEMA Trust Fund-Morogoro 53 ANNEX II: INCEPTION REPORT USAID/Tanzania Data for Development INCEPTION REPORT POST-PERFORMANCE EVALUATION OF THE CAPACITY DEVELOPMENT FOR PARTNERS OF ACCOUNTABILITY (PAMOJA TWAJENGA) ACTIVITY Award No. AID-621-A- 16-00001 November 2018 This document was produced for review by the United States Agency for International Development Tanzania Mission (USAID/Tanzania). 54 INCEPTION REPORT POST-PERFORMANCE EVALUATION OF THE CAPACITY DEVELOPMENT FOR PARTNERS OF ACCOUNTABILITY (PAMOJA TWAJENGA) ACTIVITY USAID/Tanzania Data for Development Project Submission Date: November 2018 Contract Number: AID-OAA-1-15-00024/AID-621-TO-17-00005 Activity Start Date and End Date: July 05, 2016- July 04, 2021 COR Name: Angela Mwaikambo Prepared by: Data for Development Evaluation Advisor, Jacob Laden Survey Specialist, Gerald Usika Team Lead, Lawrence Robertson, PhD Submitted by: Brenda Pearson, Chief of Party, Data for Development ME&A, Inc. 4350 East-West Highway. Suite 210 Bethesda, MD 20814 Tel: 301-652-4334 Email: bpearson@engl.com DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. 55 TABLE OF CONTENTS 1. PROJECT OVERVIEW AND OBJECTIVES 5 1.1 Introduction 5 1.2 CDPA Approach 6 2. UNDERSTANDING OF THE SCOPE OF WORK 7 2.1 Purpose of the Evaluation 7 2.2 Evaluation Approach 7 3. EVALUATION TIMELINE AND DELIVERABLES 9 4. EVALUATION TEAM 9 56 LIST OF ACRONYMS Acronym Description AAC Authorized Association Consortium AIMS Innovate, Measure, and Share AOR Agreement Officer Representative CDCS Country Development Cooperation Strategy CDPA Capacity Development for Partners of Accountability CEGO Citizens Engaging in Government Oversight COR Contracting Officer Representative CSO Civil Society Organizations DC District Council DEC Development Experience Clearinghouse DO Development Objective DRG Democracy, Human Rights, and Governance ES Ethics Secretariat GI Group Interview GICA Government Institutions Capacity Assessment GoT Government of Tanzania HDT Human Development Trust IoA Institute of Accountability IQS Indefinite Quantity Subcontract IR Intermediate Result KII Key Informant Interview LEAT Lawyers’ Environmental Action Team LOE Level of Effort M&E Monitoring and Evaluation MAT Medical Association of Tanzania N/A Not Applicable NACOPHA National Council of People Living with HIV and AIDS in Tanzania NAOT National Audit Office of Tanzania OCA Organizational Capacity Assessment OCA Organization Capacity Assessment PELUM Participatory Ecological Land Use Management Tanzania 57 Acronym Description PPR Performance Plan and Report PPRA Public Procurement Regulatory Authority RECLAIM (Re)claim Women’s Space in World Heritage RUDI Urban Development Initiatives SAM Social Accountability Monitoring SATC Supporting Accountability to Tanzania Citizens SOW Scope of Work STTA Short Term Technical Assistant TACOSODE Tanzania Council for Social Development TAMWA Tanzania Media Women’s Association USAID United States Agency for International Development WiLDAF Women in Law and Development in Africa ZLSC Zanzibar Legal Service Center 58 1.0 PROJECT OVERVIEW AND OBJECTIVES 1.1 INTRODUCTION United States Agency for International Development (USAID) has asked Data for Development to design and plan a post-performance evaluation of the Capacity Development for Partners of Accountability (CPDA) activity – known as Pamoja Twajenga [“Together we build”] in Kiswahili. The project was implemented by Chemonics International from July 2013 through early July 2017. This inception report, the first deliverable towards the evaluation, outlines a draft work plan and demonstrates Data for Development’s understanding of the assignment. The Inception Report also provides an anticipated schedule for design and implementation of the evaluation and briefly describes the members of the evaluation team, delineated by roles and responsibilities. The purpose of the activity was to build the capacity of targeted civil society organizations (CSOs) and Government of Tanzania (GoT) Institutions of Accountability (IoAs) to engage citizen in government oversight. The CDPA formed part of USAID/Tanzania’s overall Supporting Accountability to Tanzania Citizens (SATC) Project that aimed to: • Strengthen targeted CSO’s capacity to act as advocates and watchdogs; and • Strengthen GoT IoA’s capacity to inform the public and to respond to civil society accountability concerns. The activity was developed to fill gaps in relationships that limited connectivity between citizens and their local governments, CSOs and citizens, between CSOs, and; between civil society and government, particularly with on IoAs and local level government. The SATC Project complimented other elements of USAID’s Democracy and Governance portfolio under DO3: “Improved Accountability and Oversight of Public Resources through Increased Citizen Participation.” USAID issued five direct awards to CSOs to support their endeavors in improving accountable governance in agriculture and food security, health, and natural resources management and to improve their linkages to smaller local CSOs and their citizen constituents, monitor government accountability, and improve flows of information with IoAs. Awards issued at beginning of 2013 supported their advocacy and work with sub-grantees at the grassroots level to: • Share information with citizens and gather evidence about both government performance and citizen priorities; and • Develop effective advocacy messages for appropriate decision-making levels and strengthen their own general understanding of IoAs. These five USAID direct grantee CSOs were: • Lawyers’ Environmental Action Team (LEAT) • National Council of People Living with HIV and AIDS in Tanzania (NACOPHA) • Participatory Ecological Land Use Management Tanzania (PELUM) • Tanzania Council for Social Development (TACOSODE) • Women in Law and Development in Africa (WiLDAF) USAID also provided support to three GoT institutions of accountability to help them strengthen their working relationships with civil society and to improve the accessibility and utility of information they intended to make public. Increased access to public information is important because it allows citizens to play their role in ensuring representation and accountability in policy-making and service delivery. 59 These three USAID-supported IoAs were: • Ethics Secretariat (ES) • National Audit Office of Tanzania (NAOT) • Public Procurement Regulatory Authority (PPRA) Pamoja Twajenga provided this set of Tanzania CSOs and GoT IoAs, plus other CSOs and the sub-grantees of the direct grantee CSOs, selected by USAID with an array technical assistance, training, mentoring, and coaching to support targeted organizations achieve sustainable and significant impact as advocates in their respective sectors, be accountable to citizens as their constituents, to manage resources, and to perform meaningful watchdog functions. USAID/Tanzania selected ten other CSO partner for support. The 10 non￾grantee CSOs PT supported were: • Authorized Association Consortium (AAC)/ Community Wildlife Management Areas Consortium (CWMAC) • Human Development Trust (HDT) • Medical Association of Tanzania (MAT) • (Re)claim Women’s Space in World Heritage (RECLAIM) • Rural Urban Development Initiatives (RUDI) • Tanzania Media Women’s Association – Mainland (TAMWA Mainland) • Tanzania Media Women’s Association – Zanzibar (TAMWA Zanzibar) • Women Creative Solutions Association (WCSA) • Zanzibar Legal Service Centre (ZLSC) • E&D Readership and Development Agency (E&D)/SOMA CPDA/Pamoja Twajenga had four objectives: 1. Strengthen the capacity of USAID/Tanzania partner CSOs to serve as strong advocacy organizations, well-networked at the grassroots level and able to inform and equip citizens to engage with government; 2. Provide technical support to USAID’s CSO partners to support USAID’s in its management of civil society grant activities; 3. Strengthen the capacity of GoT IoAs to package information; employ a cohort of civil servants with a common understanding of ethics, procurement, auditing, and good governance rules; and generally, perform activities under their assistance agreement with USAID/Tanzania; and 4. Facilitate constructive engagement between CSOs and IoAs. 1.2 CDPA APPROACH The CDPA activity supported three IoAs that were receiving direct funding from USAID as well as five CSOs receiving direct funding, plus 10 CSOs that were not receiving funds from USAID. In addition, Chemonics worked with sub-grantees of direct grantee CSOs they were supporting. Chemonics’ provided specialized technical assistance in the form of training, coaching, mentorship, and on-the-job training. Specific areas of interventions were to be determined by the particular needs of the CSO or IoA, through Organizational Capacity Assessments (OCA), capacity building in general focused at the individual/team and at the organizational level. These include building the knowledge and skills of individuals and teams, as well as strengthening structures, tools, policies, procedures, and practices to enhance the capacity of these CSOs and IoAs to be more efficient, cost-effective, accountable, transparent, and technically proficient organizations that are accountable to their respective constituents. 60 2.0 UNDERSTANDING OF THE SCOPE OF WORK 2.1 PURPOSE OF THE EVALUATION USAID/Tanzania commissioned the evaluation to assess the adequacy and comparative advantage of the approach of Pamoja Twajenga. The evaluation will help to document the rationale for the approach, what worked well and what did not, what has been sustained, and where to improve if the Agency would like to design a similar activity in the future. The evaluation may inform future programming at the Mission and suggest approaches/strategies for USAID. The evaluation team will gather valid and reliable data, analyze these data, and answer the five evaluation questions posed by USAID in the SOW: 1. To what extent did the intended outcome of improved and constructive citizen/government engagement occur over the course of the program? 2. Did the technical support provided by CDPA to USAID’s CSO partners meet the needs and priorities of the Mission related to its management of civil society grant activities? 3. Did the activity meet its targets in strengthening the capacity of CSOs and GoT IoAs? 4. To what extent have the actions supported or encouraged through the CDPA activity been sustained by beneficiary IoAs and CSOs? 5. Are the stakeholders satisfied with the performance of the contractor in training and providing technical assistance to the direct CSOs and IoAs? 2.2 EVALUATION APPROACH The team proposes to conduct the evaluation using a participatory approach, engaging USAID/Tanzania staff, current and former Chemonics International staff, direct grantee CSO managers and staff, other CSO beneficiary management and staff, direct grantee sub-grantees in regions, and managers from three GoT IoAs: the Ethics Secretariat (ES), National Audit Office of Tanzania (NAOT), and Public Procurement Regulatory Authority (PPRA). The independent evaluation team will design a field work plan to reach needed beneficiaries—and instruments to collect valid and reliable data from project beneficiaries in these organizations and from USAID. The evaluation will employ predominantly a rapid qualitative approach and complement it with quantitative from project monitoring data. The independent Data for Development evaluation team will work collaboratively to: 1. Gather and review project documentation for a desk review to ensure that the team understand Pamoja Twajenga in order to be able to evaluate it fairly; 2. Plan fieldwork to collect valid and reliable data from project partners; 3. Develop instruments to gather data through individual and small group interviews; 4. Develop a sampling framework for data collection from; 5. Analyze the data; 6. Draft and present on the evaluation report; 7. Participate in workshop to review findings, conclusions, and recommendations to ensure feasibility and utilization; and 61 8. Finalize the post-performance evaluation after the receipt of all comments. The evaluation team will employ a mixed-methods approach, which will analyze project related documentation, project monitoring data, data collected through key KIIs, and GIs with various stakeholders involved in the program or who have benefited from program interventions. The use of multiple sources of data will allow the evaluation team to triangulate information, ensuring that findings and conclusions are robust and that recommendations are sound. The evaluation team’s goal is to generate not only an overall understanding about the program, its results, and effects but also a detailed assessment of its various components and the effectiveness of their approaches and implementing mechanisms. All primary and secondary data will be disaggregated by appropriate demographics, including group, age and gender/sex whenever possible. Review and Analysis of Performance Monitoring Data The evaluation team will also review and analyze performance monitoring data produced by the Pamoja Twajenga’s Activity will be obtained from the secondary data provided by Mission and Chemonics. These will be analyzed using descriptive statistics in MS excel. Key Informant Interviews The evaluation team anticipates conducting more than 30 semi-structured interviews with key informants from USAID, Chemonics, CSOs, sub-grantees, and the IoAs. Document review and interviews with USAID and Chemonics staff will be used to identify who to interview. In addition to doing key informant interviews (KIIs) with relevant USAID staff, the evaluation team will hold an in brief consultation meeting with the Mission DRG team. Consultation will be used to make sure there is mutual understanding and approval of the detailed evaluation design (with work plan and evaluation instruments) that is the second deliverable. 3.0 EVALUATION TIMELINE AND DELIVERABLES The anticipated timeline for the evaluation is designed to meet all deliverables and provide USAID with preliminary findings at the end of fieldwork in December 2018, and a first draft of the evaluation by mid￾January 2019. Timing Main Activities in Post-Performance Evaluation November 2018 Drafting and discussion of Inception Report (Deliverable 1) November 2018 Drafting and discussion of evaluation design, work plan and instruments November 2018 USAID review of the draft evaluation design November 2018 Revision of evaluation design November 2018 USAID approval evaluation design with work plan and instruments (Deliverable 2) December 1-2, 2018 STTA travel and in-brief preparation December 3, 2018 In-Brief for USAID (Deliverable 3) December 3-14, 2018 Data collection December 17-19, 2018 Initial analysis; participatory generation of initial findings, conclusions, and recommendations 62 Timing Main Activities in Post-Performance Evaluation December 10, 2018 Out-brief for USAID (Deliverable 4) December 15, 2018 -January 11, 2019, Data analysis January 11-21, 2019 Report writing January 21, 2019 Submit Draft Evaluation Report (Deliverable 5) January 21-February 1, 2019 USAID review of draft Evaluation Report February 1-15, 2019 Incorporate USAID comments and submit final Evaluation Report March 2019 Submit Final Evaluation report (Deliverable 6) TBD Utilization and action planning, after action review session. Dissemination to PO and Education team for their dissemination to IPs and public dissemination via the Development Experience Clearinghouse (DEC). TBD Upload to DEC 4.0 EVALUATION TEAM Data for Development has selected an experienced expatriate Short Term Technical Assistant (STTA) Lawrence R. Robertson, PH.D. who will serve as Team Leader. The evaluation team is composed of two local STTAs with extensive experience in CSO capacity building and Democracy and Governance. Edward Mhina (Capacity Development Expert) and Bernard Kindoli (Citizen Engagement/Participation Expert) who will serve in conducting the data collection and analysis and will make key contributions toward the evaluation report. Data for Development staff M&E Specialist Shakila Mayumana (Gender and Youth Specialist), Daud Siwalaze (M&E Specialist) and Gerald Usika (M&E, Survey and Data Collection Specialist) will work as part of the team for data collection and will contribute to the analysis. Jacob Laden (Evaluation Advisor) will advise the evaluation design and analysis, provide coordination and management support for fieldwork, and review all final and intermediate deliverables. Data for Development staff will support all logistics and data collection efforts, facilitate client and IP communication, conduct sampling, and conduct the data analysis to be used in the evaluation report. 63 ANNEX III: EVALUATION DESIGN WITH DATA COLLECTION INSTRUMENTS USAID/Tanzania Data for Development EVALUATION PLAN AND WORK PLAN POST-PERFORMANCE EVALUATION OF THE CAPACITY DEVELOPMENT FOR PARTNERS OF ACCOUNTABILITY (PAMOJA TWAJENGA) ACTIVITY Award No. AID-621-A- 16-00001 November 2018 This document was produced for review by the United States Agency for International Development Tanzania Mi i (USAID/T i) 64 EVALUATION PLAN AND WORK PLAN POST-PERFORMANCE EVALUATION OF THE CAPACITY DEVELOPMENT FOR PARTNERS OF ACCOUNTABILITY (PAMOJA TWAJENGA) ACTIVITY USAID/Tanzania Data for Development Project Submission Date: November 2018 Contract Number: AID-OAA-1-15-00024/AID-621-TO-17-00005 Activity Start Date and End Date: July 05, 2016- July 04, 2021 COR Name: Angela Mwaikambo Prepared by: Data for Development Evaluation Advisor, Jacob Laden Survey Specialist, Gerald Usika Team Lead, Lawrence Robertson, PhD Submitted by: Brenda Pearson, CoP, Data for Development ME&A, Inc. 4350 East-West Highway. Suite 210 Bethesda, MD 20814 Tel: 301-652-4334 Email: bpearson@engl.com DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. 65 TABLE OF CONTENTS 1. PROJECT OVERVIEW AND OBJECTIVES 4 1.1 Introduction 4 1.2 Context 4 1.3 Intended Results of CDPA Activity 6 1.4 CDPA Approach and Implementation 7 1.5 Existing Documents 9 1.6 Target Areas and Groups 9 2. EVALUATION DESIGN AND METHODOLOGY 10 2.1 Purpose of the Evaluation 10 2.2 Evaluation Approach 12 2.3 Evaluation Design 12 2.4 Required Mission and Chemonics Consultation 18 3. CAPACITY BUILDING PLAN 19 4. EVALUATION LIMITATIONS 19 5. EVALUATION TIMELINE AND DELIVERABLES 20 5.1 Evaluation Timeline 20 5.2 Evaluation Deliverables 21 6. DISSEMINATION OF FINDINGS 22 7. PROPOSED STAFFING 22 LIST OF TABLES Table 1: List of Sub-Grantees (CSOs and IoAs) and their Reach 10 Table 2: Civil Society Organizations (CSOs) Direct Grantees Coverage and Sampling 13 Table 3: Data Collection Activities by Location 16 Table 4: Evaluation Design Matrix 17 Table 5: Evaluation Timeline and Deliverables 21 Table 6: Estimated LOE per Team Member (Total LOE: 227 Days) 24 LIST OF FIGURES Figure 1: Pamoja Twajenga Result Framework 6 Figure 2: Envisioned Structure of USAID’s Overall Accountable Government Activity (Supporting Accountability to Tanzania Citizens) Through the Contractor 8 66 LIST OF ACRONYMS Acronym Description AAC Authorized Association Consortium AIMS Innovate, Measure, and Share AOR Agreement Officer Representative CDCS Country Development Cooperation Strategy CDPA Capacity Development for Partners of Accountability CEGO Citizens Engaging in Government Oversight COR Contracting Officer Representative CSO Civil Society Organizations DC District Council DEC Development Experience Clearinghouse DO Development Objective DRG Democracy, Human Rights, and Governance ES Ethics Secretariat GI Group Interview GiCA Government Institutions Capacity Assessment GoT Government of Tanzania HDT Human Development Trust IoA Institute of Accountability IQS Indefinite Quantity Subcontract IR Intermediate Result KII Key Informant Interview LEAT Lawyers’ Environmental Action Team LOE Level of Effort M&E Monitoring and Evaluation MAT Medical Association of Tanzania N/A Not Applicable NACOPHA National Council of People Living with HIV and AIDS in Tanzania NAOT National Audit Office of Tanzania OCA Organizational Capacity Assessment PELUM Participatory Ecological Land Use Management Tanzania PPR Performance Plan and Report 67 Acronym Description PPRA Public Procurement Regulatory Authority RECLAIM (Re)claim Women’s Space in World Heritage RUDI Urban Development Initiatives SAM Social Accountability Monitoring SATC Supporting Accountability to Tanzania Citizens SOW Scope of Work STTA Short Term Technical Assistant TACOSODE Tanzania Council for Social Development TAMWA Tanzania Media Women’s Association USAID United States Agency for International Development WiLDAF Women in Law and Development in Africa ZLSC Zanzibar Legal Service Center 68 1.0 PROJECT OVERVIEW AND OBJECTIVES 1.1 INTRODUCTION As part of the Contract/Task Order Number AID-OAA-1-15-00024/AID-621-TO-17-00005, Data for Development Activity, the United States Agency for International Development (USAID) has asked Data for Development to design and budget for a post-performance evaluation of the Capacity Development for Partners of Accountability (Pamoja Twajenga) activity implemented by Chemonics International during the period 2013-2017. This cooperative agreement under Award Number 2013-2017: 621-C-13-00002 primarily corresponds to the USAID Development Objective (DO) 3: Effective Democratic Governance Improved; secondarily it is crosscutting DO1: Tanzania Women and Youth Empowered and Development Objective 2: Inclusive broad-based economic growth sustained. Pamoja Twajenga’s planned outcomes include, strengthened USAID/Tanzania partner Civil Society Organizations (CSOs) capacity to serve as strong advocacy organizations, well-networked at the grassroots level and able to inform and equip citizens to engage with government; USAID is supported in its management of civil society grant activities; the capacity of Government of Tanzania (GoT) Institute of Accountability (IoAs) is strengthened to package information that is to employ a cohort of civil servants with a common understanding of ethics, procurement, auditing, and good governance rules; and finally to ensure the constructive engagement between CSOs and IoAs is strengthened. Its intended results are, organizational development of selected CSOs and non-Democracy, Human Rights, and Governance (DRG) organizations supported, development/strengthening of issue-based dialogue of selected CSOs and non-DRG organizations supported, grants management aspects of organizational development in USAID grantees supported, USAID grantees demonstrated impact and success supported, organizational development of targeted IoAs supported, and the issue-based dialogue between IoAs, CSOs, and citizens supported. 1.2 CONTEXT Problem or Opportunity Addressed by the Activity Being Evaluated The Capacity Development for Partners of Accountability (CDPA) or Pamoja Twajenga activity implemented by Chemonics International started in July 2013 and sought to develop and enhance the capacity of partner organizations (both state and non-state) in support of the Mission’s DO3, which focuses on improving democratic governance, under the Country Development Cooperative Strategy (CDCS) approved in 2014. The purpose of the activity was to build the capacity of targeted CSOs and GoT IoAs to engage citizen in government oversight. The CDPA formed part of USAID/Tanzania’s overall Supporting Accountability to Tanzania Citizens (SATC) Project that aimed to: • Strengthen targeted CSOs’ capacity in advocacy and citizen engagement for transparency and good governance; and • Strengthen GoT IoAs’ (i.e., Ethics Secretariat) capacity to inform the public and to respond to civil society accountability concerns. The major relational gaps targeted were connectivity between citizens and their local governments; CSOs and citizens; civil society amongst itself, and; between civil society and government-with a particular focus on IoAs and local level government. The SATC Project complimented other elements of USAID’s Democracy and Governance portfolio under DO3: “Improved Accountability and Oversight of Public Resources through Increased Citizen Participation.” 69 Theory of Change The activity’s approach focused on both the demand and supply side of accountability; supporting CSOs on the demand side and IoAs on the supply side. The theory of change that underpinned the activity purports that international development partner pressure on the GoT alone is unlikely to result in major changes. But that if citizens are more aware, motivated, and empowered to hold their government more accountable at the community level on a consistent long-term basis, this may lead to increased civic engagement and demand-driven improvements in governance at the district, regional and national levels, producing increased pressure for more open political competition, and a healthier, more transparent exchange between citizens and their government. Links to USAID/Tanzania’s Country Development Cooperation Strategy (CDCS) Although the design of the Capacity Development for Partners of Accountability (CDPA) activity and USAID’s support to Chemonics precedes the development and approval of the current CDCS, the activity is a critical component of the Mission’s approach to achieving DO3: Effective Democratic Governance Improved, Intermediate Result (IR) 3.1 Citizen engagement made more effective; sub-intermediate results (sub-IR) 3.1.1 Citizen access to information increased; sub-IR 3.1.2 Institutional capacity of CSOs increased; and, sub-IR 3.1.3 CSOs successful oversight at national and sub-national levels advocated and conducted. Citizen engagement is made more effective when citizens are able to access information produced by IoAs (IR 3.3). By promoting individual civic rights, including women and youth rights, IR 3.1 Citizen engagement made more effective, also supports results achievement under IR 1.1 Gender equality and youth inclusion; and IR 1.2 Health status improved. In supporting citizen oversight of local government, citizen engagement helps to improve service delivery, in line with IR 3.2 Government delivery of services improved, and IR 3.3 Government accountability increased. In addition, CSOs funded under IR 3.1 worked in priority USAID sectors of education, agriculture, and natural resource management in view of advancing achievement under IR 1.3 Lifelong learning skills improved, IR 2.2 Agricultural productivity and profitability increased in targeted value chains, and IR 2.3 stewardship of natural resources improved. Capacity Development for Partners of Accountability Scope The CDPA activity was designed to assist citizens in understanding the way governance affects them, what role they can play in improving governance, and be equipped with easily accessible and user-friendly information on their government’s performance. To achieve these aims, USAID issued five direct awards to CSOs to support their endeavors in improving accountable governance in the sectors of agriculture and food security, health, and natural resources management and to improve their linkages to smaller local CSOs and their citizen constituents, monitor government accountability, and improve flows of information with IoAs. Specifically, these awards to CSOs were issued at beginning of 2013 to support them in their advocacy efforts, and their work with sub-grantees at the grassroots level to: a. Share information with citizens and gather evidence about both government performance and citizen priorities; and b. Develop effective advocacy messages for appropriate decision-making levels and strengthen their own general understanding of IoAs. Concurrently, USAID provided support to three GoT institutions of accountability to help them strengthen their working relationships with civil society and to improve the accessibility and utility of information they intended to make public. Increased access to public information is important because it allows citizens to play their role in ensuring that they are being represented in the policy-making and service delivery of their government. Even more importantly, it permits citizens to learn about government plans or actions that might be critical to their own personal interests. The purpose of the CDPA Activity (also known as Pamoja Twajenga) was to provide selected Tanzania CSOs and GoT IoAs with an array of support (such as technical assistance, training, logistics, etc.) in order 70 for these targeted organizations to: achieve sustainable and significant impact as advocates in their respective sectors; to be accountable to citizens as their constituents; to manage resources (donor or other); and to perform meaningful watchdog functions regarding use of public resources. CDPA’s primary objectives included: 1. Strengthen the capacity of USAID/Tanzania partner CSOs to serve as strong advocacy organizations, well-networked at the grassroots level and able to inform and equip citizens to engage with government; 2. Provide technical support to USAID in its management of civil society grant activities; 3. Strengthen the capacity of GoT IoAs to package information; employ a cohort of civil servants with a common understanding of ethics, procurement, auditing, and good governance rules; and, generally, perform activities under their assistance agreement with USAID/Tanzania; and 4. Facilitate constructive engagement between CSOs and IoAs. 1.3 INTENDED RESULTS OF CDPA ACTIVITY • Result 1.1: Organizational development of selected CSOs and non-DRG organizations supported; • Result 1.2: Development/strengthening of issue-based dialogue of selected CSOs and non-DRG organizations supported; • Result 2.1: Grants management aspects of organizational development in USAID grantees supported; • Result 2.2: USAID grantees demonstrated impact and success supported; • Result 3.1: Organizational Development of targeted IoAs supported; and • Result 3.2: Issue-based dialogue between IoAs, CSOs, and citizens supported. Figure 1: Pamoja Twajenga Results Framework ILLUSTRATION OF PAMOJA TWAJENGA RESULTS FRAMEWORK 71 1.4 CDPA APPROACH AND IMPLEMENTATION Under the CDPA activity, Chemonics International supported three IoAs that were receiving direct funding from USAID as well as five CSOs receiving direct funding, and 10 CSOs that were not receiving funds from USAID. In addition, Chemonics worked with sub-grantees of the CSOs they were supporting. Chemonics’ support targeted specialized technical assistance in the form of training, coaching, mentorship, and on-the-job training. Chemonics International used participatory approaches that positioned local partners as an integral part of activities. This included mentoring them and pairing them to the extent possible with organizations and companies with higher capacity levels to ensure sharing of knowledge and best practices, ownership of interventions, and an overall leadership role in activities. Whilst specific areas of interventions were to be determined by the particular needs of the CSO or IoA, through Organizational Capacity Assessments (OCA), capacity building in general focused at the individual/team and at the organizational level. These include building the knowledge and skills of individuals and teams, as well as strengthening structures, tools, policies, procedures, and practices. The contractor was expected to enhance the capacity of these CSOs and IoAs in promoting more efficient, cost-effective, accountable, transparent, and technically proficient organizations that are accountable to their respective constituents. Additionally, the contractor was expected to work on improving linkage between the direct grantees and their sub-grantees (see Figure 1 below), as well as provide management support to USAID/Tanzania’s DRG team with grants management (which varied in terms of size, length, and programmatic areas of focus), and build the capacity of recipient CSOs to issue and be accountable for grants issued to sub-grantees. In general, CDPA employed a mix of traditional and innovative capacity building approaches. The activity engaged in strategic partnerships with established Tanzanian organizations through Indefinite Quantity Subcontracts (IQSs) to leverage expertise in communications, strategic planning, and financial and audit best practices. CDPA also engaged experts to provide advocacy and organizational lessons learned, advocacy expertise, and best practices through quarterly awareness-raising sessions and training (list of grantees and sub-grantees is annexed). In carrying out its capacity building efforts, CDPA employed a wide range of participatory methods, including document management system and a Pamoja Twajenga quality certification, one-on-one mentoring and coaching, peer-to-peer learning, roundtable discussion, an “achieve, innovate, measure, and share assessment,” and use of pool of skilled trainees and service providers. 1.5 EXISTING DOCUMENTS Several documents are available for review for this post performance evaluation of CDPA. These include but are not limited to (i) CDPA Scope of Work (SOW) (ii) CDPA annual work plans for the duration of the activity, (iii) CDPA M&E Plan, (iv) CDPA quarterly progress reports, (v) CDPA annual reports (vi) CDPA Final Report, (vii) CDCS 2014/19, (viii) SATC PAD, (ix) SOL, (x) CDPA Achieve, Innovate, Measure, and Share (AIMS) reports, (xi) Organizational Capacity Assessment Reports, (xii) Government Institutions Capacity Assessment (GiCA), (xiii) Capacity Development Plans, (xiv) Annual Report of Citizens Engaging in Government Oversight (CEGO) grantees, (xv) Annual Work Plans of CEGO Grantees, (xvi) Annual Work Plans of IoAs, (xvii) Annual Reports of IoAs, (xviii) IoAs Grant Agreements, (xix) Organization Capacity Assessments (OCA), (xx) Re-OCA reports, and (xxi) CEGOs Grant Agreements. 1.6 TARGET AREAS AND GROUPS Pamoja Twajenga’s active Geographic Regions are Dodoma, Iringa, Morogoro, Mtwara, and Zanzibar. USAID issued five direct awards to CSOs to support their endeavors in improving accountable governance in the sectors of agriculture and food security, health, and natural resources management and to improve their linkages to smaller local CSOs and their citizen constituents, monitor government accountability, 72 and improve flows of information with IoAs. Specifically, these awards to CSOs were issued at beginning of 2013. These direct-grantee CSOs were: Lawyers’ Environmental Action Team (LEAT), National Council of People Living with HIV and AIDS in Tanzania (NACOPHA), Participatory Ecological Land Use Management Tanzania (PELUM), Tanzania Council for Social Development (TACOSODE) and Women in Law and Development in Africa (WiLDAF). A total of 10 non-grantee CSOs were supported and these are: Authorized Association Consortium (AAC), Human Development Trust (HDT) , Medical Association of Tanzania (MAT), (Re)claim Women’s Space in World Heritage (RECLAIM), Rural Urban Development Initiatives (RUDI), Tanzania Media Women’s Association – Mainland (TAMWA Mainland), Tanzania Media Women’s Association – Zanzibar (TAMWA Zanzibar), Women Creative Solutions Association (WCSA), Zanzibar Legal Service Center (ZLSC), and E&D Readership and Development Agency (E&D)/Soma. Concurrently, USAID provided support to three GoT institutions of accountability to help them strengthen their working relationships with civil society and to improve the accessibility and utility of information they intended to make public. The IoA are the Ethics Secretariat (ES), National Audit Office of Tanzania (NAOT), and Public Procurement Regulatory Authority (PPRA). Table 1 below summarizes the List of Sub-grantees (CSOs and IoAs) and their reach. Table 1: List of Sub-Grantees (CSOs and IoAs) and Their Reach Grantee Sub Offices/Sub-Grantees Institutions of Accountability (IOAs) Sub Offices 1 ES 1 ES Dar es Salaam HQ 2 ES Southern Zonal Office (Mtwara) 3 ES Central Zonal Office Dodoma 4 ES Morogoro Office 5 ES Iringa Office – Southern Highland Zone Civil Society Organizations (CSOs) Direct Grantees Sub-grantees 1 LEAT 1 MUVIMA 2 ASHT 3 MBOMIPA 4 MJUMIKK 2 PELUM 1 UMADEP 2 IFTZ 3 WiLDAF 1 Mobile Legal Clinic – Dar es Salaam 2 Tanzania Women and Children Welfare Centre Temeke Dar 3 Center for Widow and Children’s Assistance – Mara 4 Rungwe Women and Orphans Center-Mbeya 5 Tanzania Network for Legal Aid Providers 4 NACOPHA 1 HIV/AIDS Clusters in 46 districts 5 TACOSODE 1 Iringa Mercy Organization (IMO) 2. Mahenge Mazombe Development Association (MMDEA) 3 Community Health and Education Organization (CEDO) 4 AFNET 5 Afya ya Jamii Tanzania-AJAT-Foundation 73 Grantee Sub Offices/Sub-Grantees 6 UMWEMA Trust Fund-Morogoro 2.0 EVALUATION DESIGN AND METHODOLOGY 2.1 PURPOSE OF THE EVALUATION USAID/Tanzania determined the importance of evaluating this approach and documenting lessons learned and best practices in order to assess the adequacy and comparative advantage of the approach in contributing to achievement of the DO. The evaluation will help to document the rationale for the approach, what worked well and what did not, what has been sustained, and where to improve if the Agency would like to design a similar activity. It is expected that findings and recommendations will inform future programming at the Mission and suggest approaches/strategies that could inform the Agency at large. Audience and Intended Uses The results of this evaluation activity will be used by USAID/Tanzania across sectors to inform decisions and design of the forthcoming CDCS and future programming. The intended users of the evaluation include USAID/Tanzania Development Objective (DO3) Team, the DRG office in particular, and the Agency at large. Evaluation Questions The intended scope of the CDPA activity with USAID covered four objectives: a) to strengthen the capacity of USAID/Tanzania partner CSOs to serve as advocacy organizations, well-networked at the grassroots level and able to inform and equip citizens to engage with government; b) provide technical support to USAID in its management of civil society grant activities; c) strengthen the capacity of government of Tanzania IoAs to package information, employ a cohort of civil servants with a common understanding of ethics, procurement, auditing, and good governance rules; and d) perform activities under their assistance agreement with USAID/Tanzania; and facilitate constructive engagement between CSOs and IoAs. 1. To what extent did the intended outcome of improved and constructive citizen/government engagement occur over the course of the program? USAID is interested in understanding the relationship between capacity development inputs and how this might be related to relevant CSOs and GoT performance outcomes. In addition, USAID is interested in any insight into the sustainability of activity investments in terms of CSOs and GoT performance. Capacity include but not limited to IoAs ability to package information, employ a cohort of civil servants with common understanding on ethics, procurement, audit, and good governance rules, work planning, monitoring and evaluation, and performance of activities under the assistance agreement with USAID/Tanzania. CSOs’ capacity relates to governance, administration, human resources management, financial management, organizational management, program management, and project performance. (Activity components on the organizational development and issue-based dialogue for results 1 and 3 will be the focus here.) 2. Did the technical support provided by CDPA to USAID’s CSO partners meet the needs and priorities of the Mission related to its management of civil society grant 74 activities? USAID is interested in understanding what worked well and what didn’t.13 The evaluation team should focus on contractor’s support in the management of civil society grants activity. This question is only focused on USAID key informants. (By agreement with USAID in the in brief, the evaluation will also explore the utility of CPDA support to one of USAID’s IoA partners to meet the needs and requirements related to mission management of their G2G award.) 3. Did the activity meet its targets in strengthening the capacity of CSOs and GoT IoAs? This will be primarily answered through performance monitoring data and background documents. Activity components on the organizational development and issue-based dialogue for results 1 and 3 will be the focus here. 4. To what extent have the actions supported or encouraged through the CDPA activity been sustained by beneficiary IoAs and CSOs? What political and economic factors challenged CSOs activities during the course of the PT program? USAID is interested in understanding how peer-to-peer learning, on-the-job trainings, communities of practices between CSOs, mentoring, coaching in work planning, gender and social inclusion, Social Accountability Monitoring (SAM) are working? 5. Are the stakeholders satisfied with the performance of the contractor in training and providing technical assistance to the direct CSOs and IoAs? Were they able to cascade the training and what level of satisfaction do sub-grantees have with the direct grantees’ training? (This question will also include reflection on the usefulness of training from IoAs and CSOs.) 2.2 EVALUATION APPROACH The team proposes to conduct the evaluation using a participatory approach, engaging USAID/Tanzania, Chemonics International, CSOs, GoT IoAs [Ethics Secretariat (ES), National Audit Office of Tanzania (NAOT), and Public Procurement Regulatory Authority (PPRA)], project beneficiaries, and other stakeholders through various phases of the evaluation. The evaluation will employ predominantly a rapid qualitative approach and complement it with quantitative from project monitoring data. In the course of this evaluation, Data for Development will work collaboratively to: 1. Identify appropriate questions, keeping in mind users and uses of the evaluation for Mission, continuum partners, and GoT decision making; 2. Identify pertinent documentation for desk review, including existing data from quarterly and annual reports; 3. Plan fieldwork and review sample framework for data collection, including assisting the team to identify participants for key informant interviews (KIIs), and group interviews (GIs); 4. Review questionnaires for KIIs and GIs; 5. Participate in workshop to review findings, conclusions, and recommendations to ensure feasibility and utilization; and 6. Serve as feedback providers for reports and other deliverables. At the same time, the evaluation team will remain independent and take steps to maximize the quality of the information and minimize the impact of various potential sources of bias on the evaluation. Accordingly, Chemonics, CSOs, and IoA staff will not be involved directly in data collection activities for the purpose of maintaining objectivity and ensuring respondent/beneficiary confidentiality as they provide feedback. In addition, to ensure that the evaluation team preserves its independence when drafting the 75 final evaluation report, it will be important for the team and USAID to agree on dissemination to USAID and non-USAID audiences. 2.3 EVALUATION DESIGN To address the evaluation questions, the evaluation team will employ a mixed-methods approach, which will analyze project related documentation, project monitoring data, data collected through key KIIs, and GIs with various stakeholders involved in the program or who have benefited from program interventions. The use of multiple sources of data will allow the evaluation team to triangulate information, ensuring that findings and conclusions are robust and that recommendations are sound. The evaluation team’s goal is to generate not only an overall understanding about the program, its results, and effects but also a detailed assessment of its various components and the effectiveness of their approaches and implementing mechanisms. All primary and secondary data will be disaggregated by appropriate demographics, including group, age and gender/sex whenever possible. TARGET AREAS AND SAMPLING A purposive rather than random sampling method will be used for selecting Regions and Districts from which the KIIs and GIs participants will be drawn. Pamoja Twajenga activity was active in Dodoma, Iringa, Morogoro, Mtwara, Dar es Salaam and Zanzibar. For the purpose of logistical feasibility, for this evaluation, it will not be possible to visit all regions. To achieve a geographic balance, also considering the coverage of the Civil Society Organizations (CSOs) direct grantees, four regions, Morogoro, Dodoma, Iringa, and Dar es Salaam will be selected for the field work. Zanzibar field work will be limited as three non-grantee CSOs are there. Unstructured informal interviews will be held to better understand their engagement with the PT program (see table 2). The CSO direct grantees and sub-grantees mostly cover the following districts: Bahi and Kongwa in Dodoma region; Mvomero and Morogoro in Morogoro Region; Iringa, Kilolo, and Mufindi in Iringa region; Rungwe in Mbeya region; and Temeke in Dar es Salaam region. WiLDAF, NACOPHA, PELUM, and TACOSODE do overlap between the districts, the selection of two districts in Morogoro [Mvomero and Morogoro District Council (DC)], Iringa (Kilolo and Mufindi) and Dodoma (Bahi and Kongwa) would still ensure maximum coverage of the CSO activities. Hence, the evaluation team will select Mvomero and Morogoro DC in Morogoro, Kilolo and Mufindi in Iringa, and Dodoma Bahi and Kongwa in Dodoma region. Temeke will be selected in the Dar es Salaam region since it is the only district with the most coverage of CSOs. Hence, a total of seven districts will be selected for field work. All the IoAs will be selected in the sampled four regions. There were a minimum of two sub-grantees supported by each of the five direct grantees. Two sub￾grantees were selected from each of the five direct grantees for a total of 10 sub-grantees to be assessed under this evaluation. The 10 sub-grantees have been selected randomly per each of the five grantees per region (from 17 sub-grantees and 46 HIV/AIDS clusters). In consultation with the Mission, districts may undergo changes in sampling depending on the diversity of activities in each of the direct grantees’ component areas. Replacements will be made on a case by case basis (see Table 2). Table 2: Civil Society Organizations (CSOs) Direct Grantees Coverage and Sampling Direct Grantee CSO Region Districts Sampled District Sampled Sub-Grantee LEAT Iringa Iringa and Mufindi Mufindi DC 1. MBOMIPA 2. MUVIMA PELUM Dodoma, Morogoro, Iringa Bahi and Kongwa; Mvomero and Morogoro; Kilolo and Mufindi 1. Morogoro MC 2. Mufindi DC 1. UMADEP 2. IFTZ/TAGRODE 76 Direct Grantee CSO Region Districts Sampled District Sampled Sub-Grantee WiLDAF Dar es Salaam, Mbeya Temeke, Rungwe Temeke 1. Tanzania Women and Children Welfare Center 2. Tanzania Network for Legal Aid Providers NACOPHA In 46 districts including Dodoma, Morogoro, Iringa Bahi and Kongwa; Mvomero and Morogoro; Iringa, Kilolo, and Mufindi 1. Bahi DC 2. Kongwa DC 1. Bahi DC Cluster 2. Kongwa DC Cluster TACOSODE Iringa, Morogoro Iringa; Kilolo and Mufindi; Mvomero and Morogoro 1. Kilolo DC 2. Mvomero DC 1. AFNET 2. Community Health and Education Organization (CEDO) Institutions of Accountability (IOAs)14 ES Dar es Salaam HQ Dar es Salaam N/A N/A N/A ES Southern Zonal Office Mtwara N/A N/A N/A ES Central Zonal Office Dodoma N/A N/A N/A ES Morogoro Office Morogoro N/A N/A N/A The evaluation team will use KIIs at the national level and GIs, KIIs, and observation at the regional and district levels. PARTICIPANTS SELECTION AND DATA COLLECTION METHODS The following section explains the participants selection and data collection method the evaluation team will use for each evaluation approach. Desk Review The evaluation team will review key documents that will have information not only to answer the research questions, but to provide background information that will support the evaluation process. Documents to be selected for review will depend on the availability, need and purpose of the review. Documents to be reviewed include but are not limited to: (i) CDPA SOW, (ii) CDPA annual work plans for the duration of the activity, (iii) CDPA M&E Plan, (iv) CDPA quarterly progress reports, (v) CDPA annual reports, (vi) CDPA Final Report, (vii) CDCS 2014/19, (viii) SATC PAD, (ix) SOL, (x) CDPA Achieve, AIMS reports, (xi) OCA Reports, (xii) GICAs, (xiii) Capacity Development Plans, (xiv) Annual Report of CEGO grantees, (xv) Annual Work Plans of CEGO Grantees, (xvi) Annual Work Plans of IoAs, (xvii) Annual Reports of IoAs, (xviii) IoA Grant Agreements, (xix) OCA, (xx) Re-OCA reports, and (xxi) CEGOs Grant Agreements. The initial set of documents to be reviewed will include all those received from USAID and Pamoja Twajenga. Upon completing an initial review, the evaluation team will request any additional documentation that are deemed important. Throughout the evaluation, the team will keep an updated checklist of all documents received noting any missing or partial documentation. The team will continuously update this checklist based on documents received and request additional documentation as needed. 14 District level stakeholders related to regional IOAs campaigns and other activities have yet to be confirmed, thus the IOAs are only presently represented in the regions. 77 Review and Analysis of Performance Monitoring Data Performance monitoring data produced by the Pamoja Twajenga’s Activity will be obtained from the secondary data provided by Mission and Chemonics. These will be analyzed using descriptive statistics in MS Excel. Quarterly and annual reporting on program indicators and trends over time will be observed from the start to the end of program implementation. Performance data will be drawn from the IPRS system to assess time series of actual versus target performance on indicators reported to the Mission and for the Performance Planning and Reporting (PPR) performance of DOs involved and its respective IRs. The evaluation team will assess performance on predetermined, valid criteria (objectives and expected results) which can be consistently measured over the activity’s life cycle. The comparison will look into facilitating and/or prohibiting factors (internally and externally) and the evaluator will provide information on lessons learned and recommendations on their effectiveness for future similar design. Upon completing an initial review of existing data, the evaluation team will determine if it possible to use the data to carry out advanced statistical analyses. Key Informant Interviews The evaluation team will conduct approximately 28-31 semi-structured interviews with key informants (see table 3). Selection of KII participants The evaluation team will work with the USAID/Tanzania Mission DRG office to identify the relevant individuals from institutional partners of CDPA. Key informants to be interviewed at the national level include senior-level staff from Pamoja Twajenga, activity staff from Chemonics International, national-level grantees, and USAID staff. A snowball sampling method will be used to select 1-2 key informants from the aforementioned stakeholder groups. During the sampling process, Chemonics and/or USAID/Tanzania Mission will be consulted by the evaluation team to ensure selection of key informants that are most relevant to interview. Based on the number of stakeholders, a total of 15 national-level KIIs will be interviewed in the selected four regions. These will include USAID staff as the funder of the activity, Chemonics staff as the implementing partner, staff from the five direct grantee CSOs staff from the other CSOs that PT worked with, and beneficiaries from the three IoAs (Ethics Secretariat-ES, National Audit Office of Tanzania￾NAOT, and Public Procurement Regulatory Authority-PPRA). This number may vary depending on the recommendation from the Mission and Chemonics. The evaluation team will also hold KIIs in Zanzibar with three CSO staff. The evaluation team will develop a modular KII guide that evaluation team members can easily tailor to each informant type. The guide will include 15-20 open-ended questions. All KIIs will be recorded with informed consent obtained prior to the start of the interview. Interview transcripts will be coded thematically to produce a summary of recurrent themes that emerge in response to each topic. When appropriate, important findings will be illustrated with quotes. Group Interviews Selection of 16 Group Interview (GI) participants At the district level, the evaluation team will attempt to reach specific stakeholders (i.e., CSOs, direct grantees staff, CSOs’ sub-grantee staff), specific names and contacts will be provided by Chemonics HQ. The evaluation team will select a sample based on the availability and location of stakeholders in the districts sampled. It is anticipated that, the Chemonics HQ would help the evaluation team to book the appointments based on availability of the participants. Due to the large number (more than 10) of potential respondents to be invited in GIs, the evaluation team will use randomized sampling to select 4-6 participants for the GI after refusals and no-shows. If a given group has only one participant, the team will 78 change to having a KII instead, following the same discussion guide as the GI. The evaluation team will use the list of training participants to purposively select them for the GI with consideration of their role in the program. Should this strategy not be feasible, the team will recruit participants through snowball sampling. GIs will be recorded with informed consent obtained prior to the start of the discussion. The discussion guide will include 9-12 questions for a 90-minute GI, starting with broader questions while probing for answers to narrower key questions. GI transcripts will be transcribed and translated by a professional firm and will be coded to produce a summary of recurrent themes that emerge in response to each topic and in alignment with research questions. When appropriate, important findings will be illustrated with key quotes. Table 3 on the following page presents a summary of the qualitative data collection activities programmed in each location. Table 3: Anticipated Data Collection Activities by Location15 Data Collection Activities Temeke and National (Dar es Salaam) Morogoro MC and Mvomero (Morogoro Region) Mufindi DC and Kilolo DC (Iringa Region) Bahi DC & Kongwa DC and National (Dodoma Region) Zanzibar (Unguja) Total KII 1 – USAID technical staff (AORs, and GATRs) 3-6 . . . . 3-6 KI12 – Pamoja Twajenga Chemonics Staff and partners: 4 . . . . 4 KII 4 IoAs – ES, NAOT, PPRA) 3-516 . . 1-3 . 4-5 KII 5 – CSOs Direct USAID Grantees CD/COP 5 . . . . 5 KII 6 – Other CSOs PT partners Country Directors/Chiefs of Party 3 2 2 2 1 12 Total KKIs 15 3 3 3 1 28-31 GI 1 – CSOs Direct Grantees other staff 2 2 2 2 . 8 GI 2-CSO Sub-grantee other Staff 2 2 2 2 . 8 Total GIs 4 4 4 4 . 16 *Actual numbers will depend on availability in the field The fielding of the KIIs and GIs in each of the four regions and seven councils will be two field teams as well as national level KIIs in Dodoma and Dar es Salaam which will be led by the team lead. They will have concurrent schedules for efficiency over a two-week data collection period. This is to be elaborated in the forthcoming detailed field plan following inputs from the Mission and IPs. This will be discussed with Chemonics or with the direct grantees if needed for planning in each of the field sites. 15 The ET anticipate several engagements with the implementing partner to clarify the stakeholder analysis which informs this table. 16 There is not IoA zonal office in Morogoro. The ET will add the coastal zonal office in Kibaha to Dar es Salaam interviews. Mbeya ES zonal office will be added if potential respondent can be reached by phone. 79 DATA ANALYSIS METHODS The evaluation team will commit both qualitative and quantitative analyses of the achievements and shortcomings of the activity in relation to the objectives, targets, and evaluation questions. All qualitative data from the KIIs and GIs will be recorded and transcribed. The evaluation team will use the qualitative software Dedoose (https://www.dedoose.com/) to code and analyze the transcribed transcripts. Microsoft Excel will be used to analyze and visualize quantitative performance monitoring data from the CDPA activity. The analyses will cover the output and outcome indicators for the activity, as well as the KII and GI data on achievement of the approach the CDPA activity operationalized. All information and data will be disaggregated, as much as possible, by the type of organization—IoA versus civil society organization, type of capacity building offered, location (national, regional, or local level) and beneficiaries of capacity development will be disaggregated by sex, and age. Table 4 below summarizes the approach that the evaluation team will use to answering each of the evaluation questions based on suggested data sources, data collection methods and data analysis method(s). Table 4: Evaluation Design Matrix Evaluation Questions Suggested Data Sources/Informants Suggested Data Collection Methods Data Analysis Method(s) 1. To what extent did the intended outcome of improved and constructive citizen/ government engagement occur over the course of the program? Documents, including annual reports, quarterly progress reports, annual evaluation reports, AIMS reports, performance monitoring data, national statistics, end of project performance report, PT project staff, CSO and IoA staff receiving technical assistance. Desk review, group and individual interviews with IoAs, grantees AORs, and CSOs. Analysis of performance monitoring data and national statistics. Recurrence analysis of themes in collected qualitative information. Requested level of disaggregation – gender, age, level of engagement – district, regional, or national level, and grantee type, etc. 2. Did the technical support provided by CDPA to USAID’s CSO partners meet the needs and priorities of the Mission related to its management of civil society grant activities? Documents, including annual Contractors performance assessment reports, AIMS reports, COR’s responses to AIMS, grantees AORs, CO, Mission technical staff, third party evaluations Desk review, key informant interviews, group interviews with CSOs Recurrence analysis of themes in collected qualitative information. 80 Evaluation Questions Suggested Data Sources/Informants Suggested Data Collection Methods Data Analysis Method(s) 3. Did the activity meet its targets in strengthening the capacity of CSOs and GoT IoAs? Documents, including annual reports, AIMS reports, COR’s responses to AIMS, grantees AORs, CO, third party evaluations Desk review and review of trend data Recurrence analysis of themes in collected qualitative information. Requested level of disaggregation – services provided, type of grantee, location (district, region), etc. 4. To what extent have the actions supported or encouraged through the CDPA activity been sustained by beneficiary IoAs and CSOs? What political and economic factors challenged CSOs activities during the course of the PT program? Documents, including quarterly and annual reports, AIMS reports, COR’s responses to AIMS, grantees AORs, third party evaluations Desk review, key informant interviews Recurrence analysis of themes in collected qualitative information. 5. Are the stakeholders satisfied with the performance of the contractor in training and providing technical assistance to the direct CSOs and IoAs? Were they able to cascade the training and what level of satisfaction do sub-grantees have with the direct grantees’ training? Documents, including AIMS reports, AORs responses to AIMS, quarterly and annual progress reports, third party evaluation, national statistics, monitoring and evaluation reports), project staff, IoAs at HQ and regional level, CSOs (direct and sub-grantees) Desk review, key informant interviews Analysis of performance data Recurrence analysis of themes in collected qualitative information. 2.4 REQUIRED MISSION AND CHEMONICS CONSULTATION In addition to doing key informant interviews with relevant USAID staff, the evaluation team will hold a minimum of two consultation meetings with the Mission DRG team and one with Chemonics to go over and agree on the the following key components of the evaluation design. The Chemonics meeting will follow after the first meeting with Mission. These consultative meeting will also help the evaluation team to get an in-depth understanding of the CDPA activity. Mission and IP staff will regularly be asked to meet with the team in the design and scoping phase and updated on key deliverables. Quick review and turn around in providing feedback on intermediate deliverables will help the team meet its tight timeline for completion. The evaluation team will need an accurate list with contact information (name, title, institutional affiliation, telephone number, and email) for Chemonics staff, Direct CSO grantees and their partners (including senior-level and technical staff), and CDPA’s main counterparts at GoT IoAs. If possible, the team will need assistance from both the Mission and the IP to contact GoT counterparts and stakeholders, other donors, and IPs that are or were active in the CDPA activity implementation. The evaluation team will also need the following from the Chemonics staff and partners: • Detailed explanation on stakeholder groups and activities; 81 • An accurate list with contact information for Chemonics project field personnel and service providers; • An accurate list with contact information for CDPA project beneficiaries including communities; and • Assistance and outreach to GoT IoA contacts at the national and regional levels; contacts for CSOs at the district levels in sites selected for fieldwork. The evaluation team will also need USAID and Chemonics project assistance to encourage participation in data collection efforts, to help increase response rates so that the team has information that is as complete as possible to evaluate the effectiveness of the project. An introduction letter from USAID will be needed to ensure participation in KIIs and GIs. 3.0 CAPACITY BUILDING PLAN Data for Development, through the evaluation process, will engage a total of five local researchers, one local institution (Utafiti Associate), and two local Data for Development staff in two-step capacity building sessions. Firstly, before data collection, evaluation staff will participate in a two-day training on note taking, field logistics management, transcription and translation of recorded audios, qualitative data collection, coding, data analysis, and subject matter knowledge as the preparation before field work in Dar es Salaam. This will aim to improve the qualitative data collection, data coding, and analysis skills for the evaluation team using a participatory approach. Participants will be trained on qualitative data analysis concepts and how to use the Dedoose software to code and analyze data. The training will involve participatory, hands on, and adult learning approaches which will enhance participants’ skills as well as experience in conducting future qualitative research. 4.0 EVALUATION LIMITATIONS It is important to identify here some limitations inherent to the design of this evaluation: • Data availability and data quality: While the evaluation team will collect and generate primary data, they will rely on the PT and the Mission providing comprehensive, good quality performance monitoring data. The assumption is that much of this was reported into the IPRS during the period of performance; if data is not available from the system or from available program documents, trend analysis of performance may be limited. • Selection bias: There are five target regions for this activity, and 10 Districts, with three IoAs (ES, NAOT, and PPRA), five CSOs direct grantees, and 10 CSOs sub-grantees. All three IoAs, five direct grantees, and 10 non-grantees was expected to be reached and all five regions covered. However, due to time constraints and geographical balance, 6-7 districts from 4-5 regions, upon approval by USAID, should be selected for evaluation. As some key informants may decline to be interviewed, there is a possibility of selection bias, i.e., those respondents who choose to be interviewed might differ from those who do not in terms of their attitudes and perceptions, affiliation with government/non-government structures, and socio-demographic characteristics and experience. In addition, the purposive nature of the site selection process introduces additional selection bias. • Recall bias: Since a number of questions raised during the interviews will address issues that took place from 2013-2017, and given that the activity is already phased out, informants might not be able to provide accurate and complete responses. They may also conflate training or support received from other programs or donors as having been from Pamoja Twajenga. 82 • Halo bias: There is a known tendency among respondents to under-report socially undesirable answers and alter their responses to approximate what they perceive as the social norm, called halo bias. The extent to which respondents will be prepared to reveal their true opinions may also vary for some questions that call upon the respondents to assess the attitudes and perceptions of their colleagues or people upon whom they depend for the provision of services. To mitigate this limitation, the evaluation team will outline confidentiality and anonymity statements to all who participate in KIIs, and GIs. The evaluation team will also conduct the interviews in a neutral setting where respondents feel comfortable. Another measure that will be taken is to have GIs with relatively homogenous participants with respect to rank. 5.0 EVALUATION TIMELINE AND DELIVERABLES 5.1 EVALUATION TIMELINE The evaluation team’s anticipated work schedule is provided below. Data for Development will have weekly meetings with the evaluation team for the duration of the work, supervising and managing the process, and ensuring smooth progress of the evaluation. The schedule is designed to provide USAID with preliminary findings at the end of fieldwork in December, and a first draft of the evaluation by mid-January. Table 5: Evaluation Timeline and Deliverables Timing Proposed Activities (SOW) November 2018 Discussion of SOW with Mission September 1-November 2018 Preparation of the work plan and evaluation design November 2018 USAID review of the work plan and evaluation design September 3-October 30, 2018 Team recruitment October 22-November 26, 2018 Preparation and submission of instruments/tools and final work plan and evaluation design October 22-November 2018 COR and CO approval of all members of the team December 3-6, 2018 IRB approvals (qualitative study will only require NORC non-human subjects’ approval) November 2,-26, 2018 Systematic Document Review December 1-2, 2018 STTA travel and in-brief preparation December 3, 2018 Team mobilization/in-briefing December 3-14, 2018 Data collection (2-3 weeks of collection with 1 day between each site back in Dar for preliminary analysis of findings and Dar interviews) December 19, 2018 Participatory findings, conclusions and recommendations meeting and STTA out briefing December 15, 2018 - January 11, 2019 Data analysis (with concurrent work on analysis during the collection period) January 11-21, 2019 Report writing January 21, 2019 Submit Draft Report January 21-February 1, 2019 USAID review of draft report February 1-15, 2019 Incorporate USAID comments and submit final report 83 Timing Proposed Activities (SOW) Early March Final report submitted TBD Utilization and action planning, after action review session. Dissemination to PO and Education team for their dissemination to IPs and public dissemination via the Development Experience Clearinghouse (DEC). TBD Upload to DEC Note: Given the short overall timeline, this schedule is predicated on timely approval of USAID/Tanzania for the Evaluation Team proposed in this document, as well as the timely feedback by USAID/Tanzania of deliverables including the instruments and the draft evaluation report. 5.2 EVALUATION DELIVERABLES 1. Evaluation Design Report with tentative evaluation work plan/schedule indicating which step/activity should occur and when to ensure that the evaluation team fully understands the assignment approach. The work plan and design will be discussed with the USAID technical team in a pre-evaluation meeting and should be approved by USAID prior to implementation. This includes the evaluation design and methodology clearly articulating how the evaluation questions will be answered. 2. Team debriefing will be held immediately following site visits and data analysis to make sure that all questions have been addressed. A debrief presentation (with a PowerPoint) of preliminary findings, issues, and recommendations will be presented/shared prior to the return of the team to their home locations. 3. The draft report. The written report will clearly describe findings, conclusions and recommendations (using the report format provided below). 4. A final report that incorporates the team’s responses to Mission comments and suggestions. 5. Final report format: executive summary, methodology, findings, and analysis of the findings of each research question. Estimated report length: 30-50 pages. 6.0 DISSEMINATION OF FINDINGS Findings for this evaluation will be disseminated to the intended users which are USAID/Tanzania Development Objective (DO3) Team, DRG office in particular, and the Agency at large through; 1. Debrief meetings: immediately after the field work, the evaluation team will prepare the preliminary findings, conclusions, and recommendations and present them to the USAID/Tanzania Mission DO3 Team for their inputs. The feedback provided will be incorporated into the in-depth analysis and report writing process. USAID/Tanzania have the mandate to disseminate the preliminary findings to the IPs and public dissemination via the DEC as they may wish. USAID/Tanzania may also request the evaluation team to do the same presentation to the IP. 2. Final Evaluation Report: The evaluation team, led by the Team Leader Lawrence Robertson, will prepare the draft report and submit to the USAID/Tanzania Mission for review. After the Mission review, the evaluation team will incorporate the comments into the report and finalize the evaluation report. The final report will be disseminated to the wider audience uploading it to the DEC. 84 7.0 PROPOSED STAFFING Data for Development has selected an exceptionally qualified team to conduct the post-performance evaluation of the CDPA activity. The team consists of one expatriate Short Term Technical Assistant (STTA) member who will travel to Tanzania; Lawrence R. Robertson, PH.D, will serve as Team Leader. The team will also include three local STTAs that have substantial experience working with CSO capacity building and Democracy and Governance as well as Edward Mhina, the Capacity Development Expert, and Bernard Kindoli, Citizen Engagement/ Participation Expert, who will conduct the data collection and analysis and make key contributions toward the evaluation report. Data for Development staff includes M&E Specialists Shakila Mayumana (Gender and Youth Specialist) and Gerald Usika (M&E, Survey and Data Collection Specialist) who will provide technical support in the evaluation and data collection and will contribute to the analysis. Jacob Laden (Evaluation Advisor) will advise the evaluation design and analysis, provide coordination and management support for fieldwork, and review all final and intermediate deliverables. Data for Development staff will support all logistics and data collection efforts in the five project sites and support the team in overall design, facilitate client and IP communication, conduct sampling, and conduct the data analysis to be used in the evaluation report. In addition, NORC and ME&A HQs will provide operational and technical support and editing and branding on draft and final deliverables prior to submission to the Mission by Data for Developments COP Brenda Pearson. The following is a detail of the Level of Effort (LOE) for the evaluation team members and the support team of Data for Development staff. Table 6: Estimated LOE per Team Member (Total LOE: 214 Days)17 Data for Development Staff - - -     Task Team Lead Citizen Engagement/ Participation Expert Capacity Developm ent/ CSO Expert M&E, Logistics and Survey Specialist Evaluatio n Advisor & Senior Research Scientist (Jacob Laden) Gender and Logistics Specialist (Shakila Mayumana) Monitoring and Evaluation Specialist￾Iringa Translator (TBD￾From Lawrence Utafiti) Robertson, PH.D/ Bernard Kindoli Edward Mhina (Gerald Usika) Daudi Siwalaze Estimated LOE (Days) Document review/desk review/work planning (evaluation design remote or in￾country) 5 3 3 1 1 1 - - 17 LOE and Team composition matches the SOW. 85 Data for Development Staff - - -     Task Team Lead Citizen Engagement/ Participation Expert Capacity Developm ent/ CSO Expert M&E, Logistics and Survey Specialist Evaluatio n Advisor & Senior Research Scientist (Jacob Laden) Gender and Logistics Specialist (Shakila Mayumana) Monitoring and Evaluation Specialist￾Iringa Translator (TBD￾From Lawrence Utafiti) Robertson, PH.D/ Bernard Kindoli Edward Mhina (Gerald Usika) Daudi Siwalaze Estimated LOE (Days) Preparations for travel and organizing data collection (contracting translators, vehicles, etc.). 2 2 2 1 - 1 2 - Instrument development, evaluation design, protocol preparation and submission 2 2 2 1 1 1 - - Preparations for data collection (scheduling) 1 1 1 1 1 1 1 - In-briefing 1 1 1 1 1 1 1 - National-level data collection in Dar es Salaam 3 3 3 1 - 1 - 12 Field visits and data analysis 12 12 12 12 1 12 12 - Preliminary analysis 4 4 3 3 2 3 2 - Out-briefing 1 1 1 1 1 1 1 - Analysis and report drafting 10 6 4 4 2 4 2 - Review and submit Final Report 5 1 1 1 2 1 - - Total 46 36 33 27 12 27 21 12 86 ANNEX IV: LIST OF INTERVIEWS 87 # of Inter views Type of Organization Name of Organization Title of the Interviewees Location of the organization Completion date 1 IoA Ethics Secretariat (ES)- HQ Director Planning, M&E (M&E focal point) Dar es Salaam December 7, 2018 2 IoA National Audit Office of Tanzania (NAOT) 1. M&E/ Project Point of Contact 2. Planning officer Dar es Salaam December 15, 2018; December 19, 2018 2 IoA Public Procurement Regulatory Authority (PPRA) 1. Ag. Capacity Building and Advisory Service 2. Director of Corporate Services Dodoma December 6, 2018; December 5, 2018 1 Donor USAID Former AOR PELUM Dar es Salaam December 17, 2018 1 Health, USAID Dar es Salaam December 18, 2018 1 NACOFA AOR Dar es Salaam December 18, 2018 1 Former AOR LEAT, Natuar Resources Manager and Water Team Leader Dar es Salaam December 20, 2018 1 Gender and Youth Specialist Dar es Salaam December 19, 2018 1 Former Iringa Hub staff, USAID Dar es Salaam December 20, 2018 1 IP Chemonics Director East & Southern Africa Region Skype December 11, 2018 2 1. Chief of Party (COP)1 2. COP2 Dar es Salaam December 12, 2018; December 17, 2018 1 Organization Development Specialist Zanzibar December 12, 2018 1 M&E Specialist Dar es Salaam December 11, 2018 1 Institutional Accountability (IoA) Specialist Iringa December 12, 2018 1 Advocacy and Linkages Specialist Dar es Salaam December 18, 2018 1 Grants Specialist Dar Es Salaam December 17, 2018 88 # of Inter views Type of Organization Name of Organization Title of the Interviewees Location of the organization Completion date 1 Subcontractor to IP Economic and Social Research Foundation (ESRF) Project Point of Contact. Head, Capacity Building Manager Dar es Salaam December 15, 2018 1 USAID Direct Grantee WiLDAF Executive Director/M&E and Project Manager Dar es Salaam December 7, 2018 1 WiLDAF CSO Subgrantee TANLAP Executive Director Dar es Salaam December 6, 2018 1 USAID Direct Grantee PELUM COP Morogoro December 4, 2018 2 PELUM CSO Subgrantee UMADEP 1. Executive Director 2. Assistant Coordinator 3. Field Officer Morogoro December 10, 2018 1 PELUM CSO Subgrantee Tanzania Grassroots Oriented Development (TAGRODE) Field Coordinator Iringa December 10, 2018 3 USAID Direct Grantee LEAT 1. Executive Director 2. Other staff 3. Project Coordinator 1. and 2. Dar es Salaam 3. Iringa December 14, 2018; December 12, 2018; December 4, 2018 1 LEAT CSO Subgrantee MBOMIPA Members of MBOMIPA Executive Committee Iringa December 5, 2018 1 LEAT CSO Subgrantee Mufindi Vijana kwa Maendeleo (MUVIMA) Project Officer Iringa December 6, 2018 1 USAID Direct Grantee TACOSODE Executive Director Dar es Salaam December 4, 2018 1 TACOSODE CSO Subgrantee Anti Female Genital Mutilation Network (AFNET) Executive Director Dodoma December 4, 2018 1 TACOSODE CSO Subgrantee Mahenge Mazombe Development Association (MMDEA) MMDEA Management Team Iringa December 6, 2018 1 TACOSODE CSO Subgrantee IMO 1. CEO 2. M&E Officer 3. Chief Finance Officer 4. Project Coordinator Iringa December 6, 2018 89 # of Inter views Type of Organization Name of Organization Title of the Interviewees Location of the organization Completion date 1 TACOSODE CSO Subgrantee CHEDO Executive Director Iringa December 13, 2018 1 TACOSODE CSO Subgrantee Afya ya Jamii Tanzania (AJAT) Project Coordinator Dodoma December 7, 2018 1 USAID Direct Grantee NACOPHA Executive Director Dar es Salaam December 13, 2018 2 NACOPHA 1. Other Staff - Manager of Community Engagement 2. M&E NACOFA - and former M&E PT Dar es Salaam December 13, 2018 1 NACOPHA Clusters Chamwino DC Management Team Dodoma December 6, 2018 1 NACOPHA Clusters BAHI DC Management Team Dodoma December 6, 2018 1 NACOPHA Clusters Iringa MC Management Team Iringa December 7, 2018 1 NACOPHA Clusters Mufindi TC Management Team Iringa December 6, 2018 1 Other CSOs-Non￾Grantees AAC-CWMAC CEO Morogoro December 12, 2018 1 Other CSOs-Non￾Grantees HDT Director of Programs Dar es Salaam December 14, 2018 1 Other CSOs-Non￾Grantees MAT Executive Director Dar es Salaam December 12, 2018 1 Other CSOs-Non￾Grantees RECLAIM Executive Director Dar es Salaam December 14, 2018 1 Other CSOs-Non￾Grantees RUDI Project Manager Dar es Salaam December 11, 2018 1 Other CSOs-Non￾Grantees TAMWA - Mainland Project Coordinator Dar es Salaam December 14, 2018 1 Other CSOs-Non￾Grantees TAMWA - Zanzibar Executive Director Zanzibar November 29, 2018 1 Other CSOs-Non￾Grantees WCSA Program Director Dar es Salaam December 11, 2018 1 Other CSOs-Non￾Grantees ZLSC Executive Director Zanzibar December 12, 2018 1 Other CSOs-Non￾Grantees E&D/Soma Executive Director Dar es Salaam December 13, 2018. 90 ANNEX V: DISCLOSURE OF ANY CONFLICTS OF INTEREST Name Gerald Usika Title Survey Specialist, NORC Organization Data for Development Project Evaluation Position? Team Leader x Team member Evaluation Award Number (contract or other instrument) AID-OAA-1-15-00024/AID-621-TO-17-00005 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Post-performance evaluation of the USAID Pamoja Twajenga Activity I have real or potential conflicts of interest to disclose. No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. NA I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date August 1, 2018. 91 Name Shakila Mayumana, ME&A Title Gender and Youth Specialist Organization Data for Development Project Evaluation Position? Team Leader x Team member Evaluation Award Number (contract or other instrument) AID-OAA-1-15-00024/AID-621-TO-17-00005 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Post-performance evaluation of the USAID Pamoja Twajenga Activity I have real or potential conflicts of interest to disclose. No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. NA I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date August 1, 2018. 92 Name Edward Mhina Title CSO Development Consultant, ME&A Organization Data for Development Project Evaluation Position? Team Leader x Team member Evaluation Award Number (contract or other instrument) AID-OAA-1-15-00024/AID-621-TO-17-00005 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Post-performance evaluation of the USAID Pamoja Twajenga Activity I have real or potential conflicts of interest to disclose. No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. NA I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date August 1, 2018. 93 Name Lawrence Robertson Title Team Lead, ME&A Organization Data for Development Project Evaluation Position? x Team Leader Team member Evaluation Award Number (contract or other instrument) AID-OAA-1-15-00024/AID-621-TO-17-00005 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Post-performance evaluation of the USAID Pamoja Twajenga Activity I have real or potential conflicts of interest to disclose. No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. NA I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date August 1, 2018. 94 Name Jake Laden Title Evaluation Advisor, NORC Organization Data for Development Project Evaluation Position? Team Leader x Team member Evaluation Award Number (contract or other instrument) AID-OAA-1-15-00024/AID-621-TO-17-00005 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Post-performance evaluation of the USAID Pamoja Twajenga Activity I have real or potential conflicts of interest to disclose. No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. NA I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date August 1, 2018. 95 Name Bernard Kindoli Title CSO Development Consultant, ME&A Organization Data for Development Project Evaluation Position? Team Leader x Team member Evaluation Award Number (contract or other instrument) AID-OAA-1-15-00024/AID-621-TO-17-00005 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Post-performance evaluation of the USAID Pamoja Twajenga Activity I have real or potential conflicts of interest to disclose. No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. NA I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date August 1, 2018. 96 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523