EVALUATION REPORT PERFORMANCE EVALUATION OF THE AFRICA BIODIVERSITY COLLABORATIVE GROUP (ABCG II) October 25, 2021 This publication was produced by ZemiTek LLC and Camris, Inc. at the request of the United States Agency for International Development. It was prepared independently by evaluators Michael Cooper and Dr. Arthur Mitchell. The authors’ views expressed in this document do not necessarily reflect the views of USAID or the United States Government. 2 PERFORMANCE EVALUATION OF THE AFRICA BIODIVERSITY COLLABORATIVE GROUP (ABCG II) EVALUATION REPORT October 25, 2021 USAID Bureau for Africa, Office of Sustainable Development, Division of Economic Growth, Environment, and Agriculture (AFR/SD/EGEA) CONTRACT # AID-OAA-C-15-00132 Cover Photo Credit: Arthur H. Mitchell, Mikumi National Park, Tanzania 3 TABLE OF CONTENTS ACRONYMS 3 EXECUTIVE SUMMARY 4 FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS 4 INTRODUCTION 10 EVALUATION PURPOSE 10 EVALUATION QUESTIONS 10 EVALUATION AUDIENCE AND INTENDED USE 10 BACKGROUND 11 ABCG II OVERVIEW 11 THEORY OF CHANGE 12 METHODOLOGY 14 FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS 15 EVALUATION QUESTION 1: THEORY OF CHANGE 15 CONCLUSIONS 15 FINDINGS 16 EVALUATION QUESTION 2: IDEAS 18 CONCLUSIONS 19 FINDINGS 19 EVALUATION QUESTION 3: INFLUENCE 19 CONCLUSIONS 20 FINDINGS 20 EVALUATION QUESTION 4: IMPACT 21 CONCLUSIONS 21 FINDINGS 21 EVALUATION QUESTION 5: ORGANIZATION AND GOVERNANCE 22 CONCLUSIONS 22 FINDINGS 23 EVALUATION QUESTION 6: PANDEMIC 24 CONCLUSIONS 24 FINDINGS 24 ISSUES AND FUTURE DIRECTIONS 24 REFERENCES 25 ANNEXES 33 Annex I. USAID ABCG II Evaluation Statement of Work Annex II. Evaluation Methodology Annex III. Key Informant Interview Summary Report Annex IV. E-Survey Report Annex V. Summary of Annual Indicator Reporting from FY 2016-2020 ABCG M&E Plans Annex VI. References and Sources of Information 4 ACRONYMS ABCG II Africa Biodiversity Collaborative Group Activity, Phase Two AFR USAID Bureau for Africa AWF African Wildlife Foundation BATS Biodiversity Analysis and Technical Support CBFM Community-Based Forest Management CBO Community-Based Organization CI CLA CoP Conservation International Collaborating, Learning, and Adapting (1) Community of Practice (2) Chief of Party COR CS CSO DRC E3 Contracting Officer’s Representative Conservation Standards, or Open Standards for the Practice of Conservation Civil Society Organization Democratic Republic of the Congo USAID Bureau for Economic Growth, Education, and Environment EGEA USAID Division of Economic Growth, Environment, and Agriculture EQ Evaluation Question FAB USAID Office of Forests and Biodiversity FGD FW-WASH FY Focus Group Discussion Freshwater Conservation & Water, Sanitation, and Hygiene Financial Year GCI Global Change Impacts IbD KAP KII LOA Impact by Design Knowledge, Attitudes, and Practice Key Informant Interview Life of Agreement LOE Level of Effort LRTR Land and Resource Tenure Rights LUM Land-Use Management JGI LUM MEL MSI MSTAS NGO PHE QA SC SD SMART SNA SOW SUM TL TNC ToC USAID USG WCS WRI WWF The Jane Goodall Institute Land Use Management Monitoring, Evaluation, and Learning Management Systems International Management Support and Technical Analysis Services Non-Governmental Organization Population, Health, and Environment Quality Assurance Steering Committee Office of Sustainable Development, USAID Specific, Measurable, Achievable (or Attainable), Relevant, and Time-Bound Social Network Analysis Statement (Scope) of Work Scaling-Up Management Team Leader The Nature Conservancy Theory of Change United States Agency for International Development United States Government Wildlife Conservation Society World Resources Institute World Wildlife Fund – US 5 EXECUTIVE SUMMARY The USAID Bureau for Africa, Office of Sustainable Development, Division of Economic Growth, Environment, and Agriculture (AFR/SD/EGEA) contracted ZemiTek LLC (Contract # AID-OAA-C-15- 00132) to conduct this final performance evaluation. As stated in the evaluation Statement of Work (SOW), the purpose of this consultancy is “to conduct an end-of-activity performance evaluation of the Africa Biodiversity Collaborative Group (ABCG) II activity, a $7.5 million cooperative agreement with the Wildlife Conservation Society (WCS) funded by AFR/SD/EGEA from September 2015 - September 2021. Subgrantees include the African Wildlife Foundation (AWF), Conservation International (CI), the Jane Goodall Institute (JGI), The Nature Conservancy (TNC), the World Resources Institute (WRI), and the World Wildlife Fund (WWF).” This evaluation is a measure of ABCG II’s performance against its stated objectives by focusing on findings, conclusions, and recommendations for six Evaluation Questions (EQ). FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS EVALUATION QUESTION 1. THEORY OF CHANGE: How has progress been made toward achieving ABCG’s theory of change? What specific assumptions in the theory of change hold true? EQ 1 FINDINGS AND CONCLUSIONS Content: The overall ABCG Theory of Change relies heavily on knowledge transfer activities to achieve improved conservation practice amongst various stakeholders. However, this knowledge transfer, and the resulting change in stakeholder conservation practice, is assumed and not clearly articulated. Given that knowledge transfer outcomes include a lot of uncertainty, this uncertainty can be partly addressed with short term objectives and assumptions that speak to the mechanisms of knowledge transfer. Process: The process to develop ABCG’s strategic priorities has increased in its inclusivity of African perspectives and inclusivity is a stated priority moving forward for ABCG. ABCG strategic planning processes did have a significant, but diminishing, gap in garnering those stakeholders’ perspectives who are most influential for ABCG outcomes, namely those African government officials, communities, and conservation actors whose actions are necessary for ABCG to achieve its outcomes. EQ 1 RECOMMENDATIONS ● Outcome measures: ABCG could gain from outcomes that are designed using SMART criteria. But this is only as valuable as a new MEL strategy that 1.) supports an adaptive management process focusing on 2.) knowledge transfers and the mechanisms the drive the behavior change for the envisioned conservation practice. ● Business modeling: ABCG could integrate business planning tools (like those common to developing a Business Model Canvas) to help inform future ABCG business planning processes. ● Consistency with ABCG learning: ABCG has taken strides to ensure its theory of change is informed through lessons learned and updated evidence. These gains could be capitalized by developing an ABCG Learning Agenda that is designed and managed to inform ABCG Business Planning needs. EVALUATION QUESTION 2. IDEAS: How has ABCG advanced understanding and dissemination of new advances and approaches in Africa? EQ 2 FINDINGS AND CONCLUSIONS Supply Driven versus Demand Lead: New conservation practices (and the knowledge to implement them) could be considered ABCG products that it is supplying against beneficiary (e.g., customer) and funder demand. ABCG faces an uphill battle given that many biodiversity outcomes are dependent on behavior for which there is less demand compared to other competing interests (i.e. getting actors to decrease bushmeat consumption during food insecurity, etc.). ABCG has had success as member organizations have 6 good practices in identifying or even cultivating demand for its products. With this, ABCG is over dependent on supply lead solutions, which has been a barrier to ABCG outcomes of improved conservation practice. EQ 2 RECOMMENDATIONS ● Push and Pull Products: As ABCG transitions to new business models, assessing the level of demand for ABCG activities and products could become more important to securing funding and achieving outcomes. Developing tools and processes for articulating the level and type of demand for various ABCG products (organized by type of beneficiary, funder, etc.) could be valuable input into any business planning. EVALUATION QUESTION 3. INFLUENCE: How has ABCG influenced conservation practice, approaches, and linkages among its member institutions, the U.S. conservation community and in Africa (governments, academic institutions, NGOs, and the private commercial sector)? EQ 3 FINDINGS AND CONCLUSIONS Working Groups Build on Pre-Existing Relationships and Programming Areas: The variance in performance amongst the Working Groups is partly attributable to the pre-existing capacity and relationships of the ABCG member organization managing that Working Group. Monitoring: More resources (funding and skill sets) are needed for monitoring to measure the linkages between influence and practice. However, these resources are not limited to funds but also the planning and analytical skill sets needed to implement monitoring strategies that are informative to adaptive management, business planning and reporting requirements. EQ 3 RECOMMENDATIONS ● Convening versus Collaboration: ABCG could benefit from operational definitions of Convening versus Collaboration as a way to offer Convening and Collaboration services and an ABCG product. These operational definitions could be expanded into strategies that are reflected in ABCG theories of change, planning, and adaptive management. ● New Skill Sets: ABCG could benefit from integrating agile/adaptive management skills around key Collaborating, Learning, and Adapting (CLA) practices and functions at different phases of the lifecycle, with some skills being institutionalized within ABCG and others being procured as needed. EVALUATION QUESTION 4. IMPACT: How has ABCG contributed to the improved capacity, alliances, activities, and other impacts of the conservation community from both the U.S. and Africa bases? EQ 4 FINDINGS AND CONCLUSIONS While Scale Up Pathways are Envisioned, they are not Planned: ABCG II has a stated priority to develop the necessary partnerships for achieving improved biodiversity conservation practices at scale, but this vision does not have a correlated strategy to achieve this goal. Some Working Groups have had success in achieving the necessary partnerships for scaling outcomes, but there is a need for strategic scale up pathway planning. Lack of Impact Measures: There is inadequate emphasis on outcomes and impact. A higher priority is placed on program outputs as reflected in the disparity of outcome measures. EQ 4 RECOMMENDATIONS ● Outcome Measures: ABCG could integrate the Conservation Standards (including SMART criteria) into its processes for developing its relevant Theories of Change with an emphasis on ensuring that outcomes are sufficiently defined and measured. 7 ● Scale-Up Pathway Planning: Scale up planning could be integrated into ABCG’s individual product development as well as business planning using pre-existing USAID tools and guidance. EVALUATION QUESTION 5. ORGANIZATION AND GOVERNANCE: How have the ABCG organizational structure, internal and external relationships, and methods of operation and collaboration facilitated or frustrated accomplishment of ABCG objectives? EQ 5 FINDINGS AND CONCLUSIONS Funding as an Incentive: There is an overreliance on USAID funding that has hindered effective business development planning. ABCG members have increasingly used ABCG to secure funding to build on pre￾existing relationships from previous programs that were successful. This was the default strategy for use of funds when a cohesive ABCG strategy was slow to materialize. Form over Function: it is not clear how the transition to new African leadership on the ABCG Steering Committee will diminish the primary ABCG barriers to performance, especially in the face of uncertainty regarding future funding. This is a stated priority for ABCG in its future planning. EQ 5 RECOMMENDATIONS ● Comparative Advantage Strategy: ABCG could develop a catalog of biodiversity problems that it can address most effectively based on its comparative advantage. ● User Profiles and Value Propositions: Identifying the types and categories of problems ABCG is best placed for, mapping the actors (e.g., funders, NGO’s) with active interests addressing the problem category could be informative to ABCG Business Planning. Especially the ABCG stated needs for developing User Profiles and Value Propositions. EVALUATION QUESTION 6. PANDEMIC: How is the COVID-19 pandemic expected to impact the objectives and goals of ABCG for the future? EQ 6 FINDINGS AND CONCLUSIONS ABCG is Built for Success: ABCG is comprised of organizations with varying levels of African membership in Working Groups and in the field through partnerships with African conservation groups, community organizations, and academic institutions. The Working Groups, such as FW-WASH, that have substantial networks and relationships with African partners were able to decentralize and delegate responsibilities to these groups to ensure that COVID disruptions were minimized. Such relationships are also conducive to ABCG’s localization priorities and strategy; hence, this performance during COVID could be valuable to any possible ABCG Localization Strategy moving forward. EQ 6 RECOMMENDATIONS ● Scenario Planning: As ABCG continues its emphasis on scenario planning around key assumptions, it could also include scenarios related to loss of access to key resources (whether due to pandemic, weather, instability, etc.). ABCG could also invest in remote management protocols and technologies (e.g., virtual, or online training) to facilitate operations during this loss of access. ● Greater Emphasis on Health: ABCG could prioritize essential health practices like WASH and provide commodities (e.g., soap, chlorine disinfectant) to supplement other activities. Doing so could incentivize greater participation amongst communities who do not see the value in conservation activities during a pandemic. 8 FULL RECOMMENDATIONS EVALUATION QUESTION 1. THEORY OF CHANGE: How has progress been made toward achieving ABCG’s theory of change? What specific assumptions in the theory of change hold true? EQ 1 RECOMMENDATIONS The evaluation used the Conservation Standards to provide the criteria for assessing ABCG’s theories of change and to organize the recommendations under 1.) Content (i.e., outcomes are SMART, causal linkages are reasonable) and 2.) Process (i.e., effectiveness of the organizational process and methods for creating the theories of change). Overall Theory of Change: Content Assumptions: There are few documented assumptions in any of ABCG theories of change. There are “informal” assumptions that are often discussed during internal deliberations, but the lack of documented assumptions that can be tested, measured, and assessed as part of an adaptive management process, hinders ABCG’s ability to accurately monitor performance. Outcome measures: Outcome measures: 76tuyfgh ABCG could gain from outcomes that are designed using SMART criteria. But this is only as valuable as a new MEL strategy that 1.) supports an adaptive management process focusing on 2.) knowledge transfers and the mechanisms the drive the behavior change for the envisioned conservation practice. Business modeling: ABCG could integrate business planning tools (like those common to developing a Business Model Canvas) to help inform future ABCG business planning processes. Overall Theory of Change: Process Consistency with ABCG learning: ABCG has taken strides to ensure its theory of change is informed through lessons learned and updated evidence. These gains could be capitalized by developing an ABCG Learning Agenda that is built around its Business Planning needs. EVALUATION QUESTION 2. IDEAS: How has ABCG advanced understanding and dissemination of new advances and approaches in Africa? EQ 2 RECOMMENDATIONS Ideas, i.e., knowledge, can be considered an ABCG product and could be designed as such which would entail some changes to how ABCG approaches designing and managing its activities. Push and Pull Products: Push and Pull products are distinguished by the level of demand from the customer. Push products have less demand while Pull products have more. ABCG has both Push and Pull products. Awareness of the level of demand for ABCG products could grow in importance as ABCG transitions to new business models. EVALUATION QUESTION 3. INFLUENCE: How has ABCG influenced conservation practice, approaches and linkages among its member institutions, the U.S. conservation community and in Africa (governments, academic institutions, NGOs, and the private commercial sector)? EQ 3 RECOMMENDATIONS Influence is the result of a relationship between ABCG and the actor envisioned to perform some type of conservation action. These relationships vary depending on the conservation action required as ABCG uses its ability to Convene and Collaborate to foster these relationships. Convening versus Collaboration: ABCG could benefit from operational definitions of Convening versus Collaboration. The two are largely conflated in ABCG practice resulting in expectations that Convening will automatically result in Collaboration and resulting conservation actions. However, this assumption 9 rarely holds true unless there is an effective strategy to capitalize on Convening and leverage it into Collaboration and Coordinated action. These operational definitions could be expanded into strategies that are reflected in ABCG theories of change, planning, and adaptive management. New Skill Sets: Many ABCG members noted a prioritization of Convening over Collaborating because Convening is easier to accomplish while Collaboration requires sustained interaction using adaptive management and program management skill sets that are not always present in the Working Groups. ABCG could benefit from integrating these skills , with some skills being institutionalized within ABCG and others being procured as needed. Customer Profiles: Influence is highly subjective and context dependent. As ABCG seeks to understand how to better use its strengths to maximize the type of influence it seeks, profiles that map out individual customer (funders, beneficiaries, etc.) needs, incentives, perceptions, etc. could help to design a tailored strategy for achieving the influence desired. There are various customer profile tools that can assist with this. EVALUATION QUESTION 4. IMPACT: How has ABCG contributed to the improved capacity, alliances, activities, and other impacts of the conservation community from both the U.S. and Africa bases? EQ 4 RECOMMENDATIONS Impact Measures: ABCG is often not able to effectively monitor its impact due to a lack of outcome measures. This is partly due to the complexity of outcomes ABCG pursues but also from a lack of impact measurement strategy that accounts for this level and type of complexity. ABCG could develop a strategy built on the Conservation Standards for measuring impact. Such a strategy could integrate various systems and complexity management tools1 in an effective monitoring strategy. Scale-Up Pathway Planning: ABCG envisions many of its activities going to scale but there is no sufficient strategy in place to develop the right partnerships in the right way at the right times in the activity lifecycles. Scale up planning could be integrated into ABCG’s business planning, individual product development and activity design using pre-existing tools and guidance.2 EVALUATION QUESTION 5. ORGANIZATION AND GOVERNANCE: How have the ABCG organizational structure, internal and external relationships, and methods of operation and collaboration facilitated or frustrated accomplishment of ABCG objectives? EQ 5 RECOMMENDATIONS Comparative Advantage Strategy: ABCG could develop a catalog of biodiversity problems that it can address most effectively based on its comparative advantage. One of ABCG’s stated comparative advantage is its ability to Convene and Collaborate, which are only as valuable as their contribution to solving biodiversity problem at hand. While ABCG has made strides in its current Business and Work plans to specify biodiversity problems it seeks to address, there is no strategy that ties its comparative advantages to these problems and how ABCG’s is better situated to solve these problems as opposed to other actors. User Profiles and Value Propositions: The types of problems ABCG seeks to alleviate are experienced by specific communities and actors. ABCG has recognized this and some of its Working Groups have used impactful localization and collective impact strategies in response. However, there is opportunity to build on this success and institutionalize the development of User/Customer Profiles (those actors who use or 1 Program Cycle Guidance Discussion Note: Complexity-Aware Monitoring: https://www.usaid.gov/sites/default/files/documents/1865/201sad.pdf 2 Taking Innovations to Scale: Methods, Applications, and Lessons: https://www.usaid.gov/sites/default/files/documents/1865/v5web_R4D_MSI-BrookingsSynthPaper0914- 3.pdf 10 fund ABCG goods and services) and accompanying Value Propositions for potential ABCG partners, clients, funders, etc. ABCG has expressed a need for both tools as they would: ● Streamline business development by focusing the ABCG programming areas ● Create formative inputs for funding proposals and other communications ● Serve as the structure for an ABCG Learning Agenda and adaptive management strategy ● Inform modification to an ABCG governance strategy that can best maximize the benefits of the ABCG comparative advantage EVALUATION QUESTION 6. PANDEMIC: How is the COVID-19 pandemic expected to impact the objectives and goals of ABCG for the future? EQ 6 RECOMMENDATIONS COVID-19 disrupted ABCG implementation but, per performance measures, not significantly. However, ABCG could implement some specific reforms that could help it prepare for future disruptions like a pandemic. Scenario Planning: The primary effect of COVID-19 on ABCG was loss of access to activities in the field and some loss of contact with conservation stakeholders at the field level where contact was more dependent on face-to-face interaction. As alluded to above, as ABCG makes improvements in its scenario planning around key assumptions, it could also include scenarios related to loss of access to key resources (e.g., due to pandemic, weather, instability). ABCG could also invest in remote management protocols and technologies (e.g., virtual, or online training) to facilitate operations during this loss of access. Greater Emphasis on Health: ABCG could prioritize essential health practices like WASH and provide commodities (e.g., soap, chlorine disinfectant) to supplement other activities. Doing so could incentivize greater participation amongst communities who do not see the value in conservation activities during a pandemic. 11 INTRODUCTION EVALUATION PURPOSE As stated in the evaluation SOW, the purpose of this evaluation is to evaluate the performance of the ABCG II project to provide lessons learned and help guide future project design. The evaluation determined whether the goals, objectives, and outcomes of ABCG II were met; establish how effectively the interventions were delivered; assess the extent to which the projects affected its beneficiaries; and determine whether the USG investment in the program received the greatest possible return. To achieve this, the evaluation assessed project performance, including its relevance, effectiveness, efficiency, and potential sustainability of results and likelihood of impact. Specifically, this evaluation was meant to 1.) measure accountability for the USAID investment in ABCG II, 2.) inform future business planning for ABCG, and 3.) provide evidence for biodiversity programming for the community of practice at large. The accountability function was accomplished by focusing on key performance measures procedures and adaptive management practices given that ABCG was mandated with implementing activities in fluid and uncertain contexts. ABCG’s methods of adaptation are not only a formative learning point, but also a measure of accountability per its mandate. This evaluation further supports the ABCG 2020 Business Plan, developed through a participatory planning process, to improve ABCG’s influence, effectiveness, impact, and sustainability of biodiversity conservation initiatives. To achieve this, ABCG identified several priorities: ● A new emphasis on African leadership ● An updated value proposition that articulates how and why this collaborative is unique ● A governance structure that can meet future needs ● Fresh fundraising strategies to ensure adequate resources to achieve goals ● A new budget that demonstrates future financial stability EVALUATION QUESTIONS The evaluation objectives were addressed by answering the following Evaluation Questions (EQ): 1. Theory of Change: How has progress been made toward achieving ABCG’s theory of change? What specific assumptions in the theory of change hold true? 2. Ideas: How has ABCG advanced understanding and dissemination of new advances and approaches in Africa? 3. Influence: How has ABCG influenced conservation practice, approaches, and linkages among its member institutions, the U.S. conservation community and in Africa (governments, academic institutions, NGOs, and the private commercial sector)? 4. Impact: How has ABCG contributed to the improved capacity, alliances, activities, and other impacts of the conservation community from both the U.S. and Africa bases? 5. Organization & Governance: How have the ABCG organizational structure, internal and external relationships, and methods of operation and collaboration facilitated or frustrated accomplishment of ABCG objectives? 6. Pandemic: How is the COVID-19 pandemic expected to impact the objectives and goals of ABCG for the future? EVALUATION AUDIENCE AND INTENDED USE The USAID Bureau for Africa, Office of Sustainable Development, Division of Economic Growth, Environment, and Agriculture (AFR/SD/EGEA) is the primary audience for the accountability measures. AFR/SD/EGEA and especially ABCG are the primary intended audience for the adaptation measures. To improve the utility of the evaluation for USAID and ABCG, the evaluation team engaged with both to identify priority issue areas within each evaluation question. Areas were identified that had higher levels 12 of uncertainty and/or importance to future ABCG planning. These areas included governance, value proportions, scale up pathways, and knowledge creation/dissemination. The secondary audience is intended to be the biodiversity community of practice at large, both within the conservation communities of the U.S. and Sub-Saharan African countries. In Africa, this includes national and local governments and decision-makers. Severe reduction of biodiversity, leading to large-scale extinctions of species and ecosystems, is the consequence of complex social, economic, and political problems facing the global population. ABCG and its member organizations are considered among the leading actors in this space. Given this position of prominence and influence, ABCG can be considered as a quality supplier of evidence and guidance for identifying and responding to biodiversity challenges in Africa and the resulting impacts on human well-being. BACKGROUND ABCG II OVERVIEW The USAID Bureau for Africa, Office of Sustainable Development, Division of Economic Growth, Environment, and Agriculture (AFR/SD/EGEA) has contracted ZemiTek LLC (Contract # AID-OAA-C￾15-00132), which in turn has contracted two independent evaluator consultants to conduct a final performance evaluation of the USAID-supported Africa Biodiversity Collaborative Group (ABCG) II activity. As stated in the evaluation Statement of Work (SOW), found in Annex I, the purpose of this consultancy is “to conduct an end-of-activity performance evaluation of the Africa Biodiversity Collaborative Group (ABCG) II activity, a $7.5 million cooperative agreement with the Wildlife Conservation Society (WCS) funded by AFR/SD/EGEA from September 2015 - September 2021. Subgrantees include the African Wildlife Foundation (AWF), Conservation International (CI), the Jane Goodall Institute (JGI), The Nature Conservancy (TNC), the World Resources Institute (WRI), and the World Wildlife Fund (WWF).” The collaborative group is managed by a Secretariat and its ABCG Director, based at AWF, a Steering Committee composed of representatives from each of these seven partner non-governmental organizations (NGOs) and a Communications and Engagement Specialist based in Nairobi, Kenya. The USAID Africa Bureau (USAID/AFR) commenced its support to ABCG with a two-year $500,000 grant to WWF in 2008–2009 to provide technical support and share lessons learned that would “assist USAID/AFR/SD, Africa Missions, and local and national organizations in Africa to increase their effectiveness in tackling major existing and emerging threats to Africa’s biodiversity and contribute to sound development based on the wise use of natural resources and maintenance of ecosystem services.” ABCG was subsequently granted $2,500,000 to support a five-year extension for 2009–2014. In 2011, ABCG received a $4,700,000 amendment to the five-year award, for a LOA (life of agreement) total cost of $7,203,924; a no-cost extension request was approved by USAID up to March 15, 2015. A performance evaluation of the 2009 - 2015 award (ABCG I) was conducted in 2015.3 3 USAID (2015). Africa Biodiversity Collaborative Group. Performance Evaluation Report. Prepared by David Callihan, Karen Menczer, and Elizabeth Freudenberger under the Management Support and Technical Analysis Africa Biodiversity Collaborative Group (from SoW) “Through long-term collaboration with its members, ABCG plays a critical convening role, and is well-known to be an objective, innovative thought leader on emerging and high-priority threats facing biodiversity in Africa. ABCG has been fortunate to have worked for many years with support from USAID to generate knowledge, foster communities of practice, and share best practices with a range of stakeholders, from local communities to global conservation professionals to policy and decision makers in Africa, the US, and beyond. ABCG work has assisted USAID and others in better understanding conservation challenges and solutions and has made important interdisciplinary connections across different technical sectors. ABCG also provides program planning, implementation, evaluation, knowledge management, and outreach support to USAID biodiversity conservation programs in Africa.” 13 Subsequently, USAID granted a three-year $4,500,000 award, termed ABCG II, which covered September 2015 - September 2018 (Cooperative Agreement No.: AID-OAA-A-15-00060). In 2018 this agreement received a two-year, $3,000,000 cost extension, making the total five-year LOA cost $7,500,000. Due to implementation delays caused by the COVID-19 pandemic, in July 2020 ABCG II received an additional 12-month no-cost extension. The end of the award’s period of performance, six years, was September 19, 2021. Further background may be found in the SOW (Annex I). As stated in the ABCG II FY 2020 Annual Report, and in line with its mission, “ABCG creates innovative conservation solutions by fostering collaborative and adaptive learning opportunities that help practitioners improve, scale, and replicate, while generating valuable user-driven knowledge that can be disseminated globally. ABCG members share their knowledge, and co-create practical tools to address conservation challenges, inform policy, and integrate biodiversity conservation into international development agendas.” Further, ABCG aims to 1) mainstream biodiversity conservation in human well￾being and development agendas; 2) promote good conservation practices; and (3) strengthen the role of social and development institutions in biodiversity conservation and human well-being. To this end, ABCG II is currently implementing its programs through six thematic groups that 1) analyze critical issues affecting sub-Saharan Africa’s biodiversity conservation efforts; 2) design and implement pilot studies to test innovative approaches for addressing those issues; and 3) synthesize collective lessons from field activities to share with multi-sector stakeholders for data-driven decision making and integration. ABCG II also promotes building strong partnerships with African institutions as the foundation for fostering communities of practice (CoP). ABCG II’s current thematic groups are: 1. Land Use Management (LUM), including the focus of an earlier group, Land and Resource Tenure Rights (LRTR) 2. Managing Global Change Impacts (GCI) on Biodiversity 3. Community-Based Forest Management (CBFM) 4. Global Health Linkages to Conservation: Population, Health, and Environment (PHE) 5. Global Health Linkages to Conservation: Freshwater Conservation and Water, Sanitation, and Hygiene (FW-WASH) 6. Integrating Gender and Vulnerable Populations in Activity Design and Implementation THEORY OF CHANGE ABCG has evolved not only its overall/strategic theory of change but the theories of change for its Working Groups as well. Given the extent of the Working Group theories of change, and the limited space for this report, we only cover the overall Strategic ABCG II ToC in this section. The evolution of ABCG’s theory of change over its history has been informed through consistent use of the Conservation Standards from early Situational/Conceptual models to Figure 1 below which is the latest ABCG theory of change. Outcomes in this theory of change are dependent on pilot studies going to scale and knowledge transfer activities leading to new, and sustained, conservation practices. Services Project (MSTAS), implemented by the Pragma Corporation and Management Systems International. USAID Africa Bureau Office of Sustainable Development, January 2015, Washington, DC. 14 Figure 1. ABCG II theory of change from the 2019-2020 workplan METHODOLOGY As specified in the SOW (Annex 1), the Evaluation Team proposed a full Evaluation Design in the Inception Report and Workplan (July 2, 2021) “an evaluation process that can successfully complete the entire evaluation exercise.” As part of the methodology, the evaluation team developed data collection tools and an analysis plan for each Evaluation Question as outlined in Annex II. Annex II presents extracts from the Inception Report methodology and updates. We present approaches to data collection and analysis, quality assurance, biases, and other potential limitations as well as dissemination and utilization of evaluation findings. The sampling, data collection methods, analysis, and reporting outlined in the Inception Report were adhered to, and Annex II contains a full listing of all qualitative, quantitative, and document review methods performed in the evaluation. The only deviation from the Inception Report was shortening the two E-surveys as an attempt to diminish the significant non￾response bias experienced in the 2015 evaluation E-surveys. Data collection tools are presented in the Key Informant Interview Summary Report (Annex III) and the E-Survey Report (Annex IV). FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS USAID has made a clear distinction among evaluation findings, conclusions, and recommendations. The following section is organized by Evaluation Question (EQ) to include: ● Findings - Empirical facts collected during the evaluation and presented as a descriptive summary of data collection, and ● Conclusions - Interpretations and judgments based on the findings and presented as performance results based on data collection and analysis. Some Findings from the 2015 ABCG Evaluation remain relevant for EQ’s 2, 3 and 4, which were the same for the 2021 Evaluation. In these cases, 2015 findings are summarized at the beginning of the section for those three EQs. 15 Recommendations, proposed actions for management, result from the Conclusions and are found as full descriptions following the Executive Summary. All recommendations for future program design and implementation flow from our conclusions, which are supported by findings. As noted in the SOW (Annex 1), “recommendations are action-oriented, practical, and specific with defined responsibility for the action.” Thus, recommendations represent a set of specific actions proposed to be considered by program management when designing or implementing similar future initiatives. EVALUATION QUESTION 1: THEORY OF CHANGE How has progress been made toward achieving ABCG’s theory of change? What specific assumptions in the theory of change hold true? CONCLUSIONS Using the Conservation Standards4, the evaluation assessed ABCG’s strategic Theory of Change (ToC) to arrive at several summary conclusions based on the content of the theory of change the process by which it was developed. Content: ABCG’s theories of change (the overall theory of change as well as the individual Working Groups theories of change) vary in the level of quality. Likewise, the absence of assumptions means that ABCG performance measures, scenario planning and risk management are not as informed as they could be. Outcomes as a change in behavior or action taken that can be measured: ABCG’s outcomes are often not measurable as documented given the lack of precision in their definition. The documentation surrounding the ToC generally describes the cause-and-effect relationships reflected in the ToC. However, there is no specific definition of each outcome that outlines the “who, what, and when” of the outcome in a manner that lends itself to the type of measurement that is conducive to accountability and learning. Process: ABCG’s overall ToC is largely a product of compromise amongst ABCG members that represents their organizational expertise and interests. There appears to be few African perspectives, evidence, or participation in the overall ToC development and varying levels of input into the six Working Group ToC. Given this conclusion, it is impossible to know the extent to which ABCG’s efforts are in line with African biodiversity and conservation priorities in terms of its ToC. Theories of Change should begin with the problem(s) that the ToC is designed to solve. This conclusion takes on more importance in the case of ABCG, which seeks to diversify its funding portfolio and develop value propositions around potential customers and funders, all of which are highly dependent on detailed problem-solving strategies. ABCG has had many informal discussions about the problems it seeks to address through its activities. Despite ABCG’s use of Situational Models (from the Conservation Standards) early in its existence, the emphasis on problem diagnostics at the strategic and Working Group levels is not as consistent and prioritized as it once was. As a result, these problem sets are rarely assessed and broken down into root drivers to inform the entry points for ABCG interventions. There appears to be little documentation on current problem diagnostic processes. Interviews with ABCG Steering Committee members noted that these processes vary widely in use across the Working Groups and are rarely documented. However, documentation requirements should be balanced against ABCG concerns over the level of effort required to meet funding requirements contrasted with available effort to implement activities. 4 Open Standards for the Practice of Conservation: https://conservationstandards.org/ 16 FINDINGS ABCG’s Theory of Change Below is a summary analysis of the Process and Content of ABCG’s ToC, providing the context to better understand the conclusions on the “how” of ABCG progress as identified in its ABCG and Working Group level ToC. Process: Prior to 2019, ABCG’s ToC were the product of internal processes. However, the Theory of Change captured in the 2020 ABCG Business Plan were the result of deliberations facilitated by Impact by Design (IbD), a research and advisory group. This expert facilitation, and the years of accumulated experience possessed by the ABCG partner organizations, contributed to an improved ToC toward the end of ABCG II implementation. Most Steering Committee respondents noted the added value of technical assistance with ToC development as well as other planning and design functions. They noted further that future technical assistance of the same sort will be critical to ABCG’s transition to different governance structures, business models, and funding sources. This technical assistance would also mitigate significant lack of resources, primarily availability of time and necessary analytical skill sets. This impediment has long been an issue for the Steering Committee members, who are obliged to perform under substantial competing interests. Content: ABCG’s overall ToC is built around maximizing what Steering Committee members described as ABCG’s Comparative Advantage, namely its ability to Convene the appropriate biodiversity conservation stakeholders (from the strategic to the field level) to collaboratively address priority problems. This Comparative Advantage is reflected in the strategic ABCG ToC where outputs and outcomes focus on using this Convening capacity to identify needs for new or pre-existing conservation practices and the knowledge to implement them. This focus also identified the method of knowledge creation, dissemination, and knowledge transfer to support the uptake of knowledge into improved practice for biodiversity conservation outcomes. However, as noted, the ToC is lacking measurable assumptions and objectives that outline the pathway from knowledge dissemination to the desired Knowledge, Attitudes and Practices needed for the conservation outcomes. Performance Measures ABCG has tracked a series of input, output, and outcome measures within each of its Annual Reports from 2016-2021. These Annual Reports largely used the same indicators to measure the same outputs, outcomes, etc. The 2019 Annual Report, however, saw significant improvement in 1.) identifying new indicators that better measure the causal chain reflected in the updated ToC, and 2.) introducing improved dissemination, data sources, and targets. These improvements make a straight comparison of indicators difficult given the transition in 2019. However, this lack of continuity is a testament to ABCG’s commitment to improving their performance through enhanced M&E and improving measurement moving forward. These improvements create the opportunity to assess ABCG performance, per its ToC, by measuring the rate by which ABCG achieves its stated outputs and outcomes annually since 2015. To do this, the evaluation followed USAID guidance on output and outcome indicator definitions to 1.) identify ABCG indicators as output versus outcome and then 2.) assess the rate by which each indicator was accomplished for the given year. The full results of this analysis are summarized in Table 1 and annually in tables presented in Annex V. However, it should be noted that all these tables only report the percentage of output and outcome targets achieved. The tables do not include the percentage of over/under-completion for the targets. This means that unless a target was 100% achieved then it not counted. The results for marginal accomplishment of Output and Outcome Targets can be found in ABCG Annual Plans. 17 Table 1. Summary of ABCG II annual output and outcome indicators FY 2016-2020 Financial Year # Output Indicators # Output Indicators Achieving Target # Outcome Indicators # Outcome Indicators Achieving Target 2016 15 5 5 1 2017 23 18 4 2 2018 35 29 14 6 2019 34 10 4 0 2020 50 19 7 1 Totals 157 81 34 10 Overall Success Rate 52% 29% The lack of Africa representation on the ABCG Steering Committee was cited most, by roughly 65%5 of Steering Committee respondents, as the primary barrier to an improved ABCG strategic ToC. Approximately 50% noted that more of a co-creation process for ToC development could create positive spillover effects into other planning and activity design functions. However, these respondents also noted that a more inclusive process does not necessarily address the need for outside technical expertise in performing addition planning, learning and adaptive management functions. Five Steering Committee members noted the value of technical assistance and facilitation provided by Impact by Design, and specifically noted the need for similar types of technical assistance as ABCG transitions to an Africa led Steering Committee and more diverse funding sources and business models. EVALUATION QUESTION 2: IDEAS How has ABCG advanced understanding and dissemination of new advances and approaches in Africa? RELEVANT 2015 AND 2021 E-SURVEY RESULTS The primary data source for this Evaluation Question in the 2015 Evaluation of ABCG I was composed of two E-surveys, one for the ABCG Listserv and one for a large group of ABCG Activity Managers. The 2015 Evaluation noted significant non-response bias (72 respondents / ca. 1400 Listserv members = ca. 5% response rate). Since low survey response rates are usually indicative of respondents feeling very strongly (positively or negatively) about the topic, relying on the results from this low response rate possibly created overly positive feedback on ABCG product (e.g., “Ideas”) dissemination methods for new advances and approaches in Africa. In an attempt to improve the 2021 E-survey response rate and reduce bias, the ABCG II evaluation shortened the expected time to complete the Listserv E-surveys from ca. 10 minutes in 2015, to ca. 3 minutes in 2021. In addition, both E-surveys were kept open for 13 days with two separate reminders sent by the ABCG Secretariat. However, despite these efforts, the 2021 evaluation had a similar non-response bias with an even lower response rate with only 121 respondents completing the survey out of 2,823 Listserv subscribers (4.3% response rate). 5 It is impossible to give exact quantifiable measures on some qualitative collection given that the KII’s were, purposefully, semi structured and designed to cast a wide net. The result is that often qualitative constructs are overlapping without clear distinctive boundaries that lend them to quantitative measurement. The evaluation did its best to analyze and report the measure accurately which means caveating them accordingly, but the lack of quantitative precision does not diminish their significance given 1.) the purpose of design of the evaluation and 2.) the level of responses reported. 18 Non-response bias6 usually stems from high levels of disinterest amongst most respondents towards the issues on which they are being surveyed. It is likely that those who do respond are doing so because they represent outlier respondents, i.e., those who have high levels of approval and support for the issues. However, it is important to note there is no conclusive evidence that this is the type of non-response bias experienced here, just a high likelihood. Such a response rate makes inclusion of analyzed responses highly questionable. While we present the two E-Survey findings in full in Annex IV, this evaluation will not draw any conclusions from them given the significant non-response bias. This prevents drawing false conclusions from faulty data. The data are presented here only because it was collected but any analysis of the data should factor in the significant non-response bias. CONCLUSIONS All conclusions for EQ2 (as well as EQ3 and EQ4) will be based on qualitative data, i.e., from interviews and documents, the Activity Manager E-Survey but not the Listserv E-Survey responses (due to the non￾response bias). However, the composition of the Listserv will be used to assess the representativeness of the audience ABCG is intended to reach with the Listserv. The main conclusions from analyzing EQ2 data concern the lack of an Africa focus and a demand driven versus supply lead. Lack of Africa Focus: The ABCG Listserv, as the primary method of disseminating ABCG products at scale, has at least 517 members inside Africa (18%). This number could be higher given that 2,387 Listserv members did not identify their nationality as requested on the listserv registration questionnaire. Fifty-six of the 72 Listserv E-Survey respondents (78%) came from countries outside Africa and thus were not representative of the African population with which ABCG seeks to collaborate. However, given the marginally low cost of the Listserv and similar dissemination methods (e.g., posting to the website) for disseminating knowledge at scale, their value-add could be significant even if performance could be improved. Demand Driven versus Supply Lead: There is no clear and consistent method or protocol in use by ABCG to determine if any new practices ABCG disseminates are in demand by the target populations. There is sometimes demand for these ABCG “products,” but there is no clear way of correlating demand vis-à-vis the resources spent producing and disseminating them. Some Working Groups have effective methods of identifying demand and using that data to design and manage activities. However, Working Group performance is largely determined by the in-house capacity and practices of the ABCG member organization managing that Working Group. FINDINGS ABCG uses various methods to disseminate its various lessons learned, best practices, etc. The primary methods include the email Listserv, publications on its website, webinars, and workshops. Each method is correlated with a different purpose. Workshops are usually designed for specific activities and physical communities while webinars are directed towards more general/virtual communities of interest. The Listserv and website postings are low-cost methods to reach all interested actors. All Steering Committee and Working Group members interviewed expressed some sort of need for improved knowledge transfer within ABCG. This is not to say that all respondents felt equally about the severity of the need, just that the need was “significant”. Most respondents phrased this in the familiar Knowledge, Attitudes, and Practice (KAP) framework, stating that ABCG assumes that the content of the communication, method of dissemination, etc. is sufficient for the population of interest. ABCG member organizations bring significant knowledge and experience in knowledge transfer to its activities, but 6 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=1691967 19 respondents identified an often-unstated assumption that current knowledge transfer activities are adequately designed and implemented for the population they seek to inform. Likewise, approximately 50% of respondents noted a lack of exploratory research or analysis to determine the best content for communication and method of dissemination for specific ABCG activities. EVALUATION QUESTION 3: INFLUENCE How has ABCG influenced conservation practice, approaches and linkages among its member institutions, the U.S. conservation community and in Africa (governments, academic institutions, NGOs, and the private commercial sector)? RELEVANT 2015 EVALUATION RESULTS The 2015 Evaluation also implemented the two E-Surveys for Listserv subscribers and Activity Managers to inform EQ3; hence, all the same non-response bias caveats also pertain to EQ3 results. The 2015 evaluation noted that “a strength of ABCG is that it also serves as a discussion platform and dissemination conduit for the broader conservation community and as an important dissemination mechanism for other work initiated by USAID/AFR/SD (including under BATS). ABCG should continue to play this role, and be alert for opportunities to include non-traditional partners, and non-ABCG members, in its discussions and activities.” This ability to convene was cited as a comparative advantage of ABCG’s. This means that this capability of ABCG is what makes it better at influencing conservation practices than other providers. However, as outlined in the following conclusions and findings, ABCG II is not capitalizing on its comparative advantage by failing to effectively collaborate across its activities. Lastly, the 2015 evaluation noted that “ABCG should develop a project monitoring and reporting system that focuses on the achievement of project outcomes, particularly in regard to the adoption of improved practices, building of partnerships and influence on policy and national programs in Africa.” This observation holds true for this 2021 evaluation as well. CONCLUSIONS Convening versus Collaboration: Per Key Informant Interviews, ABCG is very effective at Convening but less effective at Collaborating; this diminishes the returns on the resources spent to convene. Convening is a precursor to collaborating and the two require different, but overlapping, skill sets. Some skill sets ABCG has but some need to be procured or institutionalized. Working Groups Build on Pre-Existing Relationships and Programming Areas: The variance in performance amongst the Working Groups is partly attributable to the pre-existing capacity of the ABCG member organization managing that Working Group. Interviews with Working Group members suggested that performance largely stems from pre-existing relationships with key stakeholders (e.g., local conservation groups, community organizations) that have direct influence over the conservation outcomes being sought. Included in this pre-existing capacity is the level of expertise in applying the Conservation Standards, specifically the ability to co-create with community stakeholders. Monitoring: More resources are needed for monitoring to measure the causal pathways between influence and practice. However, these resources are not limited to funds but also to planning and analytical skill sets needed to fully map out the causal pathway for the various contexts within which ABCG operates. 20 FINDINGS Over half of KII respondents from ABCG II have identified the FW-WASH Working Group and its activities in Uganda as an example of success in achieving targeted outcomes. Due to this, the ABCG Steering Committee collaborated with the evaluation to use this Working Group and its activities as the subject of a summary case study, which identified the following attributes that facilitated success. These success attributes included: ● Pre-existing relationships with local partners, including government officials, community groups, conservation actors/implementers, etc. that facilitated enabling conditions and collective impact for expanded activities ● Pre-existing programs that identified needs for experimentation around advocacy activities ● Pre-existing implementation of collective impact principles to varying degrees (e.g., common measurement, backbone organization) that are utilized with ABCG funds These attributes highlight the value-add of ABCG II. The funds allowed for the flexibility required to perform the type of experimentation that is facilitated by a collective impact strategy and conducive to results. There was near consensus amongst those interviewed in the case study that this level and type of experimentation would not have been possible only with direct funding from USAID, meaning the function of ABCG as an intermediary actor was key to enabling experimentation. There was also consensus that FW-WASH (and biodiversity conservation outcomes in general) are not possible without such experimentation. This is an exceptional finding that could be explored to 1.) further explore exactly how this intermediary function enables experimentation and 2.) how this can be further built on moving forward. According to interviews with Working Group and field teams, the most common barrier to influencing behavior is the lack of effective monitoring to assess the barriers and enablers of knowledge transfer, retention and use. Given that many of these activities are knowledge transfer activities, lack of or insufficient monitoring means that ABCG II is not able to effectively assess its performance. When asked about the primary barriers to more effective monitoring, ca. 75% of respondents indicated the following barriers: ● Incentive: There is no incentive to provide monitoring data since there is no perceived correlation between monitoring and funding ● Resources: There is a heavy emphasis placed on communications and reporting, which consumes the few resources available ● Planning: Since there is little incentive, there is little emphasis placed on monitoring during activity selection and design; therefore, resources (including skill sets) are not identified or procured. Hence, when monitoring becomes an emerging priority during implementation the resources are not available. EVALUATION QUESTION 4: IMPACT How has ABCG contributed to the improved capacity, alliances, activities, and other impacts of the conservation community from both the U.S. and Africa bases? RELEVANT 2015 EVALUATION RESULTS The 2015 Evaluation used the same two E-Surveys (referenced previously under EQ2) to inform EQ4, hence all the same non-response bias caveats also pertain to EQ4 results. However, the 2015 evaluation identified two recommendations that continue to be critical barriers to ABCG impact, namely: ● Developing a project monitoring and reporting system, including tracking activity implementation, that focuses on the achievement of project outcomes, particularly regarding the 21 adoption of improved practices, building partnerships, and influence on policy and national programs in Africa. ● Improving sustainability planning, including better tracking of ABCG actions that can be sustained by mainstreaming the actions into the programs of core partners. CONCLUSIONS While Scale Up Pathways are Envisioned, they are not Planned: ABCG II has a stated priority to develop necessary partnerships for achieving improved biodiversity conservation practices at scale, but this vision does not have a correlated strategy for achieving improved practices. Some Working Groups have had success in achieving these necessary partnerships, but there is a need for strategic scale up pathway planning. Lack of Impact Measures: There is inadequate emphasis on outcomes and impact. A higher priority is placed on program outputs as reflected in the disparity of outcome measures. The full year by year breakdown of ABCG performance and impact measures are included in Annex V. FINDINGS Over half of respondents noted that given the timeline, reporting, and other requirements mandated by USAID, that activities were sometimes completed with less-than-optimal design and management to meet these requirements. Feedback from Steering Committee and Working Group members around management practices that are more conducive to impact included: ● Clear roles and responsibilities over decision making at the various phases of program lifecycles. These decisions not only included activity selection and CLA strategies but clearer authorities on strategic directions such as fundraising and planning objectives where a requirement for consensus has introduced organizational paralysis ● Inclusion of more program design, management, and M&E skill sets to help manage the inherent complexity of biodiversity conservation programs, meet funding requirements, and improve the level and type of experimentation needed. ● Clearer incentives for impact tied to funding where measurement of outcomes is a defining data point for success. Per various working group discussions, respondents noted that such an incentive structure could drive the creation of more adaptive management processes at all levels of ABCG. EVALUATION QUESTION 5: ORGANIZATION AND GOVERNANCE How have the ABCG organizational structure, internal and external relationships, and methods of operation and collaboration facilitated or frustrated accomplishment of ABCG objectives? RELEVANT 2015 EVALUATION RESULTS The 2015 evaluation found that: “there tends to be a fair amount of misunderstanding surrounding ABCG. There is a high degree of agreement and consistency within ABCG as to its purpose – which is primarily to act as an ideas generation and dissemination mechanism on high priority and emerging conservation issues for the institutions involved. Outside of the immediate membership, however, and including within USAID, the purpose of ABCG is often not clearly understood and it is frequently perceived as having a broader mandate than is actually the case.” “There is tension though about Africa leadership. Even myself I question whether our SC is the right SC for ABCG.” ~ ABCG Workshop Participant (March 5th, 2019) from Impact by Design Workshop Report 22 CONCLUSIONS Addressing EQ5 required using USAID’s best practices7 to be better able to 1.) identify the primary barriers to ABCG II performance and 2.) provide more actionable recommendations based on the effects of these barriers. Based on USAID best practice, Table 2 outlines the various dimensions of ABCG’s organization, the priority barriers for each dimension, and the effect of those barriers on ABCG II performance. Each barrier was identified by at least 90% of past and present Steering Committee respondents interviewed during the evaluation. This is significant given the wide variance in the levels and types of experience within ABCG. For example, some Steering Committee respondents had been involved for a matter of months while others had been involved for a decade or more. Likewise, some were primarily focused on specific Working Groups while others had broader experience. Table 2. ABCG organizational dimensions, priority barriers, and effects on performance Organizational Dimensions Priority Barriers Effect on ABCG II Performance Foundational Factors (hard to change embedded factors) 1.) The effects of conservation practices on biodiversity conservation outcomes requires long timelines with monitoring. 1.) There is no consistently applied strategy for identifying incentives or disincentives for the uptake of conservation practices as a part of user profiles Rules and Incentives (e.g., mandates, perceptions, norms) 1.) The time and reporting requirements attached to USAID funding consume a disproportionate share of available ABCG member resources (due to time constraints) 2.) There is a lack of funding diversity and planning for a diversified funding portfolio 3.) ABCG Members have dual incentives: i.) to secure funding for their home organizations via ABCG and ii.) a commitment to biodiversity outcomes and conservation practice 1.) ABCG is not able to fully implement solutions at times due to USAID funding requirements 2.) There is an overreliance on USAID funding that has hindered effective business development planning 3.) ABCG members have increasingly used ABCG to secure funding for their own organizations because a cohesive ABCG strategy was slow to materialize Immediate Factors (influences on short term objectives) 1.) The immediate decline of USAID funding and transition to an African lead Steering Committee 1.) The end of ABCG III funding by USAID has hastened the emphasis on member organization priorities, and it is not clear how the transition to new African leadership will alleviate core ABCG problems. FINDINGS The findings for Evaluation Question #5 are largely reflected in the “Priority Barriers” column in Table 2 and are the result of interviews with all ABCG Steering Committee and some Working Group members. These are presented in Table 2 to communicate the effects of these findings on ABCG II performance 7 Thinking and Working Politically Through Applied Political Economy Analysis: https://www.usaid.gov/documents/1866/thinking-and-working-politically-through-applied-political￾economy-analysis “I would like for it to be more of a force for change on the continent, and I don’t think it’s anywhere near that.” ~ ABCG Workshop Participant (March 5th, 2019) from Impact by Design Workshop Report 23 more easily. The effects of these findings can, in turn, help inform future ABCG organization, governance, and planning where those barriers with the most negative effects can be prioritized. EVALUATION QUESTION 6: PANDEMIC How is the COVID-19 pandemic expected to impact the objectives and goals of ABCG for the future? CONCLUSIONS ABCG is Built for Success: ABCG is comprised of organizations with varying levels of African membership at the Working Group and field levels and represented through various African conservation groups, community organizations, and academic institutions, among others. The Working Groups, such as FW-WASH, that have substantial networks and relationships with African groups, were able to decentralize and delegate responsibilities to these groups to ensure that COVID disruptions were minimized. Such relationships are also conducive to ABCG’s localization priorities and strategy; hence, this performance during COVID could be valuable to ABCG moving forward. FINDINGS In all the countries where ABCG is implementing activities there were measures restricting travel, public meetings, and gatherings. There were different experiences for each Working Group’s activities based on 1.) the COVID measures enacted in their various countries of operation and 2.) the ability to implement activities given this loss of access based on pre-existing relationships. When it became clear that it would not be possible to complete planned activities before the scheduled end the Cooperative Agreement with USAID by Sept 2020. ABCG made the request to USAID for a one-year no-cost extension in quarter three of FY 2020 which was granted. There is not much difference between the success rate for activities (as measured by their outputs) between 2019 and 2020. This may suggest that the COVID-19 pandemic did not hamper ABCG II performance. However, if only performance indicators are used, ABCG II surprisingly performed better in the 2020 reporting period during the pandemic than in 2019. However, just looking at indicators does not give a complete picture of ABCG II’s performance given that ABCG II introduced new activities and increased its scope during 2020. This included many activities that were delayed in 2019 because of the pandemic. This success during COVID was not limited to scope of activities but also the type of activities. FW￾WASH experienced higher levels of demand for health-related activities from different stakeholders to include government officials and local offices, community groups, etc. FW-WASH took strides to not only supply this demand with quality WASH programming but took advantage of the increase in demand to highlight the linkages between biodiversity outcomes and WASH, thus using the opportunity to contribute to ABCG goals. ISSUES AND FUTURE DIRECTIONS Issues There were no technical issues of note for the evaluation outside of the non-response bias for the Listserv E-Survey already referenced. The evaluation made extensive use of industry quality control protocols at each phase of the evaluation lifecycle>All data collection, analysis, and reporting materials were made available to USAID per USAID Evaluation Policy requirements and SOW tasks. 24 Future Directions The evaluation worked with ABCG to develop operational definitions for Convening and Collaboration by: 1.) Identifying ABCG examples of Internal/External Convening and Internal/External Collaboration 2.) Using the design attributes and management approach of these examples to articulate strengths and weakness in implementation 3.) Identify takeaways that can be used for ABCG business planning An initial workshop was completed with ABCG Steering Committee members and analysis of ABCG activities and planning is underway to move from #2 to #3 in the above list. REFERENCES Annex VI. References and Sources of Information provides all documents reviewed during the evaluation, including those referenced below. “Meetings, interviews, and focus group discussions,” mentioned in the SOW to be included in this section, may be found in the Inception Report and Annex II: Key Informant Interview Summary Report, which includes a list of contacts, key informants, and the KII schedule of interviews. All references cited in this report, excluding Annexes, are as follows: ABCG (2020). Africa Biodiversity Collaborative Group II FY 2020 Annual Report (October 1, 2019–September 30, 2020). Submitted to USAID Bureau for Africa /Office of Sustainable Development (AID-OAA-A￾15-00060), December 2020, Washington, DC. ABCG (2020). Africa Biodiversity Collaborative Group (ABCG) III: Collaboration for African Biodiversity (ABCG III). Proposal developed by African Wildlife Federation In partnership with Conservation International, the Jane Goodall Institute, The Nature Conservancy, Wildlife Conservation Society, World Resources Institute, and World Wildlife Fund, July 2020, Washington, DC. ABCG (2020). Africa Biodiversity Collaborative Group Strategic Business Plan. Prepared with ABCG by Impact by Design, Inc., Tacoma Park, Maryland. ABCG (2019). Africa Biodiversity Collaborative Group (ABCG II) FY 2019-2020 Work Plan, Draft Version 1I of March 2019 (September 22, 2018 - September 21, 2020). produced by Wildlife Conservation Society (WCS) for review by the United States Agency for International Development Bureau for Africa, Office of Sustainable Development (AID-OAA-A-15-00060), February 2019. ABCG (2019). Africa Biodiversity Collaborative Group Activity FY 2019-2020 Activity Monitoring, Evaluation, & Learning (MEL) Plan (September 22, 2018 - September 21, 2020). Implemented by Wildlife Conservation Society with partners African Wildlife Foundation, Conservation International, the Jane Goodall Institute, The Nature Conservancy, World Wildlife Fund, World Resources Institute, Version II: March 2019, Washington, DC. ABCG (2018). ABCG Theory of Change (Review of Strategic Focus, Results Chain Analysis, and Revised Theory of Change). PowerPoint presentation to USAID, October 2018, Washington, DC. CMP (2020). Open Standards for the Practice of Conservation, or Conservation Standards (CS), version 4.0. Conservation Measures Partnership: https://conservationstandards.org/ Cooley, Larry, and Johannes F. Linn (2014). Taking Innovations to Scale: Methods, Applications, and Lessons. Management Systems International (MSI), Results for Development Institute, Washington, DC: https://www.usaid.gov/sites/default/files/documents/1865/v5web_R4D_MSI￾BrookingsSynthPaper0914-3.pdf 25 Impact by Design (2019). ABCG Internal Assessment. Impact by Design, Inc., Tacoma Park, Maryland, March 2019. USAID (2020). Technically Strong and Politically Savvy: Enhancing Thinking and Working Politically When Practicing the Conservation Standards at USAID, Supplemental Guide 4. By Schuttenberg, H.Z., J. Stark, and K. Schueler; Biodiversity Results and Integrated Development Gains Enhanced (BRIDGE) Project, USAID Bureau for Economic Growth, Education, and Environment, Office of Forestry and Biodiversity, Washington, DC: https://biodiversitylinks.org/projects/completed-projects/bridge/bridge-resources/supplemental￾guide-4-technically-strong-and-politically-savvy-enhancing-thinking-and-working-politically-when￾practicing-the-conservation-standards-at-usaid USAID (2018). Thinking and Working Politically Through Applied Political Economy Analysis: A Guide for Practitioners. By Alina Rocha Menocal, Marc Cassidy, Sarah Swift, David Jacobstein, Corinne Rothblum, and Ilona Tservil; Center of Excellence on Democracy, Human Rights, and Governance; USAID Bureau for Democracy, Conflict, and Humanitarian Assistance (DCHA). Washington, DC: https://www.usaid.gov/documents/1866/thinking-and-working-politically-through-applied-political￾economy-analysis USAID (2016). Defining Outcomes & Indicators for Monitoring, Evaluation, and Learning in USAID Biodiversity Programming. Biodiversity How-To Guide 3. Measuring Impact, by Caroline Stem and Richard Margoluis, Foundations of Success; Marco Flores, USAID Bureau for Economic Growth, Education, and Environment/Office of Forestry and Biodiversity, Environmental Incentives, LLC, Washington, DC: https://usaidlearninglab.org/sites/default/files/resource/files/biodiversity_howtoguide3_508.pdf USAID (2016). Pathways to Scale: A guide for early-stage global health innovators on business models and partnership approaches to scale-up. USAID Center for Accelerating Innovation and Impact (CII), Washington, DC: https://www.usaid.gov/cii/pathways-scale USAID (2016). Program Cycle Guidance Discussion Note: Complexity-Aware Monitoring. Policy, Planning, and Learning (June 2016), USAID’s Office of Learning, Evaluation, and Research, Washington, DC. USAID (2015). Africa Biodiversity Collaborative Group. Performance Evaluation Report. Prepared by David Callihan, Karen Menczer, and Elizabeth Freudenberger under the Management Support and Technical Analysis Services Project (MSTAS), implemented by the Pragma Corporation and Management Systems International. USAID Africa Bureau Office of Sustainable Development, January 2015, Washington, DC: https://www.usaid.gov/sites/default/files/documents/1865/201sad.pdf 26 ANNEX I: USAID ABCG II EVALUATION STATEMENT OF WORK Africa Bureau Institutional and Technical Support Services: AID-OAA-C-15-00132 STATEMENT OF WORK Position Titles: (1) Lead Evaluator /Team Leader and (2) Evaluator Post/Bureau: AFR/SD Team Division: Division of Economic Growth, Environment, and Agriculture (EGEA), Biodiversity Team Location: Washington D.C. Point of Contact: Razia Baqai and Kirstin Siex BACKGROUND The USAID Bureau for Africa, Office of Sustainable Development, Division of Economic Growth, Environment, and Agriculture (AFR/SD/EGEA) and ZemiTek seek the services of consultant for a short￾term consultancy. The purpose of this consultancy, managed by a Lead Consultant, is to conduct an end￾of-activity performance evaluation on the Africa Biodiversity Collaborative Group (ABCG) II activity, a $7.5 million cooperative agreement with the Wildlife Conservation Society (WCS) funded by AFR/SD/EGEA from September 2015 - September 2021. Subgrantees include the African Wildlife Foundation (AWF), Conservation International (CI), the Jane Goodall Institute (JGI), The Nature Conservancy (TNC), the World Resources Institute (WRI), and the World Wildlife Fund (WWF). The consultant will report to and be under the supervision of the ZemiTek Technical Supervisor, assisted by AFR/SD/EGEA Senior Biodiversity Advisor Kirstin Siex, the AFR/SD/EGEA Biodiversity Team and the AFR/SD/EGEA Deputy Office Director. The USAID Evaluation Policy commits USAID to measure and document project achievements and shortcomings so that the Agency’s multiple stakeholders gain an understanding of the return on investment in development activities. The policy states that evaluation at USAID has two primary purposes: accountability to stakeholders and learning to improve effectiveness. ABCG HISTORY AND METHODOLOGY The African Biodiversity Collaborative Group (ABCG) is a coalition of the major US-based international conservation non-governmental organizations (NGOs) with field-based activities in Africa including African Wildlife Foundation (AWF), Conservation International (CI), the Jane Goodall Institute (JGI), The Nature Conservancy (TNC), Wildlife Conservation Society (WCS), World Resources Institute (WRI), and World Wildlife Fund (WWF). ABCG‘s mission is to tackle complex and changing conservation challenges by catalyzing and strengthening collaboration and bringing the best resources from across a continuum of conservation organizations to effectively and efficiently work towards a vision of an African continent where natural resources and biodiversity are securely conserved in balance with sustained human livelihoods. The USAID Africa Bureau (USAID/AFR) began supporting ABCG through a two-year $500,000 grant to WWF in 2008–2009 to provide technical support and share lessons learned to assist USAID/AFR/SD, Africa Missions, and local and national organizations in Africa to increase their effectiveness in tackling major existing and emerging threats to Africa’s biodiversity and contribute to sound development based on the wise use of natural resources and maintenance of ecosystem services. ABCG proposed and was awarded a five-year, $2,500,000 extension for the period 2009–2014. In 2011, ABCG was invited to submit a $4,700,000 amendment to the award, which was approved by USAID for a LOA (life of agreement) total 27 cost of $7,203,923.95. A no-cost extension request extended the end-of-agreement to March 15, 2015. A performance evaluation of the 2009 - 2015 award was conducted in 2015 (see attachment 1) In 2015 ABCG was invited to submit a proposal for a new, non-competitive follow-on award to build on the work of the previous (ABCG I) award. This new award, called ABCG II, was originally a $4,500,000, three-year award over the period September 2015 - September 2018. ABCG II’s work focused on programmatic work in four thematic areas integrating biodiversity conservation into human well-being and development: 1) Land and Resource Tenure Rights; 2) Land Use Management; 3) Managing Global Change Impacts on Biodiversity; and 4) Global Health Linkages to Conservation. A fifth working group was also formed to identify and develop strategies to respond to emerging issues affecting the biodiversity conservation issues in Africa, primarily through a small grants program. In 2018 ABCG received a two-year, $3,000,000 cost extension, bringing the total LOA cost to $7,000,000 over five years. The goals of this extension period were to continue working in thematic areas to integrate biodiversity conservation into human well-being and development as well as focus on strengthening African partner organizations and incorporate them more systematically into ABCG efforts. This was done to generate knowledge and influence conservation practice; open new spaces for women and vulnerable groups to participate in conservation and natural resource management decision making; develop and implement a Charter to document and update ABCG’s membership structure; and to take initial steps towards diversifying external funding sources to improve the sustainability of ABCG’s conservation efforts in Africa. A threats assessment process undertaken during the first months of the extension period validated the existing approaches and added one more goal: 5) Integrating Gender and Vulnerable Populations in Activity Design and Implementation. In July 2020 ABCG received a 12-month no-cost extension on the grounds of implementation delays caused by the COVID-19 pandemic. The current end of the award’s period of performance is September 19, 2021. Because all ABCG partners have substantial field programs across Africa, ABCG is able to undertake analyses and pilot studies that span multiple countries and regions as well as ensure that the learning is applied by convening groups around particular issues. Thus, ABCG activities are informed by lessons from partner field programs, and the findings, recommendations, and improvements in best practices flowing from ABCG activities feed back into long-term field programs to make them stronger and more effective. This contributes to sustainability by ensuring that (a) critical actors are invested in applying lessons flowing from ABCG’s work, and (b) best practices developed through ABCG work are incorporated into continuing discussions that ABCG partners have with their respective funding networks. Similarly, the ability to compare and contrast findings from different parts of Africa as to the effectiveness of innovative conservation strategies makes ABCG unique as these types of comparative and synthetic analyses could not be funded from individual USAID missions or even from regional programs such as the Central African Regional Program for the Environment (CARPE). Through long-term collaboration with its members, ABCG plays a critical convening role, and is well￾known to be an objective, innovative thought leader on emerging and high-priority threats facing biodiversity in Africa. ABCG has been fortunate to have worked for many years with support from USAID to generate knowledge, foster communities of practice, and share best practices with a range of stakeholders, from local communities to global conservation professionals to policy and decision makers in Africa, the US, and beyond. ABCG work has assisted USAID and others in better understanding conservation challenges and solutions and has made important interdisciplinary connections across different technical sectors. ABCG also provides program planning, implementation, evaluation, knowledge management, and outreach support to USAID biodiversity conservation programs in Africa. 28 The ABCG is not a legal entity but a voluntary partnership of conservation organizations active in Africa. The Secretariat employs two full time employees, a Director based in Washington D.C. and a Knowledge and Communication Advisor based in Nairobi, Kenya. RESPONSIBILITIES8 ● Serve as an evaluator of the final evaluation contributing to the evaluation throughout the entire evaluation process (Lead Evaluator & Evaluator) ● Work with USAID and ABCG management to plan the evaluation and logistics (Lead Evaluator & Evaluator) ● Work with the evaluation team lead to allocate evaluation responsibilities based on skills and capacity (Evaluator) ● Collaborate and support the evaluation team lead to ensure that the team fulfills obligations, communications, and deliverables in a timely manner (Evaluator) ● Oversee and support the evaluation team member to ensure that the team fulfills obligations, communications, and deliverables in a timely manner (Lead Evaluator) ● Contribute to the evaluation team’s activities and help to organize and facilitate team interaction (Evaluator) ● Manage the evaluation team’s activities and organize and facilitate team interaction (Lead Evaluator) ● Serve as liaison between USAID and the evaluation team (Lead Evaluator) ● Participate in the interpretation of findings and both oral and written presentations of findings (Evaluator)Lead interpretation of findings and both oral and written presentations of findings (Lead Evaluator) QUALIFICATIONS The Consultant must be a senior level professional with at least ten years of experience, preferably involved in similar assignments. S/he must have a Master's degree in biodiversity conservation, ecology, international development, organizational change, or equivalent. S/he must have successful experience showing competency to supervise other technical experts and to produce a high-quality report. Required qualifications include: ● Strong research and analytical skills required. ● Strong knowledge of environmental conservation in the international realm required. ● Prior experience in leading evaluation teams; excellent skills in planning, facilitation, and consensus building preferred. ● Strong team management and supervisory skills required. ● Demonstrated lead writer with strong technical English writing skills required. ● Experience working with a range of government officials, donors, local NGOs, and academia required. ● Minimum of 10 years of experience in environment and development, with at least 5 years of experience in designing, implementing, managing, monitoring, and evaluating environment programs required. ● Strong evaluation, organizational, and management skills, with a minimum of 5 years of experience with evaluation tools and methods required. ● Strong oral and written communication skills, with extensive report writing experience required. Additional experience in the following areas is preferred: ● Biodiversity conservation 8 SOW edited to include responsibilities of both the Lead Evaluator / Team Leader and the Evaluator 29 ● Strategic analysis ● Knowledge management, including social media ● Institutional capacity building ● International and domestic work experience specific to U.S. government-funded biodiversity conservation activities, especially in Africa. ● Experience working in sub-Saharan Africa. ● Assessment and evaluation experience, especially in mixed method evaluation. ● Technical expertise in results-based project design using conceptual modeling and theory-of￾change approaches to develop practical results chains, as well as in qualitative and quantitative research and data analysis, including the ability to manage large databases and survey data. ● Good familiarity with the biodiversity conservation context in Africa. ● Excellent interpersonal skills, including experience successfully interacting with host country government officials, and other stakeholders. ● Experience developing and carrying out key informant interviews as part of assessment and evaluation data collection methodologies. KEY EVALUATION QUESTIONS The purpose and goal of the evaluation is to 1) determine the extent to which the Cooperative Agreement achieved its intended original and amended objectives; 2) assess the technical, program management, and financial performance of WCS/ABCG; 3) provide recommendations on how ABCG might further its institutional objectives; 4) increase understanding on USAID’s role in furthering shared objectives, 5) highlight lessons for USAID in facilitating associations of implementing partners; and 6) inform the design of potential future activities. Performance evaluations focus on descriptive and normative questions: what a particular project or program has achieved (either at an intermediate point in execution or at the conclusion of an implementation period); how it is being implemented; how it is perceived and valued; whether expected results are occurring; and other questions that are pertinent to program design, management, and operational decision-making. Performance evaluations often incorporate before and after comparisons, but generally lack a rigorously defined counterfactual. A mix of qualitative and quantitative methods is optimal for performance evaluations. The goal is to use robust qualitative and quantitative methods that generate the highest quality and most credible evidence that corresponds to the questions being asked, taking into consideration time, budget, and other practical considerations. The key evaluation questions are: ● Has progress been made toward achieving ABCG’s theory of change? Do the assumptions in the theory of change hold true or does the theory of change need to be adjusted? ● How has ABCG moved understanding and dissemination of new advances and approaches in biodiversity conservation in Africa? ● How has ABCG influenced conservation practice, approaches, and linkages among its member institutions, the U.S. conservation community, and in Africa (governments, academic institutions, NGOs, and the private commercial sector)? ● How has ABCG contributed to the improved capacity, alliances, activities, and impacts of the conservation community from both the U.S. and Africa bases? ● How have the ABCG organizational structure, internal and external relationships, and methods of operation and collaboration facilitated or frustrated accomplishment of ABCG objectives? ● How is the COVID-19 pandemic expected to impact the objectives and goals of ABCG for the future? 30 SCOPE OF WORK, DELIVERABLES & TIMING The Consultant will provide senior technical expertise and support to the USAID Bureau for Africa and ABCG to determine the extent to which the Cooperative Agreement and ABCG are performing towards intended objectives, and to provide recommendations on how to increase the overall effectiveness of subsequent collaborations. The consultant will provide senior technical expertise in the area of data analysis, survey design and implementation, informant interviewing, and qualitative analysis for the ABCG evaluation. The consultant together with the lead evaluator will review program documentation provided by USAID/AFR/SD/EGEA and WCS/ABCG staff to understand the context and illustrative activities and document the objectives, implementation experience, and expected and actual results. Using survey tools designed by the consultant team and approved by the ZemiTek Technical Supervisor and USAID POC, the consultant will conduct Key Informant Interviews to assess whether performance was aligned with technical, management, and financial expectations. The consultant will receive support from the team in selecting and providing contact information for key informants. The consultant is expected to schedule interviews or other modes of data collection with stakeholders. The consultant together with the team lead will also develop and conduct an e-survey with ABCG Listserv subscribers to gather their perspectives on ABCG. Since data collection is proposed through an e-survey and/or telephone surveys travel may not be required to complete this work. TIME FRAME The Consultant will work closely with USAID/AFR, ABCG and other evaluators to develop a final report and presentation on the ABCG evaluation’s findings. These tasks will be carried out approximately between May 15 and October 30, 2021, according to the approximate time frames described with the deliverables below. Although the intermediate deliverables listed below will be reviewed and approved by AFR/SD/EGEA to inform the overall evaluation, they may be made available to USAID for internal use and as such should be delivered in a well-written and useful format (i.e., good draft form), whether or not they are included as appendices to the final report. The consultant team is requested to propose an evaluation process that can successfully complete the entire evaluation exercise. The consultant team should propose workday allocation (LOE) against team member(s). DELIVERABLES Illustrative Detailed Deliverables/Output Schedule Approximate Due Date Desk Review Completed 06/11/21 Draft inception Report with data collection tools 07/02/21 Final Inception Report with data collection tools 07/16/21 Key Informant Interview & E-Survey Summary Reports 08/20/21 Draft Evaluation Report submitted to USAID 09/03/21 Comments received from USAID 09/10//21 Second Draft Evaluation Report submitted to USAID and shared with Partners 09/24//21 Presentation to USAID and Partners 09/28//21 Final Evaluation Report submitted to USAID 10/25/21 The consultant together with the lead consultant will develop and submit to USAID an inception report with a proposed outline of the final report and an indicative methodology for evaluation. 31 The consultant together with the lead consultant will develop a tool/questionnaire for collecting data. The data collection tool and methodology developed will be shared with the USAID team for approval. Key Informant Interview Summary Report: The consultant together with the lead consultant will provide a narrative report highlighting the key findings from key informant interviews. The report will include the interview guide used and identify the individuals interviewed along with their relationship to the program. This report should be added as an annex to the evaluation report. E-survey for ABCG Listserv Subscribers: Survey will validate information gathered from key informant interviews. Tabulation Report of all returned e-surveys: This report will tabulate frequency of responses and characteristics of respondents and include analysis of the data. The report should be added as an annex to the evaluation report. Draft Evaluation Report. The Consultant will deliver the report through the Lead Evaluator, to the ZemiTek Technical Supervisor. The report should be jointly completed by both evaluators. The report will be a comprehensive report of all deliverables associated with this contract, and will: ● represent a thoughtful, well-presented, well-researched, and well-organized effort to objectively evaluate what worked in this project, what did not work, and why; ● be a high-quality technical report, in a professional writing style; ● address all questions included in this scope of work; ● include all the key sections: cover sheet, table of contents, acronym list/glossary of terms, executive summary, background, objectives, questions, methods, findings, conclusions and lessons learned, and recommendations; ● include the scope of work as an appendix; ● include an introduction that adequately describes the project, explains where it is implemented, includes contextual information, and includes the “theory of change” or development hypotheses that underlie the project; ● describe the evaluation methodology in detail and all tools used such as questionnaires, checklists, and discussion guides. All tools including the Key Informant Interview Summary Report, E-Survey Report, Tabulation Report and any others developed will be included in an appendix in the final report; ● describe findings with gender disaggregate outcomes and impacts; ● describe any limitations to the methods (e.g., selection bias, recall bias, unobservable differences between groups, etc.); ● present the findings as analyzed facts, evidence, and data and should not be based on anecdotes, hearsay, or a compilation of opinions. Findings should be specific, concise, and supported by strong quantitative or qualitative evidence; ● properly identify sources of information and list them in an appendix; ● clearly distinguish between conclusions, findings, and recommendations; ● support any recommendations by a specific set of findings; and ● provide recommendations that are action-oriented, practical, and specific with defined responsibility for the action. The format for the evaluation report, not to exceed 30 pages excluding references and annexes, is as follows: 1. Executive Summary—concisely state the findings and recommendations (2 pp) 2. Table of Contents (1 pp) 3. Introduction—purpose, audience, and synopsis of task (1 pp) 4. Background—brief overview of WCS/ABCG as related to the problem and purpose of the evaluation (2–3 pp) 32 5. Methodology—describe evaluation methods, including constraints and gaps (1 pp) 6. Findings/Conclusions/Recommendations—for each key question (17 pp) 7. Issues—provide a list of key technical and/or administrative, if any (1–2 pp) 8. Future Directions (2–3 pp) 9. References (including hot linked bibliographical documentation, meetings, interviews and focus group discussions) 10. Annexes—annexes that document the evaluation methods, schedules, interview lists and tables— should be succinct, pertinent, and readable The USAID asks for conciseness, clarity, and brevity in the evaluation report. The report will be submitted in English, electronically. The report will be disseminated within USAID and among implementing partners and stakeholders. An evaluation de-briefing presentation to USAID and partners: An in-person (or virtual, depending on COVID-19 restrictions) PowerPoint presentation of the evaluation process and findings intended for USAID and WCS/ABCG staff and partners. The team will present the major findings of the evaluation through a PowerPoint presentation after submission of the second draft report. The debriefing will include a discussion of achievements and issues as well as any recommendations the team has for possible modifications to project approaches, results, or activities. The evaluation team will consider USAID comments and revise and resubmit a second draft for review and comment. With permission from USAID, the evaluation team may share the second draft of the evaluation with key partners prior to the USAID/Partners presentation. Following the presentation to USAID/Partners, the evaluation team will consider USAID and Partners comments and revise and submit a final evaluation report. Final Evaluation Report: The team will submit, through the lead evaluator, a final report that incorporates the responses to comments and suggestions after USAID/AFR/SD provides written comments on the team’s draft and second draft evaluation reports as well as incorporate feedback from the USAID/Partners Meeting, as appropriate. The report will be submitted in English, electronically. The report will be disseminated within USAID and among implementing partners and stakeholders. Evaluation Reports must follow all USAID Branding and Graphic Standards (see http://www.usaid.gov/branding/USAID_Graphic_Standards_Manual.pdf). In addition, the cover of an evaluation report should provide enough information that a reader can immediately understand that it is an evaluation and what was evaluated. As described in Evaluation Report How-To Note, all evaluation report covers should: 1. Include a title block in USAID light blue background color; 2. Include the word “Evaluation” at the top of the title block and center the report title underneath that. The title should also include the word “evaluation”; 3. Include the following statement across the bottom of the cover page: “This publication was produced at the request of the United States Agency for International Development. It was prepared independently by [list authors and/or organizations involved in the preparation of the report]”; and 4. Feature one high-quality photograph representative of the project being evaluated and include a brief caption on the inside front cover explaining the photo with photographer credit. Logistical Support The Consultant will be responsible for arranging his/her own logistical support for this assignment. The consultant is expected to work closely and collaboratively with all other evaluation consultants. 33 Oversight and Management The Consultant will report to and be under the supervision of the ZemiTek Technical Supervisor, assisted by AFR/SD Senior Biodiversity Advisor Kirstin Siex. Key Documents Cooperative Agreement and amendments Annual Work Plans and Annual Reports Documents, webinars, video and PowerPoints available on ABCG.org ABCG Brochure Evaluation of the 2009-2015 ABCG award Key Informants AFR/SD/EGEA: Jessica Torrens-Spence, Kirstin Siex, Brian Hirsch E3/FAB: Cynthia Gill, Andy Tobiason, Mary Rowen, Natalie Bailey, et al WCS/ABCG management team: Michael Painter ABCG core team (representatives of members): Peter Veit (WRI), Kimberly Holbrook (TNC), Eric Coppenger (AWF), Janet Edmond (CI), Nikhil Avani (WWF), Stella Mercurio (JGI), and Michael Painter (WCS) ABCG Secretariat: Rebecca Goodman, Evelyn Namvua 34 ANNEX II: EVALUATION METHODOLOGY As specified in the SOW, we proposed in the Inception Report Evaluation Design and Workplan (July 2, 2021) “an evaluation process that can successfully complete the entire evaluation exercise.” As part of the methodology, the evaluation team developed tools, including questionnaires, for collecting data from multiple sources. This Annex presents extracts from the Inception Report methodology and updates. We present approaches to data collection and analysis, quality assurance, biases, and other potential limitations as well as dissemination and utilization of evaluation findings. Data collection tools are presented in the Key Informant Interview Summary Report (Annex III) and the E-Survey Report (Annex IV). ZemiTek understands the importance of assessing ABCG II’s accomplishments and lessons learned over the past five years and presenting this body of information in a manner that is accessible and useful to USAID and the ABCG II partners in informing the development of future investments. The ABCG 2020 Strategic Business Plan emphasized that “ABCG is moving into a new phase. We are building from past successes and developing a future for our organization that will allow us to achieve even more for the conservation of African biodiversity and remain a strong, sustainable organization.”9 To this end, and in support of the ABCG Business Plan, evaluation implementation will culminate in a Final Performance Evaluation Report for ABCG II, scheduled for submission in October 2021. A proposed outline of the final report is included in this Inception Report. A core tenet of ZemiTek’s approach is to employ evaluation that will improve programming and, ultimately, advance development effectiveness. In alignment with this approach, the evaluation team will continue to actively involve USAID, ABCG II, and other stakeholders in the evaluation process. It has begun by engaging key stakeholders to gain a full understanding of primary and secondary intended users and to identify the range of potential evaluation uses. ZemiTek will refine and implement an evaluation that targets these purposes and creates evaluation products that will be useful to the key stakeholders. The Evaluation Design included: ● A full data collection and sampling plan with data management and quality controls protocols ● A detailed evaluation design matrix (including the key questions, methods, data sources to be used to address each question, and the data analysis plan for each question); ● Draft key informant interview (KII) and online E-survey questionnaires and protocols; ● Known limitations to the evaluation design and mitigation strategies; and ● A dissemination and utilization plan. The Workplan included: ● A feasible evaluation schedule, deliverable dates, and logistical arrangements; and ● A list of the members of the team, delineated by roles and responsibilities. EVALUATION QUESTIONS This evaluation will address the following questions (EQs) that serve a mixture of accountability and learning purposes: 1. Theory of Change: How has progress been made toward achieving ABCG’s theory of change? What specific assumptions in the theory of change hold true? 2. Ideas: How has ABCG advanced understanding and dissemination of new advances and approaches in Africa? 9 ABCG (2020). Africa Biodiversity Collaborative Group Strategic Business Plan. Prepared with ABCG by Impact by Design, Inc., Tacoma Park, Maryland. 35 3. Influence: How has ABCG influenced conservation practice, approaches and linkages among its member institutions, the U.S. conservation community and in Africa (governments, academic institutions, NGOs, and the private commercial sector)? 4. Impact: How has ABCG contributed to the improved capacity, alliances, activities, and other impacts of the conservation community from both the U.S. and Africa bases? 5. Organization & Governance: How have the ABCG organizational structure, internal and external relationships, and methods of operation and collaboration facilitated or frustrated accomplishment of ABCG objectives? 6. Pandemic: How is the COVID-19 pandemic expected to impact the objectives and goals of ABCG for the future? EVALUATION DESIGN The methodology identifies three evaluation phases (1) Inception/Design; (2) Data Collection; and (3) Data Analysis, Reporting, and Dissemination. The evaluation methodology, inclusive of data collection methods, instrumentation, sampling, and analytical approaches is designed to answer these six evaluation questions. MIXED METHODS DATA COLLECTION: To overcome the biases inherent in any one data collection approach, and to leverage the strengths of various methodologies, the evaluation will collect data using five primary methods (see Data Collection section below for a more detailed description): ● A systematic Desk Review of relevant literature and project documents that maximizes use of secondary data, including project monitoring data ● Semi-structured Key Informant Interviews (KIIs) with program participants and leaders of key stakeholder groups ● Focus Group Discussions (FGDs) with a range of stakeholders will be examined as to feasibility. FGD’s will de identified and designed as the evaluation is implemented to provide the ability to probe deeper into specific issues or triangulate other collected data, etc. ● Online E-surveys of Listserv members, Activity managers, and other stakeholders ● Identification and analysis of selected Case Studies TRIANGULATION OF DATA: Data from these varied streams will be compared against one another to determine whether findings are divergent or convergent. The extent to which multiple informational streams provide consistent findings will inform the certitude and internal validity of evaluation conclusions. Should the different sources diverge, additional analysis may be undertaken, and conclusions would be stated in a way that is reflective of the nuanced data. The team will ensure that a large portion of data collected across methods will address the same questions (EQs). PARTICIPATORY NATURE: The team relied heavily on input from ABCG leadership, Steering Committee, and other stakeholders in developing the evaluation design and will rely heavily on the participation of informants and groups during data collection. The team will also use participatory unstructured discussion if possible. This is done given the levels of complexity involved in ABCG activities due to the uncertainty involved in the biodiversity problems it addresses and the environments it addresses them in. Participatory methods assist in storytelling and identifying unknowns involved in managing the complexity. FLEXIBLE APPROACH: The approach to applying these methodologies is flexible, and the team recognizes that other forms and sources of data collection may be identified as the evaluation progresses. If the use of new data sources or methods would result in a substantive change to the work plan, then ZemiTech will contact the Contracting Officer’s Representative (COR) and discuss options before coming to a final decision. GENDER-ORIENTED APPROACH: Qualitative interview guides for key informants and other respondents will include questions designed to elicit information on perceptions of women particularly with respect to 36 factors that facilitated or hindered change and any positive or negative consequences of the changes brought about by the project. To the extent possible, the evaluation will also collect, analyze, and present sex-disaggregated data, in accordance with USAID Evaluation Policy. Cross-Cutting Priority Issues ABCG and USAID have identified the following priority issues as useful areas of focus within the evaluation questions. These priority issues are relevant to several evaluation questions and will be evaluated using industry standards and best practices for each issue to assess ABCG performance and capability on that issue. For example, industry standards for value proposition development will be used to assess 1.) ABCGs processes for creating its value propositions and 2.) its ability to update them. In addition to industry best practices, questions that relate to each issue area are included in the various KII modules and E-surveys and all documents being reviewed are being analyzed (i.e. coded) for evidence related to these issues. Value Proposition: ABCG has a unique value proposition model given that it has two categories of customers,1.) a more diverse portfolio of funders (as opposed to sole reliance on USAID) and 2.) Africa based biodiversity partners (CSO, Governments, etc.) as ABCG members. Hence the evaluation will work with Activity Managers and the ABCG Steering Committee to: • Identify current documented ABCG value propositions • Outline categories of or specific Africa based potential members, implementers, etc. and any completed or envisioned customer profile research • Assess value proposition creation processes to include customer identification, profile development and use of value propositions in business planning Governance: ABCG has a stated priority of greater involvement of Africa-based biodiversity actors in its governance structure (with correlated targets outlined in the 2020 Business Plan). To help inform an ABCG strategy for achieving this, the evaluation will look at the value proposition of ABCG to the specific type of actor it aims at integrating into its governance structure and to what ends. Hence the evaluation will work with ABCG to clarify the attributes or identification of specific potential African governance partners, intended functions of these potential partners, and the value proposition of ABCG to these potential partners considering the envisioned function. Knowledge Creation and Dissemination: There are two primary manners by which ABCG disseminates knowledge: 1. Creating new knowledge through piloting, i.e., testing, innovations by the ABCG Working Groups. This new knowledge specific to testing the innovations is then disseminated. 2. Disseminating pre-existing best practices and lessons learned through various methods (e.g., workshops, webinars). For #1, the evaluation will consider how the needs for pilots are identified (per Working Group), how innovations are designed and tested through piloting and planned for scaling. This will help assess the adequacy of the processes to produce this “new knowledge” as an ABCG practice critical to its value proposition and comparative advantage. For #2, the evaluation will assess how well best practices are identified (per Working Group), i.e., their applicability to customer problem areas, and the method of dissemination in relation to recipient preferences and capacity. This will assist ABCG in determining its value proposition to different customer cohorts and methods by which they engage with cohorts to maximize the potential for uptake of disseminated knowledge. Scale-Up Pathways: The evaluation will use two types of frameworks to assess different ACBG activities that are intended to be scaled in some way, one framework for policy type activities and one for technical 37 assistance and training type activities.10 These frameworks will be used to assess ABCG’s design and management for scale up pathways across dimensions of planning, establishing pre-conditions, implementation, and MEL. By using these dimensions, the evaluation will be able to provide more specific and actionable evidence for future business planning. Phase 1: Inception Objective: Perform the due diligence required for and draft the Inception Report Timing: May 24th-July 12th. Phase 1 covered the due diligence work required to create this Inception Report. It included a series of discussions with ABCG and USAID, document review that focused on ABCG Annual Reports, Articles, Work Plans, Best Practices from a literature review on biodiversity programming, scale up pathways, Social Network Analysis, and evaluation methodologies. The phases consisted of the following tasks: 1. Desk Review and Literature Review 2. Design consultations with USAID and ABCG 3. Drafting of Inception Report 4. Presentation of Inception Report to USAID and ABCG Phase 2: Data Collection The team will collect data to answer the evaluation questions and identify evidence of progress toward intended outcomes as well as findings related to opportunities and corrective, or adaptive, actions needed to improve outcomes and programming. Interviews and other sampling will maintain confidentiality and privacy, and results will be disaggregated by sex as much as possible. The team will collect data using methods described below. Each method has strengths and weaknesses, including time and resource requirements as well as varying degrees and types of bias. To the extent possible, similar data will be collected across various methods and informants to facilitate triangulation (described in Data Analysis below). The interviews will be organized and conducted to address the evaluation questions, recognizing that interview tools and data collection processes will likely need to be refined at each successive stage. Beginning with the desk review, the team will identify emergent findings or hypotheses during regular consultations that will be explored over the course of the evaluation. Desk Review: The team has begun the evaluation by examining relevant background documents acquired through USAID, ABCG, and other sources (Inception Report Annex II). These include design documents, quarterly/annual reports, monitoring data, relevant assessments, evaluations, and workplans as well as a 2020 business plan. The team is organizing and analyzing this data as a first step toward answering the six evaluation questions. The team is developing and applying a document review guide to focus the desk review and will be updated as needed given the desk review will continue through the lifespan of the evaluation as materials emerge. Key Informant Interviews (KIIs): Much of the evaluation's data will come from KIIs with a range of stakeholders and conducted by phone or virtually (e.g., Zoom meeting). Inception Report Annex III provides a preliminary list of primary contacts, including likely respondents for KIIs. Selection of respondents will be made through a five-stage sampling process. The team will: 1. Begin with the list of key respondents provided in the SOW and expanded in Inception Report Annex III. 2. Ask ABCG and USAID to make recommendations for additional individuals and offices to interview. 10 https://msiworldwide.com/sites/default/files/additional-resources/2018-11/ScalingUp_3rdEdition.pdf 38 3. Review project documents to identify additional informants not included on the provided lists. 4. Combine the lists and purposively sample respondents with an aim of maximizing representativeness across multiple key dimensions including activity theme, gender, and length of time involved in the project. 5. Over the course of conducting interviews scheduled from this more focused list, identify additional respondents through “snowball sampling”, i.e., soliciting recommendations from informants during the interviews. KII questionnaires are presented in Inception Report Annex IV, with KIIs adapted to different groups of respondents. These will be emailed to people first about a week prior to beginning interviews to ensure that respondents understand the questions and come into the KII prepared. The KII instruments will be tested on non ABCG and USAID respondents (albeit with limited value) and then heavy emphasis will be placed on any necessary changes after the initial KII’s before they are used at scale. Because of the time constraints and volume of data collection, only one member of the evaluation team will be present at each KII but if this creates quality issues in the initial KII’s then it will be modified. Focus Group Discussions (FGDs): Due to the nature of this evaluation, during which no travel to project sites is included, the ability to have live face-to face group discussions will not be possible. However, the feasibility of limited virtual FGDs will be explored. If considered feasible, then a FGD methodology and questionnaires will be developed from the evaluators’ experience conducting FGDs. Online E-surveys: The team will administer online surveys for Listserv Subscribers and Activity Managers to gather data that will address the evaluation questions. The two E-surveys (Inception Report Annex V) will be designed, in part, to elicit quantitative Likert Scale responses, which will be relatively quick to complete and will likely result in higher response rates compared to open-ended questions. Case Studies: Case studies can produce more action-oriented evaluation results that focus on the variables that drove success as defined by changes in behavior. For example, as ABCG piloted numerous innovations, Case Studies could highlight prioritized success in those pilots and assess the factors that facilitated or mitigated it. Our Case Study strategy is less driven by targeted outcomes and instead seeks to identify those areas of success and failure in activities that must frequently adapt their approach to achieve their aims, encounter high levels of complexity/uncertainty or are piloting innovative approaches to problems that make prescribed theories of change difficult. Hence our Case Study strategy will focus on identifying key learning points in relation to the EQs for identified ABCG priority areas including: • Value propositions • Governance • Scale-up pathways In addition, while not all ABCG pilot projects are meant to achieve scale in the same way, all the pilots we have reviewed aim at some sort of expansion, replication, or collaboration (Table 1). Hence, an evaluation of how these aims are pursued, in relation to the EQs, could offer up valuable evidence on what problems, solutions, and methods of scale would be best suited for ABCG. Evaluating these aims along general USAID lifecycle phases would allow for this evidence to be correlated to specific tasks in specific phases, hence making the evidence more actionable and easier to use in ABCG policy and planning. 39 Table 1: Methods for taking pilot projects to scale Approach Methods Expansion Growth, Restructuring, Spin Off Replication Policy Adoption, Grafting, Diffusion and Spill Over, Commercialization Collaboration Formal Partnerships, Joint Ventures, Alliances, Coalitions Evaluation Data Collection Strategy Our methodology strategy, described here, focuses primarily on Phase 2, Data Collection, and drives the phasing or steps for data collection as we use multiple methods to: ● Outline performance data that is comparable across Working Groups ● Provide statistical perception data from Activity Managers and Listserv Members ● Create experiential data sets, through Case Studies, on individual examples for specific learning points on ABCG priorities ● Develop actionable evidence pertaining to value propositions, governance, and scale up pathways Main tasks and other details of these four data collection steps are described as follows and as outlined in the Workplan, many of these steps/tasks will be undertaken at the same time. Sampling Table 2 outlines the sampling strategy for each method of data collection, its data source, method of sampling and refence to the Inception Report Annex where the sampling materials (lists of respondents, etc.) are found for that specific method of collection. Some sampling approaches are TBD depending on USAID/ABCG decisions making before the finalization of the Inception Report. The composition of the listserv (including names, emails, etc.) will not be shared publicly due to the sensitivity of the information. Table 2: Sample Frame Collection Method Data Source Sampling Reference Qualitative KII’s Steering Committee Members 100% Coverage IR Annex III KII’s Working Group Leaders 100% Coverage IR Annex III KII’s Task Managers TBD TBD KIIs Case Study Respondents (TBD) TBD TBD FGD Case Study Respondents (TBD) TBD TBD Document Review ABCG Website, Internal Referrals Rolling IR Annex II (updated throughout evaluation) Quantitative E-Survey Activity Managers 100% Coverage for Receiving Survey IR Annex III E-Survey Listserv Members 100% Coverage for Receiving Survey NA Step 1: Identify case study candidate examples through multiple methods Timing: July 5th - July 13th Objective: To initially identify Case Study candidate examples through multiple methods 40 Tasks: 1. Document Review and listing case study candidates based on criteria 2. USAID/ABCG nomination of case study candidates based on criteria Task #1: Document Review. The evaluation team will perform a document review of relevant materials to include Annual Reports, Work Plans, Business Plans, Pause and Reflect Summaries, ABCG Blogs and Articles, Steering Committee Presentations/Minutes and Working Group materials. This document review was done to build understanding, inform the evaluation design and identify candidates of optimal/suboptimal performance with higher learning potential given the presence of various criteria for the example: ● Outcome oriented, meaning examples were centered on changes in behavior or actions taken by specific actors. Hence training sessions does not meet the threshold but the use of training by beneficiaries does. ● Level of specificity in terms of actions taken, identification of actors, barriers overcome, challenges identified ● Example speaks to a specific area of ABCG priority (value proposition, governance, scale up pathways) or area of uncertainty/risk ● Examples lead to answering the EQs Based on these criteria, the evaluation team will finalize a list of case study candidates to be reviewed by USAID and ABCG. Task #2: USAID/ABCG nomination of Case Study candidates. Using the above criteria, USAID and ABCG will develop a list of possible case study candidates Step 2: Conduct KIIs to collect comparable evidence across Working Groups and the Steering Committee and conduct E-Surveys for Listserv members and Activity Managers Timing: July 13th - August 13th Objectives: To use Key Informant Interviews to collect comparable evidence across Working Groups and Steering Committee on how pilots are selected, designed, planned, managed, and scaled as activities where ABCG exercises key business model components of governance, value propositions and scale up pathways. Case Study candidates will be integrated into KII questions to validate their learning potential. To collect additional evidence through E-surveys. Tasks: 1. Evaluation team conducts Working Group, Steering Committee and other KIIs 2. Evaluation team conducts E-Surveys for Listserv members and Activity Managers Task #1: Conducting KIIs KIIs will be performed via Zoom, Google Meet, WhatsApp, Signal, or other Audio/Visual conference platforms that are most conducive to the respondents. The evaluation team will reach out to each KII respondent before the KII to give them options on which platform they prefer. All KII’s will be recorded (all recordings will be destroyed by the evaluation team upon completion of the evaluation) and summary sheets developed. Summary sheets will be made available to USAID in an annex to the final report. USAID must request the recordings from the Evaluation team before they are deleted. If it is determined that FGD’s will be done virtually, then they will be performed in the same manner as KII’s. Modules are components of data collection instruments devoted to a specific theme, issue, etc. For this evaluation’s KII we have organized them by: ● ABCG/Working Group Theory of Change ● Phase of Working Group Pilot Lifecycle (selection, design, implementation, learning) 41 These modules (Table 3) were selected because they are the optimal method for collecting data that 1.) speaks to the evaluation questions and 2.) explore key priority themes of governance, value proposition, scale up. Because we are not able to collect data amongst beneficiaries at scale then we are not able to triangulate USAID and ABCG responses with these critical data points (and triangulating using written material is difficult given the authors of the written material are USAID and ABCG). Asking ABCG and USAID respondents direct questions about “the influence of ABCG” (Q3) or how ABCG “ideas” are used (Q2) would entail a significant level of bias which means that questions around “ideas” and “influence” must be answered in a way that mitigates this bias without access to beneficiaries. Table 3: KII format KII Module Function of Module Possible General Questions, Probes, and Prompts (always asking for references to document & support respondent claims)11 Theory of Change Outline the respondents understanding of the ABCG or Working Group Theory of Change, Key Assumptions and Risks What do you consider to be ABCG’s primary two or three objectives? What are the critical assumptions pertaining to these objectives? What are the biggest risks in achieving these objectives? Do you feel there have been changes to these objectives, assumptions, and risks? What was the reason for the change? Activity Selection Identify the key problem areas the activity is meant to help resolve Outline who problem it is, why it is, where it is and when it is. Ensure the problem was identified in an inclusive manner What is the envisioned technical, process, or organizational innovation to address the problem? What are some of the problem statements for the activity? Why are these problems deemed most critical and warranting of an activity? How was this decided? What idea or influence is ABCG hoping to advance? What type of scale is envisioned for this activity? 11 This table borrows heavily (specifically the probing questions) from MSI’s Scaling Up: From Vision to Large Scale Change found at https://msiworldwide.com/sites/default/files/additional-resources/2018- 11/ScalingUp_3rdEdition.pdf 42 KII Module Function of Module Possible General Questions, Probes, and Prompts (always asking for references to document & support respondent claims)11 Activity Planning Clarify the activity model, specific innovations, or project to be scaled up Identify the medium/long-term intent of the activity (expansion, replication, collaboration) Identify the planned method or pathway of achieving the intent, i.e., how the activity will be scaled Determine the organizational roles involved in the activity and expected roles in scale up Outline the intended scope of the scaling up effort and the where and amongst whom the scaling will occur What organizational process, and technical factors are/were critical to success in the activity? Is it necessary to replicate all elements of the model on a large scale? Does the organization that carried out the activity project have the desire and organizational capacity to expand its operations and deliver services on a substantially larger scale? If not, which organization(s) are best suited and motivated to implement the model on a scaled-up basis or to serve as partners in implementing the model? Should the effort include policy change by the government or rely exclusively on voluntary adoption by private and non-governmental organizations? Is there a need for one or more intermediary organizations to support the scaling up process? If so, what help is needed, and which organizations are best suited to perform these roles? Along what dimension(s) should scaling up take place? What would scaling up look like if it were successful in terms organizational performance? Activity Implementation Outline how a constituency was built to legitimize the change envisioned by the activity Understand how this constituency was used to implement and adapt the activity What was done to persuade relevant decision makers, funders, and opinion leaders that new solutions are necessary and desirable? What was done to persuade relevant decision makers, funders, and opinion leaders that the proposed model is successful, cost-effective, and feasible? Which organizations, organizational units, or individuals were responsible for key decisions regarding the funding and implementation of scaling up? Who has the authority to make decisions within these organizations? What arguments, appeals, or advocacy strategies were used to be persuasive to decision makers? What were the most effective networks and alliances for carrying out this advocacy, and how can they be most efficiently mobilized and organized? How was buy-in from the leadership and staff of potential implementing organizations best achieved? Activity Learning Identify the key learning points that were most important to success and where the data came from to inform the learning What changes were made during implementation? What prompted these changes? What information was used to identify the need for the change? How was this information used? To this end, the evaluation asks questions about how ABCG identifies, designs, implements, and learns from its activities (primarily pilots and best practice dissemination activities). Interviewers will give probing questions and prompts that solicit answers specific to the role of ABCG “ideas” (Q2), “influence” (Q3) and we will also code responses using these constructs during analysis. Likewise, structuring KII’s in this 43 manner helps to isolate evidence on priority themes as a literature review on scale up pathways, value propositions and governance have yielded important evidence that was used to inform the prompts and probing questions for each module. The interviewer administering the KII will be assigned specific Working Groups they will cover so that, in addition to the evidence from the literature review, the interviewer can tailor the questions and probes to the specific priorities and activities of that Working Group. Hence, the evaluation will be able to compare responses across modules while still yielding evidence specific to that Working Group. At the same time no specific module is devoted to a specific evaluation question, for example a single evaluation question can be informed by every module depending on how the respondent responds. Task 2: E-Surveys. The evaluation team will perform two E-Surveys, one for the composition of the ABCG Listserv and one for ABCG Activity Managers. This task will include: • identifying survey champions, developing survey message, and delivering message to respondents for each E-Survey three days before survey is administered • administering surveys to the entire ABCG Listserv and Activity Manager roster and sending reminder email Both E-Surveys will be performed using Survey Monkey12 with responses being anonymized, but all aggregations made readily available to USAID upon request. The 2015 ABCG evaluation performed two similar E-Surveys (with the same data sources for sampling) with very low response rates that introduce risk to the use of those survey results. To mitigate this risk, and in accordance with best practice, the evaluation team will work with ABCG and USAID to identify one or two “Survey Champions”, individuals who each respondent group recognizes as an authority, a respected and/or trusted leader to communicate a voice of support for the survey and its value. Once identified the Evaluation Team will work with each respective champion to draft an email that will be sent to each respondent list to convey this message to increase the response and completion rate over the 2015 evaluation. In addition, we have attempted to shorten each survey and introduce more closed ended questions to improve response and completion rates. Step 3: Finalize case study examples using data from document reviews, KII’s, and other consultations Timing: August 13th - August 17th Objective: To finalize case study examples using data from document reviews, KII’s, and other consultations Tasks: 1. Evaluation Team presents list of Case Study candidates (based on document review and KIIs) to ABCG and USAID for review and collaboration on USAID/ABCG list of candidates 2. Evaluation Team proposes “final” list of Case Study candidates to Steering Committee for approval (no more than 2) Task #1: Collaboration on Case Study candidates. Using gathered evidence, the Evaluation Team will present a compiled list of candidates to USAID and ABCG for discussion and agreement on a final two candidates for presentation to the Steering Committee Task #2: Presentation and Agreement. The evaluation team will either submit via email or present via video the Case Study candidates to the Steering Committee for final agreement 12 Survey Monkey: https://www.surveymonkey.com/ 44 Step 4: Perform remaining data collection and analysis on no more than two case studies. Timing: August 17th - September 8th (continuing into Phase 3 of the evaluation) Note: In Phase 3: KII & E￾Survey Summary Reports due 20 August / Draft Evaluation Report due 3 September Objective: To perform remaining data collection on case studies (no more than 2 case studies) Tasks: 1. Identify necessary data collection (e.g., additional document review) for summary case studies 2. Perform necessary data collection, produce summary sheets Until the final Case Studies are identified it is not possible to present a design of how they will be conducted. Case Studies will be identified during Phase II through the KIIs and Document Review primarily. Case Study candidates will be identified by the evaluation team by applying the case study selection criteria to candidates that merge through Phase II data collection. By the end of Phase II, the evaluation team will propose case study candidates to USAID along with proposed case study designs. Given the timing of these nomination the case studies will not be extensive but will instead focus on specific issues within a case study that can be completed in a short timeline during Phase III. Upon approval of the Case Study selections the designs will be finalized and approved by USAID before they are implemented. Phase 3: Analysis, Reporting, and Dissemination The team will use a variety of data analysis techniques to support the development of evaluation findings, conclusions, and recommendations. Both at the mid-way point and upon completion of data collection, the Lead Evaluator will organize internal working sessions to discuss emerging findings. The TL will capture these preliminary findings and conclusions in a matrix that categorizes findings, conclusions, and recommendations by evaluation question. This will ensure that the team (a) develops a systematic and thorough response to each evaluation question, (b) assures that preliminary analysis accounts for gender and social dimensions, (c) identifies any gaps where additional clarification or analysis may be necessary, and (d) develops a rigorous basis for developing the Draft Evaluation Report. An evaluation design matrix (Table 3) provides preliminary data sources as well as data collection and analysis methods for each of the six EQs. Qualitative Analysis The team will use qualitative analytical techniques to code, collate, and interpret data captured through notes collected from interviews, discussion groups, and desk reviews. Based on preliminary findings, the team will develop codes to be applied to further notes. One more check-in at the mid-point could further clarify codes. The final codebook will be developed upon completion of data collection. The team will develop a findings matrix summarizing the results of early data analysis and update bi-weekly throughout the data collection. Team members will prepare electronic summaries of each KII. We will identify response themes and key points, meeting frequently to resolve questions about interpretation, meaning, context, and analysis. Team members will code data and use relational content analysis to identify response categories and patterns as well as elucidate emergent themes, contextual factors, and trends. We will disaggregate the interview and discussion group data by sex and length of involvement in the project to capture differential outcomes among groups. The team will employ a data analysis strategy in which qualitative data is first analyzed independently, then in parallel. Findings from each data set will inform and substantiate findings from other data types using triangulation. The team will also use different forms of frequency, trend, theme, and pattern analysis to compare results across respondent groups. This will allow findings that identify who is benefiting from 45 what types of activities, which elements of an activity are contributing to the achievement of outputs and outcomes, which elements are not, and most importantly, why? Our Case Study strategy requires specific analytical methods integrated into each step/phase. Part of the purpose behind using case studies was to identify well evidenced examples, hence each phase has iterative data collection from different sources (document reviews, KIIs, ABCG nominations, etc.) devoted to identifying candidates for the additional data collection. The candidates will be assessed according to their potential to contribute to ABCG priorities, specifically value propositions, governance and scale up pathways and answering the EQs. The Evaluation Team will propose candidates based on their assessment of the quality of evidence supporting the candidate and the relevance of the data to the stated priority areas. Table 4 outlines how each evaluation question will be addressed in terms of data collection and analysis. Given that each method of data collection will be used to answer each evaluation question there is little variance in the table. Table 4: Evaluation Design Matrix EVALUATION QUESTION DATA SOURCE DATA COLLECTION METHOD DATA ANALYSIS METHOD EQ 1. Theory of Change: How has progress been made toward achieving ABCG’s theory of change? What specific assumptions in the theory of change hold true? Program documents, key informants from ABCG (Steering Committee and Working Groups) and USAID Desk review, KIIs Case study approach combining data from KIIs Comparison of program indicators and targets against achievements Qualitative data analysis emphasizing contextual analysis of program adaptation such as content analysis, story mapping and assumptions testing EQ 2. Ideas: How has ABCG advanced understanding and dissemination of new advances and approaches in Africa? Program documents, key informants from ABCG and USAID Listserv Subscribers, ABCG Activity managers, Online E-surveys of implementers and beneficiaries. Desk review, KIIs Case study approach combining data from KIIs and E-surveys Qualitative content analysis, comparison, and pattern analysis. Disaggregated by gender Quantitative: Likert scale responses in E-surveys Re-analysis (coding by thematic areas of “ideas’, “influence”, “scale up pathway”, “governance”) of key findings and conclusions from the perspective of identifying lessons learned and recommendations EQ 3. Influence: How has ABCG influenced conservation practice, approaches and linkages among its member institutions, the U.S. conservation community and in Africa (governments, academic institutions, NGOs, and the private commercial sector)? Program documents, key informants from ABCG and USAID Listserv Subscribers, ABCG Activity managers, Online E-surveys of implementers and beneficiaries. Desk review, KIIs Case study approach combining data from KIIs and E-surveys Qualitative content analysis, comparison, and pattern analysis. Disaggregated by gender Re-analysis (coding by thematic areas of “ideas’, “influence”, “scale up pathway”, “governance”) of key findings and conclusions from the perspective of identifying lessons learned and recommendations Quantitative: Likert scale responses in E-surveys EQ 4. Impact: How has ABCG contributed to the improved capacity, alliances, activities, and other impacts of the Program documents, key informants from ABCG and USAID Online E-surveys of implementers and beneficiaries. Desk review, KIIs Qualitative content analysis, comparison, and pattern analysis. Disaggregated by gender Quantitative: Likert scale responses in E-surveys 46 EVALUATION QUESTION DATA SOURCE DATA COLLECTION METHOD DATA ANALYSIS METHOD conservation community from both the U.S. and Africa bases? Listserv Subscribers, ABCG Activity managers, Case study approach combining data from KIIs and E-surveys Re-analysis of key findings and conclusions from the perspective of identifying lessons learned and recommendations EQ 5. Organization & Governance: How have the ABCG organizational structure, internal and external relationships, and methods of operation and collaboration facilitated or frustrated accomplishment of ABCG objectives? Program documents, key informants from ABCG and USAID Listserv Subscribers, ABCG Activity managers, Online E-surveys of implementers and beneficiaries. Desk review, KIIs Case study approach combining data from KIIs and E-surveys Qualitative content analysis, comparison, and pattern analysis. Disaggregated by gender Quantitative: Likert scale responses in E-surveys Re-analysis of key findings and conclusions from the perspective of identifying lessons learned and recommendations EQ 6. Pandemic: How is the COVID-19 pandemic expected to impact the objectives and goals of ABCG for the future? Program documents, external materials on COVID experience of similar programs, key informants from ABCG and USAID. ABCG Activity managers. Online E-surveys of implementers and beneficiaries. Desk review, KIIs Re-analysis of key findings and conclusions from the perspective of identifying lessons learned and recommendations Quantitative: Likert scale responses in E-surveys Quantitative Analysis Data that are intrinsically quantitative (e.g., responses to Likert Scale questions in E-surveys) will be statistically analyzed using standard descriptive methods to the extent that it will assist in the analysis, interpretation, or reporting of evaluation findings, the team will produce graphs and/or tables. Given the high number of respondents in both E-surveys, Social Network Analysis (SNA) is an option for demonstrating the various relationships critical to evaluation questions. However, there are significant barriers to SNA providing value in this evaluation due to: • The risk of low E-survey response rates (meaning an actionable representation of critical relationships is much less likely) • The homogeneity of Listserv members with a lack of Africa based membership (while the Listserv is more diverse than in 2015 the current level homogeneity may introduce bias) Hence the use of SNA will need to be re-visited when these two risks have been fully assessed upon completion of the E-Surveys. Steps for Analysis, Reporting, Dissemination, and Utilization Although some data collection pertaining to case studies will overlap into Phase 3 Analysis, this final phase will involve data analysis and reporting. Given that much of the data collection will be complete before Phase 2, some analysis on completed data collection could be performed towards the end of Phase 1. Objectives: • To perform mixed method analysis per the data collection protocols • To prepare, submit, and present the Draft Final Report to USAID, Second Draft to USAID and Steering Committee, make PowerPoint presentation to USAID and partners and receive feedback, and then deliver the last version of Final Report. 47 Timing: August 9th-October 11th Tasks: 1. Perform analysis per the Inception Report with all relevant materials being made available to USAID upon request 2. Draft and present drafts of the Final Report to USAID and Steering Committee 3. Solicit and receive verbal and formal written feedback from USAID and Steering Committee 4. Produce an Annex to Final Report that outlines Evaluation Team’s response to feedback including modifications (as a table) 5. Submit Final Report The Evaluation Team can present the Draft Final Report in a manner that is most conducive to the results being understood and utilized. Hence it may be that issues emerge during the evaluation that are of priority to ABCG and require a particular focus or the presentation could be a workshop that facilitates next steps, etc. The Evaluation Team will work with ABCG and USAID to decide which presentation format is most conducive. The Evaluation Team will request formal written feedback from USAID and the Steering Committee on the prescribed timetable (per the SOW) and will create an Annex for the Final Report documenting this feedback and the evaluation’s response. Data Management All evaluation data collection instruments, desk review materials, analysis tools, etc., will be handled only by the Evaluation Team from design to collection, analysis, dissemination, and deletion. No Personal Identification information will be requested from respondents, nor will any be documented. All materials will be stored on ZemiTek’s secure Google Drive with access rights given only to the Evaluation Team and USAID upon request. All qualitative collection summary sheets, qualitative code books (for KII’s, FGDs and document review) and metadata from E-Surveys will be made available to USAID upon request during the evaluation and at completion of the evaluation. While KII summary sheets will be developed there will be no attributable information collected on data sources. All summary sheets and data management will ensure that no identifying information of data sources is included. All meta data and summary sheets can be provided upon request that have been scrubbed of identifying information of the respondent. Limitations and Risk Management Below is a list of some noteworthy potential limitations and risks the team has identified for this evaluation as well as mitigation strategies. RECALL BIAS: Usually recall bias stems from respondents being too far from activities, in terms of time, to have correct recall on the questions being asked. In the case of ABCG, there is a risk of recall bias stemming from Q1 on ABCG’s theory of change. There was no theory of change that drove the initial USAID investment, the evaluation found that there was no documented theory of change until approximately 2015 but that there was no agreement on or common knowledge of a theory of change. After 2015 ABCG at large did not develop a Theory of Change until the 2019-2020 Work Plan (when all the individual Working Groups also developed their own Theories of Change). As the evaluation addresses Q1, it will be important to ensure that collected evidence (specifically KII’s) speak to documented theories of change and not undocumented theories of change. This is to ensure accountability to documented objectives and mitigate ABCG being evaluated against undocumented objectives. Without a documented theory of change, when respondents are asked about objectives, there is the opportunity to introduce bias based on the respondent’s views of contemporary activities. 48 To mitigate this bias the evaluation will seek to map the process by which the original and subsequent theories of change were developed, why, how and by whom. This will yield important learning points on how ABCG adapted its approach but will also ensure that correct objectives are mapped out across time to assess ABCG performance against. RESPONSE BIAS: The team will minimize bias by including language in the informed consent that there are no inherent benefits to them from participating in the evaluation and that there will be no retaliation or direct consequences against them for their responses. For the E-surveys conducted during the 2015 evaluation, a non-response bias was clear in both the ABCG Listserv Subscribers and Activity Managers survey data, especially the former. It was noted that while the Listserv had approximately 1,400 contacted subscribers, only 72 completed the questionnaire. The 2015 Listserv questionnaire was longer making it less likely to get a response. To improve response the team is: ● Shortening the length of each survey by eliminating some questions and transitioning some open￾ended questions to multiple choice, etc. ● Identifying survey champions who are trusted, respected individuals that are recognizable to each survey group to send a message to each group communicating the importance of the survey, how short it is, etc. SELECTION BIAS: The overwhelming limitation on this evaluation is the lack of scaled access to those who were intended beneficiaries of ABCG activities, hence this evaluation cannot confirm any impact beyond the isolated cases in which it is able to collect data from intended beneficiaries. Because there is no widespread access to beneficiaries, there is a significant risk of bias of result due to data collection being focused on USAID and ABCG actors. The team will minimize this bias by identifying a range of informants for case studies that includes beneficiaries and others with differing degrees and types of involvement with the activity. In addition, we are using triangulation to identify case studies and substantiate claims by respondents, in so doing we mitigate possible bias from a single source of information. Other forms of selection bias occur when only certain cohorts of stakeholders are available. The 2015 Evaluation had a very heavy bias towards American over African respondents; hence, the composition of the Listserv will be assessed to caveat any reported results based on bias introduced by the demographics of the respondents. Per USAID’s Evaluation Policy, “Evaluations of USAID projects will be undertaken so that they are not subject to the perception or reality of biased measurement or reporting due to conflict of interest or other factors”. Hence if the evaluation is not able to triangulate ABCG or USAID perception data with outcome data from beneficiaries that outlines the uptake of new practices, changes in behavior, actions taken, new capacity, etc., then the perception data can only be used to document outputs. LACK OF COLLABORATION: The ABCG Steering Committee and Leadership is in a state of transition as a new Steering Committee enters in the next quarter. This has contributed to difficulties in soliciting the collaboration required to complete a quality Inception Report and it poses a significant risk moving forward specifically in terms of scheduling data collection but more broadly in securing the collaboration needed for the formative components of the evaluation that are meant to be of value to ABCG moving forward. To mitigate this risk, we have removed most of the formative (i.e. “learning”) components of the evaluation that would require more ABCG engagement and will focus on the accountability portions that strictly assess ABCG performance. RISK MANAGEMENT: The evaluation is maintaining a Risk Register that outlines the primary risks to the quality of the evaluation that will be updated throughout the lifecycle of the evaluation. This Risk Register will be made available to USAID upon request. 49 WORKPLAN Schedule of Evaluation Activities We propose an evaluation process that can successfully complete the entire evaluation exercise. The work plan (Table 5) identifies three evaluation phases: (1) Start-up and Inception Planning; (2) Data Collection; and (3) Data Analysis, Reporting, and Dissemination. The work plan also includes our proposed evaluation team’s level of effort (LOE) represented by work-day allocation against specific tasks. 50 Table 5: Evaluation Work Plan Deliverables and Quality Assurance Deliverables As identified in the SOW, the deliverables or outputs for this evaluation and their submission or completion dates are found in Table 6. Table 6: Deliverables Schedule (Source: SOW) Illustrative Deliverables / Outputs Approximate Due Date Desk Review completed 6/11/21 Draft inception Report with data collection tools 7/02/21 Final Inception Report with data collection tools 7/16/21 Key Informant Interview & E-Survey Summary Reports 8/20/21 First Draft Evaluation Report submitted to USAID 9/06/21 Comments received from USAID 9/10//21 Second Draft Evaluation Report submitted to USAID and shared with Partners 9/22//21 Evaluation De-briefing Presentation to USAID and Partners 9/28//21 Final Evaluation Report submitted to USAID 10/15/21 First Draft Evaluation Report. The Lead Evaluator will deliver the report to USAID through the ZemiTek Technical Supervisor. As specified in the SOW, the contents of the Evaluation Report, not to exceed 30 pages excluding references and annexes, will be as shown in the Box. The SOW further specifies that “the report will be a comprehensive report of all deliverables associated with this contract, and will: • represent a thoughtful, well-presented, well-researched, and well-organized effort to objectively evaluate what worked in this project, what did not work, and why • be a high-quality technical report, in a professional writing style • address all questions included in this scope of work • include all the key sections: cover sheet, table of contents, acronym list/glossary of terms, executive summary, background, objectives, questions, methods, findings, conclusions, and lessons learned, and recommendations • include the scope of work as an appendix • include an introduction that adequately describes the project, explains where it is implemented, includes contextual information, and includes the “theory of change” or development hypotheses that underlie the project • describe the evaluation methodology in detail and all tools used such as questionnaires, checklists, and discussion guides. All tools including the Key Informant Interview Summary Report, E-Survey Report, Tabulation Report, and any others developed will be included in an appendix in the final report • describe findings with gender disaggregate outcomes and impacts 2 • describe any limitations to the methods (e.g., selection bias, recall bias, unobservable differences between groups, etc.) • present the findings as analyzed facts, evidence, and data and should not be based on anecdotes, hearsay, or a compilation of opinions. Findings should be specific, concise, and supported by strong quantitative or qualitative evidence • properly identify sources of information and list them in an appendix • clearly distinguish between conclusions, findings, and recommendations • support any recommendations by a specific set of findings • provide recommendations that are action-oriented, practical, and specific with defined responsibility for the action” Second Draft Evaluation Report. Comments and edits from the first draft will be incorporated into the draft report, which will be disseminated within USAID and among implementing partners and stakeholders. Evaluation De-Briefing Presentation to USAID and Partners: The evaluation team will prepare and conduct an in-person (or virtual, depending on Covid-19 restrictions) PowerPoint presentation of the evaluation process and findings intended for USAID and WCS/ABCG staff and partners. The debriefing will include a discussion of the contents of the Evaluation Report, including achievements, issues, and recommendations for project approaches, results, or activities. Final Evaluation Report. Following the de-briefing presentation to USAID and Partners, the evaluation team will take all comments, make revisions as needed, and submit the final evaluation report. The final report will then be disseminated within USAID and among implementing partners and stakeholders. As specified in the SOW, the Evaluation Report will comply with all USAID Branding and Graphic Standards (http://www.usaid.gov/branding/USAID_Graphic_Standards_Manual.pdf). Quality Assurance ZemiTek will use an innovative approach to achieve a high quality, influential evaluation that intends to enhance ABCG development effectiveness. This approach to quality assurance (QA) has proven effective for driving learning, accountability, and performance improvement. At key stages of the evaluation, the ZemiTek evaluation team will apply its approach primarily through using five QA checklists to assess and ensure reporting quality and compliance with USAID requirements at key points during the evaluation, i.e., for the Inception Report, preliminary findings and final presentations, draft Evaluation Report, and final Evaluation Report. Team Composition The evaluation team is composed of consultant evaluators and supervisors from ZemiTek and Camris (Table 7). Table 7: Evaluation Team Roles and Responsibilities Name and Role Responsibilities Evaluators Mr. Michael Cooper Team Leader & Lead Evaluator ● Serve as an evaluator of the final evaluation contributing to the evaluation throughout the entire evaluation process. ● Work with USAID and ABCG management to plan the evaluation and logistics. ● Work with the evaluation team lead to allocate evaluation responsibilities based on skills and capacity ● Oversee and support the evaluation team member to ensure that the team fulfills obligations, communications, and deliverables in a timely manner 3 Name and Role Responsibilities ● Manage the evaluation team’s activities and organize and facilitate team interaction. ● Serve as liaison between USAID and the evaluation team ● Lead interpretation of findings and both oral and written presentations of findings. Dr. Arthur Mitchell Evaluator ● Serve as an evaluator of the final evaluation contributing to the evaluation throughout the entire evaluation process ● Work with USAID and ABCG management to plan the evaluation and logistics ● Work with the evaluation team lead to allocate evaluation responsibilities based on skills and capacity ● Collaborate and support the evaluation team lead to ensure that the team fulfills obligations, communications, and deliverables in a timely manner ● Contribute to the evaluation team’s activities and help to organize and facilitate team interaction ● Participate in the interpretation of findings and both oral and written presentations of findings Supervision and Backstopping Mr. Ian MacNairn Technical Supervisor and ZemiTek Chief of Party, USAID/AFR Institutional and Technical Support Services, USAID/AFR/SD ● Technical supervision of the evaluation process and team ● Participation in coordination meetings Mr. Malik Jaffer Administrative Supervisor, Camris ● Administrative supervision and assistance to the evaluation process and team ● Participation in coordination meetings ANNEX III: KEY INFORMANT INTERVIEW SUMMARY REPORT Much of the evaluation's data came from Key Informant Interviews (KIIs) with a range of stakeholders and conducted by phone or virtually (e.g., Zoom meeting). The objective is to use KIIs to collect comparable evidence across Working Groups and the Steering Committee on how pilots are selected, designed, planned, managed, and scaled as activities where ABCG exercises key business model components of governance, value propositions and scale up pathways. All six Evaluation Questions were addressed during the interviews. Selection of respondents was made through a five-stage sampling process. The team: 1. Began with the list of key respondents provided in the SOW and expanded in Inception Report Annex III 2. Asked ABCG and USAID to make recommendations for additional individuals and offices to interview 3. Reviewed project documents to identify additional informants not included on the provided lists 4. Combined the lists and sampled respondents with the aim of maximizing representativeness across multiple key dimensions including activity theme, gender, and length of time involved in the project 5. Over the course of conducting interviews scheduled from this more focused list, identified additional respondents through “snowball sampling”, i.e., soliciting recommendations from informants during the interviews KIIs were conducted between 30 September and 19 September. One Focus Group Discussion (FGD) was held with 6 participants. Respondents were individuals from (1) USAID AFR/SD/EGEA, (2) USAID E3/FAB, (3) WCS/ABCG management team, (4) ABCG core team – Steering Committee (representatives of member organizations), (5) ABCG Secretariat, and (6) Thematic Activity Task Working Group Leads. Table 1 shows all KII respondents and FGD participants with their positions and contacts. Table 2 provides dates when KIIs and FGDs were conducted but not indicated specific respondents. Table 1. Key Informants Interviewed No Key Informant & Position Contacts USAID AFR/SD/EGEA 1 Kirstin Siex, Senior Biodiversity Advisor, USAID Bureau for Africa (AFR/SD/EGEA) 2 Jessica Torrens-Spence, Deputy Division Chief, Biodiversity and Natural Resources Advisor, USAID Bureau for Africa (AFR/SD/EGEA) USAID E3/FAB 4 Andrew Tobiason, Biodiversity Conservation Advisor, USAID 5 Mary Rowen, Senior Biodiversity Policy Advisor, USAID 6 Natalie Bailey, Biodiversity & Natural Resources Specialist, USAID, and former ABCG Coordinator (Nov 2009 – July 2014) WCS/ABCG management team 7 Michael Painter, WCS mpainter@wcs.org ABCG core team – Steering Committee (representatives of member organizations) 2 No Key Informant & Position Contacts 8 Peter Veit, WRI World Resources Institute (WRI) Peter.Veit@wri.org 9 Kimberly Holbrook, TNC The Nature Conservancy (TNC) kholbrook@tnc.org 10 Eric Coppenger, AWF African Wildlife Foundation (AWF), 1100 New Jersey Ave SE, Suite 900, Washington, DC 20003. Office: +1 202-939-3352 (?) ecoppenger@awf.org 11 Janet Edmond, CI Conservation International (CI) jedmond@conservation.org 12 Nikhil Advani, WWF World Wildlife Fund (WWF-US) Nikhil.Advani@wwfus.org 13 Stella Mercurio, JGI The Jane Goodall Institute (JGI) smercurio@janegoodall.org 14 Stephanie Keller, JGI The Jane Goodall Institute (JGI) sKeller@janegoodall.org 15 Michael Painter, WCS (See no. 9) ABCG Secretariat 16 Rebecca Goodman, Director, Africa Biodiversity Collaborative Group ABCG, c/o African Wildlife Foundation, 1100 New Jersey Ave SE, Suite 900, Washington, DC 20003. Office: +1 202-939-3352; Skype: rebecca.abcg; rgoodman@abcg.org 17 Evelyn Namvua, Knowledge & Communication Advisor, ABCG ABCG, Nairobi, Kenya enamvua@abcg.org Thematic Activity Task Working Group Leads 18 Hedley Grantham, Land Use Management (LUM) Wildlife Conservation Society hgrantham@wcs.org 19 Marc Fourrier, Community-Based Forest Management (CBFM) The Jane Goodall Institute mfourrier@janegoodall.org 20 Chris Zganjar, Global Change Impacts (GCI) The Nature Conservancy czganjar@TNC.ORG 21 Janet Edmond, Global Health Linkages to Biodiversity Conservation: Population Health and Environment (PHE) Conservation International jedmond@conservation.org 22 Colleen Sorto, Global Health Linkages to Biodiversity Conservation: Freshwater Conservation and Water, Sanitation & Hygiene (FW-WASH) Conservation International csorto@conservation.org 23 Janet Edmond, Integrating Gender and Vulnerable Populations in Activity Design and Implementation Conservation International jedmond@conservation.org 24 David Williams, AWF African Wildlife Foundation (AWF) dwilliams@awf.org 25 Kendall Jones, WCS Wildlife Conservation Society (WCS) kendallrawardjones@gmail.com 3 Table 2: KII and FGD Dates in 2021 and Sources of Qualitative Data Collection Date Method/Source Date Method/Source 6/30 KII/SC Member 8/9 FGD/USAID 7/9 KII/SC Member 8/9 KII/Activity Manager 7/14 KII/SC Member (2) 8/9 KII/SC Member 7/20 KII/SC Member (3) 8/10 KII/ABCG Partner 7/21 KII/SC Member (3) 8/11 KII/Activity Manager 7/22 KII/SC Member 8/11 KII/Field Officer 7/28 KII/SC Member 8/16 KII/USAID 7/29 KII/SC Member (2) 8/17 KII/External Expert 8/4 KII/SC Member KIIs were performed via Zoom, Google Meet, or other virtual platforms that were most conducive to the respondents. The evaluation team reached out to each KII respondent before the KII to give them options on which platform they prefer. All KII’s were recorded (all recordings were destroyed by the evaluation team upon completion of the evaluation) and summary sheets developed for each KII. Summary sheets were made available to USAID in the evaluation designated Google drive. FGD’s were performed in the same manner as KII’s. Modules are components of data collection instruments devoted to a specific theme, issue, etc. For this evaluation’s KII we have organized them by: • ABCG/Working Group Theory of Change • Phase of Working Group Pilot Lifecycle (selection, design, implementation, learning) These modules (Table 3) were selected because they are the optimal method for collecting data that 1.) speaks to the evaluation questions and 2.) explore key priority themes of governance, value proposition, scale up. Because we are not able to collect data amongst beneficiaries at scale then we are not able to triangulate USAID and ABCG responses with these critical data points (and triangulating using written material is difficult given the authors of the written material are USAID and ABCG). Asking ABCG and USAID respondents direct questions about “the influence of ABCG” (Q3) or how ABCG “ideas” are used (Q2) would entail a significant level of bias which means that questions around “ideas” and “influence” must be answered in a way that mitigates this bias without access to beneficiaries. Table 3: KII format and modules 4 KII Module Function of Module Possible General Questions, Probes, and Prompts (always asking for references to document & support respondent claims)13 Theory of Change Outline the respondents understanding of the ABCG or Working Group Theory of Change, Key Assumptions and Risks What do you consider to be ABCG’s primary two or three objectives? What are the critical assumptions pertaining to these objectives? What are the biggest risks in achieving these objectives? Do you feel there have been changes to these objectives, assumptions, and risks? What was the reason for the change? Activity Selection Identify the key problem areas the activity is meant to help resolve Outline who problem it is, why it is, where it is and when it is. Ensure the problem was identified in an inclusive manner What is the envisioned technical, process, or organizational innovation to address the problem? What are some of the problem statements for the activity? Why are these problems deemed most critical and warranting of an activity? How was this decided? What idea or influence is ABCG hoping to advance? What type of scale is envisioned for this activity? Activity Planning Clarify the activity model, specific innovations, or project to be scaled up Identify the medium/long-term intent of the activity (expansion, replication, collaboration) Identify the planned method or pathway of achieving the intent, i.e., how the activity will be scaled Determine the organizational roles involved in the activity and expected roles in scale up Outline the intended scope of the scaling up effort and the where and amongst whom the scaling will occur What organizational process, and technical factors are/were critical to success in the activity? Is it necessary to replicate all elements of the model on a large scale? Does the organization that carried out the activity project have the desire and organizational capacity to expand its operations and deliver services on a substantially larger scale? If not, which organization(s) are best suited and motivated to implement the model on a scaled-up basis or to serve as partners in implementing the model? Should the effort include policy change by the government or rely exclusively on voluntary adoption by private and non-governmental organizations? Is there a need for one or more intermediary organizations to support the scaling up process? If so, what help is needed, and which organizations are best suited to perform these roles? Along what dimension(s) should scaling up take place? What would scaling up look like if it were successful in terms organizational performance? Activity Implementation Outline how a constituency was built to legitimize the change envisioned by the activity What was done to persuade relevant decision makers, funders, and opinion leaders that new solutions are necessary and desirable? 13 This table borrows heavily (specifically the probing questions) from MSI’s Scaling Up: From Vision to Large Scale Change found at https://msiworldwide.com/sites/default/files/additional-resources/2018- 11/ScalingUp_3rdEdition.pdf 5 KII Module Function of Module Possible General Questions, Probes, and Prompts (always asking for references to document & support respondent claims)13 Understand how this constituency was used to implement and adapt the activity What was done to persuade relevant decision makers, funders, and opinion leaders that the proposed model is successful, cost-effective, and feasible? Which organizations, organizational units, or individuals were responsible for key decisions regarding the funding and implementation of scaling up? Who has the authority to make decisions within these organizations? What arguments, appeals, or advocacy strategies were used to be persuasive to decision makers? What were the most effective networks and alliances for carrying out this advocacy, and how can they be most efficiently mobilized and organized? How was buy-in from the leadership and staff of potential implementing organizations best achieved? Activity Learning Identify the key learning points that were most important to success and where the data came from to inform the learning What changes were made during implementation? What prompted these changes? What information was used to identify the need for the change? How was this information used? KII Module Function of Module Possible General Questions, Probes, and Prompts (always asking for references to document & support respondent claims)14 Theory of Change Outline the respondents understanding of the ABCG or Working Group Theory of Change, Key Assumptions and Risks What do you consider to be ABCG’s primary two or three objectives? What are the critical assumptions pertaining to these objectives? What are the biggest risks in achieving these objectives? Do you feel there have been changes to these objectives, assumptions, and risks? What was the reason for the change? Activity Selection Identify the key problem areas the activity is meant to help resolve Outline who problem it is, why it is, where it is and when it is. Ensure the problem was identified in an inclusive manner What is the envisioned technical, process, or organizational innovation to address the problem? What are some of the problem statements for the activity? Why are these problems deemed most critical and warranting of an activity? How was this decided? What idea or influence is ABCG hoping to advance? What type of scale is envisioned for this activity? 14 This table borrows heavily (specifically the probing questions) from MSI’s Scaling Up: From Vision to Large Scale Change found at https://msiworldwide.com/sites/default/files/additional-resources/2018- 11/ScalingUp_3rdEdition.pdf 6 KII Module Function of Module Possible General Questions, Probes, and Prompts (always asking for references to document & support respondent claims)13 Activity Planning Clarify the activity model, specific innovations, or project to be scaled up Identify the medium/long-term intent of the activity (expansion, replication, collaboration) Identify the planned method or pathway of achieving the intent, i.e., how the activity will be scaled Determine the organizational roles involved in the activity and expected roles in scale up Outline the intended scope of the scaling up effort and the where and amongst whom the scaling will occur What organizational process, and technical factors are/were critical to success in the activity? Is it necessary to replicate all elements of the model on a large scale? Does the organization that carried out the activity project have the desire and organizational capacity to expand its operations and deliver services on a substantially larger scale? If not, which organization(s) are best suited and motivated to implement the model on a scaled-up basis or to serve as partners in implementing the model? Should the effort include policy change by the government or rely exclusively on voluntary adoption by private and non-governmental organizations? Is there a need for one or more intermediary organizations to support the scaling up process? If so, what help is needed, and which organizations are best suited to perform these roles? Along what dimension(s) should scaling up take place? What would scaling up look like if it were successful in terms organizational performance? Activity Implementation Outline how a constituency was built to legitimize the change envisioned by the activity Understand how this constituency was used to implement and adapt the activity What was done to persuade relevant decision makers, funders, and opinion leaders that new solutions are necessary and desirable? What was done to persuade relevant decision makers, funders, and opinion leaders that the proposed model is successful, cost-effective, and feasible? Which organizations, organizational units, or individuals were responsible for key decisions regarding the funding and implementation of scaling up? Who has the authority to make decisions within these organizations? What arguments, appeals, or advocacy strategies were used to be persuasive to decision makers? What were the most effective networks and alliances for carrying out this advocacy, and how can they be most efficiently mobilized and organized? How was buy-in from the leadership and staff of potential implementing organizations best achieved? Activity Learning Identify the key learning points that were most important to success and where the data came from to inform the learning What changes were made during implementation? What prompted these changes? What information was used to identify the need for the change? How was this information used? To this end, the evaluation asked questions about how ABCG identifies, designs, implements, and learns from its activities (primarily pilots and best practice dissemination activities). Interviewers gave probing questions and prompts that solicited answers specific to the role of ABCG “ideas” (Q2) and “influence” 7 (Q3). The team also coded responses using these constructs during analysis. Likewise, structuring KII’s in this manner helps to isolate evidence on priority themes because a literature review on scale up pathways, value propositions, and governance have yielded important evidence that was used to inform the prompts and probing questions for each module. The interviewer administering the KII was assigned specific Working Groups so that, in addition to the evidence from the literature review, the interviewer can tailor the questions and probes to the specific priorities and activities of that Working Group. Hence, the evaluation will be able to compare responses across modules while still yielding evidence specific to that Working Group. At the same time, no specific module is devoted to a specific evaluation question, for example a single evaluation question can be informed by every module depending on how the respondent responds. The complete KII questionnaire follows. However, as noted, KII questions were adapted to different groups of respondents. The questionnaire was made available to all respondents at least a week prior to interviews to ensure that respondents understand the questions and come into the KII prepared. 8 KEY INFORMANT INTERVIEW (KII) QUESTIONNAIRE General KII Instrument (modified for each cohort of respondent, Steering Committee and Working Groups) The United States Agency for International Development has commissioned Zemitek to perform a performance evaluation on its funding to ABCG. As such, we are very interested in the work you do with the (title of USAID Staff, Working Group, or Steering Committee). We would like to ask you a few questions about your work and your opinions in this interview which should take no more than 60 minutes depending on how much you want to share and the substance of the conversation. Any question you do not want to answer we can skip and your involvement in the interview is completely voluntary. Please feel free to ask for clarification if anything is confusing and if you think I am leaving something out with my questions feel free to include anything you feel is relevant. If this is agreeable to you, can we begin? If so then the interview is comprised of five sections that correlate to how ABCG identifies the need for an activity, how the activity is designed, implemented, and learned from. By asking you questions in this format we hope to gain an understanding of how ABCG uses it influence to spread biodiversity ideas that lead to improved outcomes. We might request your thoughts on additional people to talk to or materials on issues that we identify in the interview. Module #1: Theory of Change (ToC) Insert relevant ToC here for reference during KII 1.) What do you feel are the biggest strengths of the theory of change (e.g., ABCG, Working Group ToC)? Why? 2.) What are the biggest weaknesses? Why? 3.) Can you describe the process by which the theory of change is updated? Do you feel this process is sufficient? Why or why not? Module #2: Activity Selection 1.) What is the process for identifying an activity? How is the need identified? Who is involved? 2.) What information is used in these processes? 3.) How does ABCG focus on problems that relate to its strengths, namely using ABCG influence to spread best practices and other ideas? 4.) Module 1 Probes and Prompts What do you consider to be ABCG’s primary two or three objectives? What are the critical assumptions pertaining to these objectives? What are the biggest risks in achieving these objectives? Do you feel there have been changes to these objectives, assumptions, and risks? What was the reason for the change? Module 2 Probes and Prompts What are some of the problem statements for the activity? Why are these problems deemed most critical and warranting of an activity? How was this decided? What idea or influence is ABCG hoping to advance? What type of scale is envisioned for this activity? 9 Module #3: Activity Planning 1) Generally speaking, how are activities planned? What is the process? Who is involved? 2) How are ABCG ideas and influence integrated into the activity planning? 3) How is the ultimate goal of the activity planned for? - meaning how is the use of the knowledge, new capacity, etc., planned for? (Make heavy use of probes and prompts) Module #4: Activity Implementation 1.) How is the sustainability of the activity managed during implementation? Meaning what is expected after the activity is completed? How is this factored into implementation? (Make heavy use of probes and prompts) Module 3 Probes and Prompts What organizational process, and technical factors are/were critical to success in the activity? Is it necessary to replicate all elements of the model on a large scale? Does the organization that carried out the activity project have the desire and organizational capacity to expand its operations and deliver services on a substantially larger scale? If not, which organization(s) are best suited and motivated to implement the model on a scaled-up basis or to serve as partners in implementing the model? Should the effort include policy change by the government or rely exclusively on voluntary adoption by private and non-governmental organizations? Is there a need for one or more intermediary organizations to support the scaling up process? If so, what help is needed, and which organizations are best suited to perform these roles? Along what dimension(s) should scaling up take place? What would scaling up look like if it were successful in terms organizational performance? Module 4 Probes and Prompts What was done to persuade relevant decision makers, funders, and opinion leaders that new solutions are necessary and desirable? What was done to persuade relevant decision makers, funders, and opinion leaders that the proposed model is successful, cost-effective, and feasible? Which organizations, organizational units, or individuals were responsible for key decisions regarding the funding and implementation of scaling up? Who has the authority to make decisions within these organizations? What arguments, appeals, or advocacy strategies were used to be persuasive to decision makers? What were the most effective networks and alliances for carrying out this advocacy, and how can they be most efficiently mobilized and organized? How was buy-in from the leadership and staff of potential implementing organizations best achieved? 10 Module #5: Activity Learning 1. How is success defined for each activity? How do you know when success is achieved? 2.) How are changes made during implementation? What information Is used to inform these changes? 3.) How have you modified your practices based on feedback on the success and failures of activities? Module #6: COVID 1.) What has been the biggest impact from COVID? 2.) What steps were taken to mitigate this? Was this successful? Why/why not? Module 5 Probes and Prompts What changes were made during implementation? What prompted these changes? What information was used to identify the need for the change? How was this information used? 11 ANNEX IV: E-SURVEY REPORT The evaluation team performed two E-Surveys, one for the composition of the ABCG Listserv and one for ABCG Activity Managers. ABCG uses various methods to disseminate its various lessons learned, best practices, etc. The primary methods include the email Listserv, publications on its website, webinars, and workshops. This task included: ● identifying survey champions, developing survey message, and delivering message to respondents for each E-Survey three days before survey is administered ● administering surveys to the entire ABCG Listserv and Activity Manager roster and sending reminder email Both E-Surveys were performed using Survey Monkey15 with responses being anonymized, but all aggregations made readily available to USAID upon request. ABCG LISTSERV SUBSCRIBERS E-SURVEY The SOW specified that the consultants “will develop and conduct an e-survey with ABCG Listserv subscribers to gather their perspectives on ABCG.” The 2015 ABCG Evaluation16 conducted two similar E-Surveys (with the same data sources for sampling) but with very low response rates that introduced risk to the use of those survey results. The 2015 Evaluation noted significant non-response bias (72 Listserv respondents / ca. 1400 Listserv members = ca. 5% response rate). Relying on the results from this low response rate more than likely created overly positive feedback on ABCG product (e.g., “Ideas”) dissemination methods for new advances and approaches in Africa. To mitigate this risk, and in accordance with best practice, the 2021 evaluation team worked with ABCG and USAID to identify one or two “Survey Champions”, i.e., individuals who each respondent group recognizes as an authority, a respected and/or trusted leader to communicate a voice of support for the survey and its value. Once identified, the Evaluation Team worked with each respective champion to draft an email that was sent to each respondent list to convey this message and to increase the response and completion rate over the 2015 evaluation. In addition, the team attempted to shorten each survey and introduce more closed ended questions to improve response and completion rates. In an attempt to improve the 2021 E-survey response rate and reduce bias, the ABCG II evaluation also shortened the expected time to complete the Listserv E-surveys from ca. 10 minutes in 2015, to ca. 3 minutes in 2021. In addition, both E-surveys were kept open for 13 days with two separate reminders sent by the ABCG Secretariat. However, despite these efforts, the 2021 evaluation had a similar non-response bias with an even lower response rate (4.3%) with only 121 Listserv respondents completing the survey out of 2,823 Listserv subscribers contacted. 15 Survey Monkey: https://www.surveymonkey.com/ 16 USAID (2015). Africa Biodiversity Collaborative Group. Performance Evaluation Report. Prepared by David Callihan, Karen Menczer, and Elizabeth Freudenberger under the Management Support and Technical Analysis Services Project (MSTAS), implemented by the Pragma Corporation and Management Systems International. USAID Africa Bureau Office of Sustainable Development, January 2015, Washington, DC. 12 Non-response bias17 usually stems from high levels of disinterest amongst most respondents towards the issues on which they are being surveyed. It is likely that those who do respond are doing so because they represent outlier respondents, i.e., those who have high levels of approval and support for the issues. However, it is important to note there is no conclusive evidence that this is the type of non-response bias experienced here, just a high likelihood. From the evaluators’ experience with administering E-surveys to a large number of respondents, this result is unfortunately not uncommon. Such a response rate makes inclusion of analyzed responses questionable if not actually useless. Therefore, this evaluation cannot draw any conclusions from them given the significant non-response bias. This prevents drawing false conclusions from faulty data. All conclusions for EQ2, EQ3, and EQ4, the only questions relevant to the Listserv survey, are based on qualitative data, i.e., from KII interviews, documents, and the Activity Manager E-Survey and not from the Listserv E-Survey responses (due to the non-response bias). The data is presented here only because it was collected but any analysis of the data should factor in the significant non-response bias. From e-survey results, the ABCG Listserv, as the primary method of disseminating ABCG products at scale, has at least 517 members outside of Africa (18%). This number would be expected to be significantly higher given that 2,387 Listserv members either did not respond at all or did not identify their nationality as requested on the Listserv questionnaire. Fifty-six of the 72 Listserv E-Survey respondents (78%) came from countries outside Africa and thus were not representative of the African population with which ABCG seeks to collaborate. However, given the marginally low cost of the Listserv and similar dissemination methods (e.g., posting to the website), their value-add could be significant even if their performance could be improved. The ABCG Listserv Subscribers E-Survey Questionnaire follows. This E-survey is designed to address the Evaluation Questions (EQ 2 - 4) on Ideas, Influence, and Impact. Responses are assigned numbers where applicable, i.e., Likert scale, for analysis. 17 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=1691967 13 ABCG LISTSERV SUBSCRIBERS’ E-SURVEY QUESTIONNAIRE The time you take to complete this questionnaire is very much appreciated. Approximate time to complete the questionnaire is less than 5 minutes. All responses are kept anonymous. Respondent Profile Information 1. What is your nationality? ……………………………….. 2. In which country are you currently located? ……………………………….. 3. What is your gender? ● Female ● Male ● Non-binary ● Prefer not to respond 4. What is your age? ● Under 20 ● 20-29 ● 30-39 ● 40-49 ● 50-59 ● 60 or over ● Prefer not to respond 5. Which of the following best describes your work? ● Non-governmental organization (NGO) ● African NGO / CBO ● Donor government / funding foundation ● African government ● Multilateral organization (e.g., UN) ● University / research institution ● Consulting / contractor ● Private sector ● Student ● Unemployed or retired ● Other ………………. 6. If you work in conservation, which of the following sectors pertain to this work? (Please select all that apply) ● Biodiversity ● Forestry ● Marine ● Coastal ● Wildlife management ● Rangelands ● Climate change ● Other ………………. 14 7. How long have you been an ABCG Listserv subscriber? ● Less than one year ● One – five years ● More than five years 8. How frequently have you visited the ABCG website over the past six months (www.abcg.org)? ● Weekly ● Several times a month ● Monthly ● One – four times ● Not at all over the past 6 months Ideas (EQ2) 9. How useful do you find each of the following types of Listserv messages? Message topic Very useful Somewhat useful Not very useful Not at all useful Do not know New Publications Presentation & Webinar Summaries Job alters Announcements on future events Other ……………… 10. Are summaries of lessons learned on new advances and approaches easy to access or locate? ● Very easy ● Somewhat easy ● Not very easy ● Not at all easy ● Do not know 11. Does ABCG summarize lessons learned on new advances and approaches from the theme areas (6 focus topics) in a way that is useful to you? ● Very useful ● Somewhat useful ● Minimally useful ● Not very useful ● Not at all useful ● Do not know 12. Which of the following themes, if any, would benefit from additional summaries of lessons learned on new advances and approaches and on ABCG’s contribution? (Please select all that apply) ● Land and Resource Tenure Rights (LRTR) ● Land Use Management (LUM) ● Community-Based Forest Management (CBFM) ● Global Change Impacts (GCI) ● Global Health Linkages to Biodiversity Conservation: Population Health and Environment (PHE) ● Global Health Linkages to Biodiversity Conservation: Freshwater Conservation and Water, Sanitation & Hygiene (FW-WASH) ● Integrating Gender and Vulnerable Populations in Activity Design and Implementation 15 ● No themes need additional summaries Influence (EQ3) 13. Have you applied knowledge gained from ABCG in your professional work? ● Yes ● No ● Do not know 14. Have you applied knowledge gained from ABCG in your professional work? ● Yes ● No ● Do not know 15. To what degree have the following been useful in helping you acquire new knowledge, tools, or approaches from ABCG products? ABCG Products Very useful Somewhat useful Not very useful Not at all useful Do not know Website Publications Presentations (in-person) Webinars (virtual) Conferences or events Social Media (e.g., Facebook, Instagram, Twitter, LinkedIn) ABCG ACTIVITY MANAGERS E-SURVEY ABCG Activity Managers were defined using the same inclusion criteria as the 2015 survey for consistency resulting in a 85% response rate. The ABCG Activity Managers E-Survey Questionnaire follows. This E￾survey is designed to address all Evaluation Questions (EQ 1 - 6) directly or indirectly. Responses were assigned numbers where applicable, i.e., Likert scale, for analysis. ABCG ACTIVITY MANAGERS E-SURVEY QUESTIONNAIRE Respondent Profile and Activity Information The time you take to complete this questionnaire is very much appreciated. Approximate time to complete the questionnaire is less than 5 minutes. All responses are kept anonymous. 1. What is your nationality? ……………………………….. 2. In which country are you currently located? ……………………………….. 3. What is your gender? ● Female ● Male ● Non-binary ● Prefer not to respond 4. What is your age? ● Under 20 16 ● 20-29 ● 30-39 ● 40-49 ● 50-59 ● 60 or over ● Prefer not to respond 5. With which ABCG theme were you engaged? ● Land and Resource Tenure Rights (LRTR) ● Land Use Management (LUM) ● Community-Based Forest Management (CBFM) ● Global Change Impacts (GCI) ● Global Health Linkages to Biodiversity Conservation: Population Health and Environment (PHE) ● Global Health Linkages to Biodiversity Conservation: Freshwater Conservation and Water, Sanitation & Hygiene (FW-WASH) ● Integrating Gender and Vulnerable Populations in Activity Design and Implementation 6. How effective was ABCG in disseminating the new conservation advances and approaches? ● Highly effective ● Moderately effective ● Minimally effective ● Not at all effective ● Do not know 7. Based on your organization’s involvement in this activity, how satisfied were you with each of the following? Topic Very satisfied Somewhat satisfied Somewhat dissatisfied Very dissatisfied Do not know ABCG activity or project approval process ABCG project management process ABCG governance procedures ABCG financial management process Overall engagement 8. What are the primary problems you see in ABCG’s performance in the following areas (leave blank if no response desired)? ● ABCG Activity or Approval Process ● ABCG project management process ● ABCG governance ● ABCG financial management ● Overall management 17 ANNEX V: SUMMARY OF ANNUAL INDICATOR REPORTING FROM FY 2016-2020 ABCG M&E PLANS Table 1: Summary of Indicator Reporting from FY 2016 ABCG Annual Plan ABCG Thematic Area # Output Indicators # Output Indicators Achieving Target # Outcome Indicators # Outcome Indicators Achieving Target Central Administration No Actuals Reported LRTR 4 3 4 1 Global Change 3 0 0 0 Land Use Management 3 1 0 0 Global Health (PHE) 2 0 1 0 Global Health (WASH) 3 1 0 0 Totals 15 5 5 1 Overall Success Rate 34% 20% Table 2: Summary of Indicator Reporting from FY 2017 ABCG Annual Plan ABCG Thematic Area # Output Indicators # Output Indicators Achieving Target # Outcome Indicators # Outcome Indicators Achieving Target Central Administration No Actuals Reported LRTR 5 3 4 2 Global Change 4 4 0 0 Land Use Management 5 3 0 0 Global Health (PHE) 3 3 0 0 Global Health (WASH) 6 5 0 0 Totals 23 18 4 2 Overall Success Rate 78% 50% Table 3: Summary of Indicator Reporting from FY 2018 ABCG Annual Plan ABCG Thematic Area # Output Indicators # Output Indicators Achieving Target # Outcome Indicators # Outcome Indicators Achieving Target Secretariat 4 3 7 1 LRTR 5 5 3 2 Global Change 6 5 1 0 Land Use Management 9 5 1 1 Global Health (PHE) 4 4 2 2 Global Health (WASH) 7 7 0 0 Totals 35 29 14 6 Overall Success Rate 83% 43% Table 4: Summary of Indicator Reporting from FY 2019 ABCG Annual Plan ABCG Thematic Area # Output Indicators # Output Indicators Achieving Target # Outcome Indicators # Outcome Indicators Achieving Target Secretariat 10 5 1 0 Land Use Management 5 1 2 0 18 ABCG Thematic Area # Output Indicators # Output Indicators Achieving Target # Outcome Indicators # Outcome Indicators Achieving Target Global Change Impacts 2 2 0 0 Community Based Forest Management 6 0 0 0 Global Health, Pop Health and Environment 6 0 1 0 WASH 5 2 0 0 Totals 34 10 4 0 Overall Success Rate 29.4% 0% Table 5: Summary of Indicator Reporting from FY 2020 ABCG Annual Plan ABCG Thematic Area # Output Indicators # Output Indicators Achieving Target # Outcome Indicators # Outcome Indicators Achieving Target Secretariat 13 6 1 0 Land Use Management 9 2 2 1 Global Change Impacts 3 2 1 0 Community Based Forest Management 8 3 1 0 Global Health, Pop Health and Environment 8 3 1 0 WASH 9 3 1 0 Totals 50 19 7 1 Overall Success Rate 38% 14.2% 19 ANNEX VI: REFERENCES AND SOURCES OF INFORMATION No Documents Reviewed Annual Reports 1 ABCG (2020). Africa Biodiversity Collaborative Group II FY 2020 Annual Report (October 1, 2019–September 30, 2020). Submitted to USAID Bureau for Africa /Office of Sustainable Development (AID-OAA-A-15- 00060), December 2020, Washington, DC. 2 ABCG (2019). Africa Biodiversity Collaborative Group II FY 2019 Annual Report (October 1, 2018–September 30, 2019). Submitted to USAID Bureau for Africa /Office of Sustainable Development (AID-OAA-A-15- 00060), December 2019, Washington, DC. 3 ABCG (2018). Africa Biodiversity Collaborative Group II FY 2018 Annual Report (October 1, 2017–September 30, 2018). Submitted to USAID Bureau for Africa /Office of Sustainable Development (AID-OAA-A-15- 00060), December 2018, Washington, DC. 4 ABCG (2017). Africa Biodiversity Collaborative Group II FY 2017 Annual Report (October 1, 2016–September 30, 2017). Submitted to USAID Bureau for Africa /Office of Sustainable Development (AID-OAA-A-15- 00060), December 2017, Washington, DC. 5 ABCG (2016). Africa Biodiversity Collaborative Group II FY 2016 Annual Report (October 1, 2015–September 30, 2016). Submitted to USAID Bureau for Africa /Office of Sustainable Development (AID-OAA-A-15- 00060), December 2016, Washington, DC. 6 ABCG (2015). Africa Biodiversity Collaborative Group Final Report for the Biodiversity Analysis and Technical Support (BATS)Program, USAID/AFR/SD. USAID/Africa Award # RLA-A-00-07-00043. Washington, DC: USAID-Bureau for Africa. (FY 2008 – 2015) Work Plans and Business Plans 7 ABCG (2020). Africa Biodiversity Collaborative Group (ABCG) III: Collaboration for African Biodiversity (ABCG III). Proposal developed by African Wildlife Federation In partnership with Conservation International, the Jane Goodall Institute, The Nature Conservancy, Wildlife Conservation Society, World Resources Institute, and World Wildlife Fund, July 2020, Washington, DC. 8 ABCG (2020). Africa Biodiversity Collaborative Group Strategic Business Plan. Prepared with ABCG by Impact by Design, Inc., Tacoma Park, Maryland. 9 ABCG (2020). Africa Biodiversity Collaborative Group Membership Charter. ABCG, January 2020. 10 ABCG (2019). Africa Biodiversity Collaborative Group (ABCG II) FY 2019-2020 Work Plan, Draft Version 1I of March 2019 (September 22, 2018 - September 21, 2020). produced by Wildlife Conservation Society (WCS) for review by the United States Agency for International Development Bureau for Africa, Office of Sustainable Development (AID-OAA-A-15-00060), February 2019. Monitoring Plans, Evaluation Reports, Assessments 11 ABCG (2019). Africa Biodiversity Collaborative Group Activity FY 2019-2020 Activity Monitoring, Evaluation, & Learning (MEL) Plan (September 22, 2018 - September 21, 2020). Implemented by Wildlife Conservation Society with partners African Wildlife Foundation, Conservation International, the Jane Goodall Institute, The Nature Conservancy, World Wildlife Fund, World Resources Institute, Version II: March 2019, Washington, DC. 12 Impact by Design (2019). ABCG Internal Assessment. Impact by Design, Inc., Tacoma Park, Maryland, March 2019. 13 ABCG (2018). ABCG Theory of Change (Review of Strategic Focus, Results Chain Analysis, and Revised Theory of Change). PowerPoint presentation to USAID, October 2018, Washington, DC. 14 ABCG (2018). Maximizing Impact (Review of Strategic Direction, Current Governance Structures, Roles and Responsibilities of the Secretariat, Needed Capacities, and Available Resources. PowerPoint presentation to USAID, October 2018, Washington, DC. 15 USAID (2015). Africa Biodiversity Collaborative Group. Performance Evaluation Report. Prepared by David Callihan, Karen Menczer, and Elizabeth Freudenberger under the Management Support and Technical Analysis Services Project (MSTAS), implemented by the Pragma Corporation and Management Systems International. USAID Africa Bureau Office of Sustainable Development, January 2015, Washington, DC. 16 ABCG (2009). The Future of Biodiversity in Africa: Report of a Consultation (2007 – 2009). Africa Biodiversity Collaborative Group, produced by IUCN/US on behalf of ABCG for USAID, Washington DC. 20 Best Practice and Lessons Learned 17 ABCG (2018). Nature Protected and Lives Impacted: ABCG Success Stories 2015-2018. Africa Biodiversity Collaborative Group, Washington, DC. 18 ABCG (2018). Land Use Planning Lessons Sharing. Report of the workshop held by the ABCG Land Use Management Working Group in Libreville, Gabon, September 4-6, 2018. Compiled by Daphne Wang, World Wildlife Fund-Gabon. USAID Bureau for Africa /Office of Sustainable Development, Washington DC. 19 ABCG (2018). The Convergence Factor: Lessons from Integrating Freshwater Conservation and WASH in Africa. By Lydia Cardona, Janet Edmond, and Colleen Sorto, produced by Wildlife Conservation Society on behalf of Africa Biodiversity Collaborative Group. USAID Bureau for Africa /Office of Sustainable Development, Washington DC. 20 ABCG (2017). Exploring Cross-Sector Linkages Between Population, Health, Environment, Nutrition, and Food Security: A Review of Best Practices and Lessons Learned. By Janet Edmond, Kimberly Holbrook, Alice Macharia, Jimmiel Mandima, and Nathalie Simoneau, Africa Biodiversity Collaborative Group, USAID Bureau for Africa /Office of Sustainable Development, Washington, DC. Other ABCG Documents 21 ZemiTek (2020). ABCG II Performance Evaluation Statements of Work for Lead Evaluator and Evaluator. Africa Bureau Institutional and Technical Support Services: AID-OAA-C-15-00132, AFR/SD, Division of Economic Growth, Environment, and Agriculture (EGEA), Biodiversity Team, Washington, DC. 22 Walter, E., Sorto, C., Edmond, J., Mercurio, S., and Rozenberg, E. (2020). Freshwater Conservation and WASH Advocacy Strategy Workshop: Facilitator’s Guide. Africa Biodiversity Collaborative Group and IRC, Washington, DC. 23 ABCG (2019?). Africa Biodiversity Collaborative Group Fact Sheet, ABCG, Washington, DC. 24 ABCG (2017?). Piloting Mechanisms for Strengthening African Conservation Leadership and Organizational Capacity. The Nature Conservancy (TNC, with Maliasili Initiatives) and the World Resources Institute (WRI), recipients of the Africa Biodiversity Collaborative Group’s (ABCG) Emerging Issues small grant. 25 ABCG (2017). Gender Analysis Report – One Health: Integrating Freshwater Conservation, WASH and Rangeland Management in South Africa’s Mzimvubu Catchment. By Janet Edmond, Nolubabalo Kwayimani, Colleen Sorto, and Brittany Ajroud, produced by Conservation International and Conservation South Africa on behalf of Africa Biodiversity Collaborative Group, USAID Bureau for Africa /Office of Sustainable Development, Washington, DC. 26 ABCG (2017?). Africa Biodiversity Collaborative Group Fact Sheet, ABCG, Washington, DC. 27 ABCG (2015?). A Vision for the Future of Biodiversity in Africa. Africa Biodiversity Collaborative Group, USAID/AFR/SD. USAID/Africa Award # RLA-A-00-07-00043. Washington, DC 28 Lichtenfeld, Laly (2011). A 21st Century Approach to African Wildlife Conservation: Helping Africa’s People Engage in the Conservation Arena. African People & Wildlife Fund, brown bag presentation to ABCG, hosted by The Nature Conservancy, September 29, 2011, Washington, DC. Other Documents Relevant to Evaluation Report Preparation 29 GlobeScan Inc. & WWF-US (2021). Covid-19 One Year Later: Public Perceptions About Pandemics and Their Links to Nature: China, Myanmar, Thailand, US and Vietnam: Findings From 2021 Survey. Produced by GlobeScan, commissioned by WWF-US, written by Dr. Daniel Bergin, Timothy Cheng, Crystal Yang, Dr. Eugene Kritski, and Wander Meijer. WWF-US, Washington, DC., May 2021, 228 pp. 30 GlobeScan Inc. & WWF (2021). AS ABOVE – Summary and Infographics., WWF-International, Gland, Switzerland, May 2021, 11pp. 31 USAID (2020). Monitoring, Evaluating, and Learning from Biodiversity Conservation’s Benefits Across Sectors. Biodiversity Results and Integrated Development Gains Enhanced (BRIDGE) Project, USAID Bureau for Economic Growth, Education, and Environment, Office of Forestry and Biodiversity. REQ-EGEE-18- 000127 (Measuring Impact II) implemented by prime recipient Environmental Incentives, LLC in partnership with Foundations of Success, and ICF Macro, Inc., July 2020, USAID, Washington, DC. 32 CMP (2020). Open Standards for the Practice of Conservation, or Conservation Standards (CS), version 4.0. Conservation Measures Partnership: https://conservationstandards.org/ 21 33 Cooley, Larry, and Johannes F. Linn (2014). Taking Innovations to Scale: Methods, Applications, and Lessons. Management Systems International (MSI), Results for Development Institute, Washington, DC: https://www.usaid.gov/sites/default/files/documents/1865/v5web_R4D_MSI-BrookingsSynthPaper0914- 3.pdf 34 USAID (2020). Technically Strong and Politically Savvy: Enhancing Thinking and Working Politically When Practicing the Conservation Standards at USAID, Supplemental Guide 4. By Schuttenberg, H.Z., J. Stark, and K. Schueler; Biodiversity Results and Integrated Development Gains Enhanced (BRIDGE) Project, USAID Bureau for Economic Growth, Education, and Environment, Office of Forestry and Biodiversity, Washington, DC: https://biodiversitylinks.org/projects/completed-projects/bridge/bridge￾resources/supplemental-guide-4-technically-strong-and-politically-savvy-enhancing-thinking-and-working￾politically-when-practicing-the-conservation-standards-at-usaid 35 USAID (2018). Thinking and Working Politically Through Applied Political Economy Analysis: A Guide for Practitioners. By Alina Rocha Menocal, Marc Cassidy, Sarah Swift, David Jacobstein, Corinne Rothblum, and Ilona Tservil; Center of Excellence on Democracy, Human Rights, and Governance; USAID Bureau for Democracy, Conflict, and Humanitarian Assistance (DCHA). Washington, DC: https://www.usaid.gov/documents/1866/thinking-and-working-politically-through-applied-political￾economy-analysis 36 USAID (2016). Defining Outcomes & Indicators for Monitoring, Evaluation, and Learning in USAID Biodiversity Programming. Biodiversity How-To Guide 3, Measuring Impact, by Caroline Stem and Richard Margoluis, Foundations of Success; Marco Flores, USAID Bureau for Economic Growth, Education, and Environment/Office of Forestry and Biodiversity, Environmental Incentives, LLC, Washington, DC: https://usaidlearninglab.org/sites/default/files/resource/files/biodiversity_howtoguide3_508.pdf 37 USAID (2016). Pathways to Scale: A guide for early-stage global health innovators on business models and partnership approaches to scale-up. USAID Center for Accelerating Innovation and Impact (CII), Washington, DC: https://www.usaid.gov/cii/pathways-scale 38 USAID (2016). Program Cycle Guidance Discussion Note: Complexity-Aware Monitoring. Policy, Planning, and Learning (June 2016), USAID’s Office of Learning, Evaluation, and Research, Washington, DC. 39 USAID (2015). Africa Biodiversity Collaborative Group. Performance Evaluation Report. Prepared by David Callihan, Karen Menczer, and Elizabeth Freudenberger under the Management Support and Technical Analysis Services Project (MSTAS), implemented by the Pragma Corporation and Management Systems International. USAID Africa Bureau Office of Sustainable Development, January 2015, Washington, DC: https://www.usaid.gov/sites/default/files/documents/1865/201sad.pdf