MIDTERM EVALUATION OF THE PROTECTING ECOSYSTEMS AND RESTORING FORESTS IN MALAWI (PERFORM) PROJECT July 2017 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Sergio Cambronero, Keith J. Forbes, Austin Tibu, Blessings Manda, Jim Phillips, and Merrill Anne Jordan of International Business & Technical Consultants, Inc. (IBTCI). MIDTERM EVALUATION: OF THE PROTECTING ECOSYSTEMS AND RESTORING FORESTS IN MALAWI (PERFORM) PROJECT Originally Submitted August 16, 2017 Revised Version Submitted September 14, 2017 Revised Version Submitted October 30, 2017 AID-OAA-I-15-00022/AID-612-TO-17-00001 FINAL VERSION International Business & Technical Consultants, Inc. (IBTCI) 8618 Westwood Center Drive, Suite 400 Vienna, VA 22182 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Cover – Members of the Takondwa Farmers Club, Chaphonya Village, Mzimba District, Northern Malawi. Photo captured during the visit of the evaluation team in July 2017. TABLE OF CONTENTS Table of Contents Acronyms Executive Summary .................................................................................................................................................... 1 Evaluation Purpose ........................................................................................................................................................ 1 Evaluation Questions .................................................................................................................................................... 1 Methods ........................................................................................................................................................................... 1 Sampling and Limitations ............................................................................................................................................. 2 TOC Comparisons, PMEP Targets and Data Quality .......................................................................................... 2 Findings, Conclusions, and Recommendations ...................................................................................................... 2 Evaluation Purpose & Evaluation Questions ................................................................................................... 7 Evaluation Purpose ........................................................................................................................................................ 7 Evaluation Questions .................................................................................................................................................... 7 Project Background ..................................................................................................................................................... 8 Evaluation Design, Methods, & Limitations..................................................................................................... 8 Data Collection Methods .......................................................................................................................................... 10 Data Collection Tools and Adjustment to Context ........................................................................................... 11 Sampling ......................................................................................................................................................................... 12 Limitations ..................................................................................................................................................................... 12 Brief Overview of PMEP Targets and Data Quality ............................................................................................ 13 Approach to TOC Analysis .................................................................................................................................... 15 Brief Context for the Consideration of PERFORM's TOC .............................................................................. 16 Approach to TOC Comparisons ............................................................................................................................ 16 Findings, Conclusions, and Recommendations by Evaluation Question ......................................... 18 ANNEXES Annex 1. Evaluation Statement of Work ......................................................................................................... 1 Annex 2. Evaluation Design, Methods and Sources................................................................................... 16 Annex 3. Data Collection Tools and Resources .......................................................................................... 24 Annex 4. Conflict of Interest Forms for Team Members ....................................................................... 32 Annex 5. Statement of Differences ................................................................................................................... 34 Annex 6. TOC Comparisons ................................................................................................................................. 35 ACRONYMS AFOLU Agriculture, Forestry, and Other Land Use BMU German Federal Ministry for the Environment, Nature Conservation and Nuclear Safety BMZ German Federal Ministries of Economic Cooperation and Development CA Conservation Agriculture CEPA Center for Environmental Policy and Advocacy CICOD Circle for Integrated Community Development CIP Commodity Import Program CSO Civil Society Organizations DC District Councils DESC District Environmental Sub-Committee DFID United Kingdom Department for International Development DFO District Forestry Officers DoF Department of Forestry DP Development Partners EAD Environmental Affairs Department FGD Focus Group Discussions FLRA Forest Landscape Restoration Assessment FRIM Forestry Research Institute of Malawi FTF Feed the Future GCC Global Climate Change GEF Global Environmental Facility GCF Green Climate Fund GHG Greenhouse Gas GIS Geographic Information System GIZ German International Cooperation Agency GOM Government of Malawi GVH Group Village Headman INDC Intended Nationally Determined Contribution INC Initial National Communication IP Implementing Partner JICA Japan International Cooperation Agency KHP Kawandama Hills Plantations KII Key Informant Interviews LAMIS Local Assembly Management Information System LDF Local Development Fund LOP Life of Project LUANAR Lilongwe University of Agriculture and Natural Resources MAIWD Ministry of Agriculture, Irrigation and Water Department MASAF Malawi Social Action Fund MASDAP Malawi Spatial Data Platform MCA Millennium Challenge Account MGDS Malawi Growth and Development Strategy MISST Malawi Improved Seed Systems and Technologies MNREM Ministry of Natural Resources, Energy and Mining MOU Memorandum of Understanding MRPAP Malawi REDD+ Action Plan MTE Midterm Evaluation MZUNI Mzuzu University NCSP National Communications Support Programme NFLRS National Forest Landscape Restoration Assessment and Strategy NFMS National Forest Monitoring System NGO Non-Governmental Organization NTCCC National Technical Committee on Climate Change OFSP Orange-fleshed sweet potatoes PERFORM Protecting Ecosystems and Restoring Forests in Malawi PMEP Project Monitoring and Evaluation Plan REDD+ Reducing Emissions from Deforestation and Forest Degradation RExG REDD+ Expert Group RS Remote Sensing SOP Standard Operating Procedures SOW Scope of Work SRBMP Shire River Basin Management Program UN United Nations UNDP United Nations Development Program UNEP United Nations Environmental Program UNFCCC United Nations Framework Convention on Climate Change UN-REDD United Nations Reducing Emissions from Deforestation and Forest Degradation USAID United States Agency for International Development USFS U.S. Forest Service USFS IP USFS International Programs VNRMC Village Natural Resources Management Committee 1 EXECUTIVE SUMMARY USAID’s Protecting Ecosystems and Restoring Forests in Malawi (PERFORM) was specifically designed to align with Malawi’s Growth and Development Strategy (MGDS) and to promote forest conservation and green growth. It is the main implementation vehicle for the low-emissions partnership between the U.S. and GOM. The project period is five years, from 2014 to 2019, and constitutes a key activity under “Development Objective 2: Sustainable Livelihoods Increased” of USAID Malawi’s CDCS. EVALUATION PURPOSE The primary purpose of the PERFORM mid-term performance evaluation is to assess the effectiveness of this USAID funded activity, considering both its design and implementation processes. The evaluation also seeks to inform management of course corrections, if any, are needed. Additionally, through this evaluation, USAID seeks strategic and programmatic options for future engagement to support climate change mitigation and adaptation in Malawi. Consistent with USAID’s Evaluation Policy, the primary goal of this mid-term evaluation is to provide evidence so that USAID/Malawi can determine whether PERFORM is on track to achieve its desired results or not. EVALUATION QUESTIONS In compliance with the approved SOW, the PERFORM mid-term performance evaluation will be guided by the following evaluation questions: 1a. How effectively has PERFORM supported the GOM’s national climate change goals including the development of a GHG inventory, implementing its National REDD+ Action Plan, and developing and implementing its Intended Nationally Determined Contribution (INDC) under the UN Framework Convention on Climate Change? 1b. Do the pilot projects at the local level effectively support the national goals, and are they achieving meaningful livelihood benefits at the site level? 2. What lessons and implications does PERFORM offer for REDD+ readiness scaling up and likely impacts on REDD+ outcomes? This can also include a synthesis of lessons from the REDD+ Readiness Program. 3. How sustainable are the efforts at the local and national level? Are GOM officials and local communities likely to be able to continue these programs on their own when PERFORM ends? 4. How well is PERFORM integrated with: a) other USAID initiatives (e.g., Feed the Future): and b) other DPs efforts in GCC and forest conservation? 5. How are gender issues integrated into the activity? Have women been able to participate fully and benefit from it, including from income-generating activities? For cross cutting subjects such as gender and capacity building, how effectively do they contribute to PERFORM’s intended results? METHODS In crafting a relevant and useful response to the main evaluation questions, Key Informant Interviews (KIIs) were the core data source used to assessing progress (a total of 42 individual interviews), with a selected number of group interviews. KIIS were supported by semi-structured interview protocols, which included three Village Farmer Clubs with an average of 25 participants per interview. A mini￾survey was planned with the representatives of five GVHs in Machinga district, but this had to be converted to a group discussion due to English language limitations. Group interviews enabled greater flexibility in question/topic emphasis. Although interviewed as a group, a small number of PERFORM key counterparts (5) have been noted as individual responses since the format enabled KI specific feedback. Likewise, for triangulation, the evaluation team also collected supplementary data from external actors (e.g., opinion leaders, public sector officials, and peer implementer/development partner staff) who 2 offered an informed vantage point regarding the project implementation context, and specific factors affecting intended outcomes. SAMPLING AND LIMITATIONS In terms of the characteristics of the sample reached by the data collection activities, the evaluation team sought to cover a balanced sample but also consider specific social, cultural and demographic aspects that are contained in the geographic coverage covered by PERFORM. Sampling tracked the scope and nature of the activities undertaken, where the selection of sites and KIs was based on information provided by PERFORM, and included variable numbers of participants when using a group format. English language literacy hindered the application of mini-surveys or questionnaires, limiting some of the triangulation across the field visits. Specific biases referring to recall, response and selection were noted and considered in the analysis of the data. Other limitations were related to the provision of project documents less than a week before the MTE team traveled to Malawi, and technical project documents were only obtained in country. This limited insights that would have been useful prior to commencing the in-country research. TOC COMPARISONS, PMEP TARGETS and DATA QUALITY Based on the feedback provided and responsive to a subsequent request by reviewers that included contributors from USAID/Washington, IBTCI has incorporated an additional dimension to the final structure of the evaluation report. This added dimension is offered in the form of analytical tables (Annex 9) that examine the implications of the evidence collected to PERFORM’s theory of change, for each of the evaluation questions. Moreover, in furthering the utilization of data collected by PERFORM, a remaining challenge is the relevance and usefulness of data collected under broader indicators (reported annually), which may have limited applicability in addressing specific (possibly junctural) challenges, as well as emerging opportunities. In this respect, planned activities such as a learning survey need to carefully consider the knowledge gaps which may have emerged in the course of the first half of the LOP. Across the six indicators the main drivers of change are capacity building and a critical facilitation role that is consistent with the five outcomes and associated sub-outcomes of PERFORM’s results framework. Evidence, for example, presented in the sections to follow that relates to Indicators 1.1, 1.2 and 3.2, supports the advancement of PERFORM Outcomes (1 and 2) and is also directly relevant to the noted crosscutting result on policy and systems strengthened. Likewise, evidence linked to additional resources is supported by target achievements under indicators 4.1 and 5.2 which tracks USAID/Malawi CDCS priorities for purposeful alignment and overlap of interventions, partnerships and advancement of its 3-C strategy. FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS EVALUATION QUESTION 1A – FINDINGS PERFORM worked with EAD and multiple sectoral institutions, and ably built upon the foundation of past GHG inventory efforts in Malawi, which had been supported by various development partners. The GOM REDD+ Action Plan 2014 – 2019 (MRPAP) was developed with support from the USAID Malawi REDD+ Readiness Program (MRRP), in which USFS IP was an important partner, and the United Nations Programme for Reducing Emissions from Deforestation and Forest Degradation in Developing Countries (UN-REDD). Past support from USFS was praised as a catalyst in the process. PERFORM has built upon these prior efforts and has moved REDD+ readiness in Malawi forward through support for local scale forest inventories and carbon stock assessments, capacity building at DoF, operationalizing the MRPAP, and improving policy and legal frameworks. KIs indicated that the GOM has been regarded as slow to develop what they referred to as the REDD+ action plan (already completed, it is likely that they meant the REDD+ strategy). PERFORM is working with the RExG to accelerate the timeline for the development of the National REDD+ Strategy. While some KIs have suggested ministries hiring their own consultants to contribute to these processes is less preferable than building the capacity of GOM 3 staff to do so, the MTE team noted clear evidence that PERFORM has indeed made great efforts to build capacity through trainings for EAD, CEPA, and DoF; as well as supporting advanced degrees for DoF personnel. The project supported EAD through an INDC options assessment, and on determining the investments needed to support different GHG mitigation scenarios. The project also supported GOM staff within the different sectoral ministries working on the AFOLU elements. An achievement of the project was its building of a partnership around the INDC consisting of GOM, academia, the private sector, civil society organizations (CSO), and development partners (DPs). EVALUATION KEY TAKEAWAYS PERFORM has built upon prior efforts and has moved REDD+ readiness in Malawi forward through support for local scale forest inventories and carbon stock assessments, capacity building at DoF, operationalizing the MRPAP, and improving policy and legal frameworks. The GOM has been regarded as slow to develop what they referred to as the REDD+ action plan (REDD strategy). PERFORM is working with RExG to accelerate the timeline for the development of the National REDD+ Strategy. Creating an opportunity to "affect positive transformational change in the forestry and natural resource management sector" will require a greater commitment from the GoM in significantly advancing its preparatory work. The main (and important) contribution of the PERFORM’s agricultural interventions have been to increase food security, which helped to reduce the negative effects from 2015 El Niño drought. There is insufficient evidence to show that the income obtained from these activities could replace the potential income from illegal charcoal sales. The KHP Corymbria out-grower scheme shows the greatest promise as a livelihood scheme. The introduction of cookstoves has been greatly successful, with significant penetration and replication potential, reducing the need for fuel wood, with implications for decreased pressure on surrounding forests. Implementation on the ground evidences important efforts to effectively integrate with other USAID￾funded activities, and engage in substantive collaboration with other DPs. Field visits confirmed, for example, work done in concert with activities such as FtF MISST and CIP. EVALUATION QUESTION 1A – CONCLUSIONS The project has contributed significantly to the GHG Inventory, MRPAP, and INDC. In addition, building upon its GHG Inventory support activities, PERFORM has also supported the Third National Communication to the UNFCCC. All evidence indicates that the level of progress has been good. However, at a more detailed level of analysis, GOM institutions beyond EAD and DOF could benefit from technical support related to the analysis and use of land use data (e.g., agricultural sector). Some concerns were expressed regarding the use of external consultants versus GOM staff, but it was unclear to the MTE team whether or not these concerns were more related to GOM staff attempting to secure development resources. Ample evidence suggests that the project is using a wide range of well accepted capacity development strategies. The project and DOF clarifying the difference between the MRPAP and REDD+ National Strategy will be key to ensuring that stakeholder expectations and the GOM “no￾regrets” approach to REDD+ are well aligned. EVALUATION QUESTION 1A – RECOMMENDATIONS To avoid misunderstanding and frustration among those stakeholders who are mainly interested in the revenue generation aspects of REDD+ projects, PERFORM, DoF, and other GOM counterparts should clarify for stakeholders the difference between the MRPAP and the REDD+ national strategy, and its implications for eventual REDD+ projects. To ensure a similar level of technical capacity, the project should consider expanding technical capacity building related to obtaining and analyzing forest data (such as RS/GIS) to other relevant GOM institutions. This will also ensure that policymakers can understand and effectively use geospatial data products, and better operationalize REDD+ policy across GOM. 4 EVALUATION QUESTION 1B – FINDINGS The project’s efforts at the district and local scale can be divided into a package of activities aimed at benefiting beneficiary villages; as well as the forestry inventories and C stock assessments. The village level activities, addressing local drivers of deforestation and enhancing rural livelihoods in the districts of Mzimba, Ntchisi, and Machinga, focus on villages adjacent to protected areas, with improved seeds, and new varieties. While most of the crops provided were already being grown by the farmers before PERFORM interventions, the project provided improved varieties with higher yield and/or early maturing times. Evidence indicated that the excess production was mostly used for consumption, perhaps related to the 2015 drought. The project used village farmer clubs as an entry point, the members of which appear to overlap closely with the VRRMCs. Improved cookstoves were another important local intervention, with penetration rates of 70 to 100% in the target villages visited. Woodlots introduced to further reduce pressure on surrounding forests; those visited by the MTE team showed slow growth, possibly attributable to the drought associated with the 2015 El Niño. In terms of conservation agriculture, beneficiaries mentioned having been introduced by the project to practices for forest conservation, alternating crops with trees, and water/soil conservation. The efforts through project grantee Kawandama Hills Plantations (KHP) includes sustainable charcoal as well as alternative livelihoods elements (Corymbria out-grower scheme), through a unique and successful private sector partnership. Another element of PERFORM local level work was the forest inventory and carbon stock assessment efforts, which were highly lauded at all levels of government. EVALUATION QUESTION 1B – CONCLUSIONS The pilot projects encompass agriculture, energy/forestry, livelihood activities; as well as local forestry inventories and C stock assessments. Their support for national climate change and REDD+ goals is mixed. The main (and important) contribution of the agricultural interventions has been to increase food security, especially in ameliorating the negative impacts of the 2015 El Niño drought. While some excess crop production is sold in local markets, no evidence was obtained to show that the income obtained could replace the potential income from illegal charcoal sale. In terms of meaningful livelihood benefits, the KHP Corymbria out-grower scheme shows the greatest promise. The introduction of cookstoves has been greatly successful, with significant penetration and replication potential, reducing the need for fuel wood, with implications for decreased pressure on surrounding forests. The trees in the woodlots visited by the MTE team appeared stunted, potentially related to the 2015 drought, and current unavailability of water, and it is difficult to see how they could contribute to providing fuel wood in any meaningful way within the life of the project. The local forestry inventories and carbon stock assessments have been extremely successful, and recognized as such by DFO as well as national level GOM officials. Their contributions in terms of data, methodology, analysis, and outputs have been recognized, as well as their development of standard operating procedures. These are already influencing national level processes related to forestry, as well as significantly advancing GOM climate change and REDD+ goals. EVALUATION QUESTION 1B – RECOMMENDATIONS To increase cash income for farmers, the project should continue its efforts to increase the production of marketable commodities. PERFORM should consider significantly ramping up planned efforts to pool farmer production and sell in greater volume through local farmer organizations. The project should consider expanding the use of the KHP Corymbria out-grower scheme model to other high value cash crops through private sector partners. EVALUATION QUESTION 2 – FINDINGS PERFORM’s efforts to scale up REDD+ readiness include support to the RExG to draft the National REDD+ Strategy, as well as supporting the National Charcoal Strategy and the National Forest Landscape Restoration Assessment and Strategy (NFLRS). The project supported the launch of the 5 NFMS Roadmap in 2015, and has improved stakeholder capacity to monitor and analyze land use and land cover in Malawi. The project has provided GIS/RS training for the DoF and other stakeholders. Additional efforts to scale up REDD+ in Malawi through broader and deeper technical expertise include awarding an in-kind grant for a future GIS lab at the College of Forestry and Wildlife (Dedza). Both PERFORM and DoF staff members have lectured at the College of Forestry and Wildlife, as well as at other academic institutions such as MZUNI and LUANAR. With respect to the data management aspects of scaling up REDD+, some stakeholders suggested that open data platforms (e.g., MASDAP) can enhance the use of the data and products. At the district level, LAMIS may be a possibility, while noting the challenges related to capacity and paper records. EVALUATION QUESTION 2 – CONCLUSIONS PERFORM has been successful in contributing to multiple aspects of REDD+ readiness, through activities such as supporting the NFMS, National Charcoal Strategy, and NFLRS. The project has actively engaged RExG and its constituent bodies, and has effectively used it as a venue for outreach and ensuring buy-in from a wide range of stakeholders, including GOM, academia, civil society, and the media. Through lecturing at academic institutions, the project is helping ensure a future cadre of trained experts in REDD+ related issues. The ongoing support towards the development of a GIS Lab at the College of Wildlife and Forestry (Dedza) further advances the goal of institutionalizing sustainable capacity development. The storage and management of forestry data could be improved, but is subject to reliable electricity and internet bandwidth limitations. EVALUATION QUESTION 2 – RECOMMENDATIONS Notwithstanding limitations related to stable electricity provision and internet connectivity, the project should consider an assessment of the current data handling and sharing infrastructure at DOF and FRIM as key REDD+ related institutions. Identifying an effective and strategically positioned data repository could provide a more sustainable and consistent source of updated data to support REDD+ scaling. The connectivity needed for the transfer of large quantities of data will be a challenge to these efforts. EVALUATION QUESTION 3 – FINDINGS Evidence indicates the importance of project support to the GHG inventory, and its appreciation by KIs, reflecting its value as a fundamental requirement of the UNFCCC. This points to expected sustainable progress in meeting additional commitments, stemming from the enhanced capabilities strategically developed inside the EAD. KIs also pointed to multi-donor funding that overlaps the project’s goals, such as support from the International Energy Agency and GIZ. Discussions with senior project staff suggested that watershed management will be incorporated into the National Resilience Plan. The project has provided hands-on trainings on technologies introduced to beneficiaries at the local level. Collaboration with FtF MISST, and CIP ensure a degree of continuation of some of the agricultural interventions such as OFSP and improved legume varieties. EVALUATION QUESTION 3 – CONCLUSIONS There are two drivers to the sustainability of national level efforts, one institutional, and the other financial. With respect to the former, obligations under the UNFCCC will require the updating of GHG inventories (for inclusion in the national communications), as will the INDC. This reflects a soft sustainability, essentially meaning that the GOM will have to continue these efforts using its own or donor resources. The other driver is financial, and some KIs reported that some of the national level efforts could receive financing via MGDS prioritization, noting that forestry was a GOM priority. Another source of intermediate sustainability would be other bi- and multi-lateral donors supporting the national level efforts. At the local level, cookstoves appeared to be the most enduring technology, and replication of crops such as OFSP show some promise of sustainability, as does the chicken pass-on program. Based on the evidence obtained, it is unlikely that woodlots will continue, as tree growth was 6 limited, and it is difficult to conceive that farmers would regularly dedicate the time and water resources to irrigate the woodlot trees. EVALUATION QUESTION 3 – RECOMMENDATIONS The project should continue its efforts to introduce cookstoves and build capacity on their construction. The project should consider strengthening linkages to the National Technical Committee on Climate Change (NTCCC) to further influence national priorities;1 and the District Environmental Sub-Committee (DESC) of the DCs, to enhance collaboration with other projects, incorporate lessons learned into the District Development Planning Process, and take advantage of complementary financial opportunities for target communities. Lastly, the project should explore market-linkages for smallholder farmers in target communities to stimulate further adoption of current and future technologies introduced. EVALUATION QUESTION 4 – FINDINGS Reports from PERFORM IPs and project documents indicate close collaboration with the German Federal Ministries of Economic Cooperation and Development (BMZ), the German Federal Ministry for the Environment, Nature Conservation and Nuclear Safety (BMU), and DFID towards the National Forest Landscape Assessment. Furthermore, the reports indicated strong integration with a number of USAID funded activities in the Machinga District (e.g. FtF MISST and NJIRA). Regarding other DPs, the evaluation team noted that the project coordinated with the Shire River Basin Management Program (SRBMP) in Machinga District, working through the DOF to avoid the duplication of effort. The evaluation team found evidence of more opportunities for integration with DPs in the target areas and nationally on climate change and forest conservation. EVALUATION QUESTION 4 – CONCLUSIONS Implementation on the ground evidences important efforts to effectively integrate with other USAID￾funded activities, and engage in substantive collaboration with other DPs. Field visits confirmed, for example, work done in concert with activities such as FtF MISST and CIP. This is also the case with other DPs in the context of the NFLRS and INDC. Likewise, intentional avoidance of duplication of efforts was observed with the World Bank SRBMP in Machinga. Future opportunities with other DPs could include JICA DFR, currently in its start-up phase. EVALUATION QUESTION 4 – RECOMMENDATIONS The project should continue efforts with the DCs to explore opportunities for collaboration with other projects (e.g. MCA’s Circle for Community Development (CICOD) and World Bank’s MASAF IV) to improve outcomes and ensure sustainability. EVALUATION QUESTION 5 – FINDINGS Evidence obtained suggested that women were fully integrated into national level processes. While their numbers relative to men remain small, this is related to their current distribution within GOM institutions. Some women are in leadership positions related to REDD+ readiness. At the local scale, all pilot sites visited by the evaluation team showed high numbers of women participants (often above 90%), with women clearly placed in leadership positions in terms of the group’s productive activities, as well as in organizational roles and some decision-making functions. Women directly benefit from local scale activities, and referred to concrete household benefits availed through additional cash that was directly derived from income-generation activities supported by PERFORM. However, decisions regarding the use of excess cash income are ultimately made by men. EVALUATION QUESTION 5 – CONCLUSIONS                                                              1 The infrequent convening of this body and the members’ requirement for sitting fees will remain a challenge. 7 At the level of participation and engagement both nationally and especially locally, women are (at local level, usually over 80%) represented in project activities. Notwithstanding their small numbers relative to men, reflective of GOM staffing, at national level, some women drive key elements of REDD+ readiness. At the local level, regardless of the intensive participation of women both in farmer clubs, and in the actual implementation of project activities within the villages; and irrespective of whether the societies were matrilineal or patrilineal, discussions with farmer groups, and representatives of GVHs, indicated that decisions regarding cash income and land use were strongly influenced or made exclusively by men. As such, while women have been able to participate in income-generating activities, at the household level, their ability to benefit from them is controlled by men. As discussed in the appropriate sections above, capacity building has made significant contributions towards the national level goals. At the local level, women’s labor is contributing towards the implementation of the local scale activities, but, as discussed above in question 1b, the main impact of these has been on improving food security. EVALUATION QUESTION 5 – RECOMMENDATIONS The project should consider leveraging existing savings & loans groups, particularly those which have linkages to business interests and investments, to introduce new incentives/expansions to its initial income generation activities. ADDITIONAL RECOMMENDATION ON DATA QUALITY AND USE Given the limited applicability of data collected under broader standard indicators in addressing specific (possibly junctural) challenges, as well as emerging opportunities, planned PERFORM activities such as a learning survey need to carefully consider the knowledge gaps which may have emerged in the course of the first half of the LOP. Moreover, learning surveys should be contextualized against USAID’s broader framework for “Collaborate, Learn and Adapt” (CLA / Learning Lab), as well as provide clearly identified opportunities to contribute to learning across the USAID/Malawi portfolio. 8 EVALUATION PURPOSE & EVALUATION QUESTIONS EVALUATION PURPOSE The primary purpose of this mid-term performance evaluation is to assess the effectiveness of the PERFORM design and implementation processes. The evaluation will also inform management of course corrections, if any, are needed. Additionally, through this evaluation, USAID seeks strategic and programmatic options for future engagement to support climate change mitigation and adaptation in Malawi. Major objectives of this evaluation are to: 1. Review, analyze, and evaluate the effectiveness of the PERFORM activity in achieving program objectives and contributing to USAID/Malawi’s efforts on climate change mitigation and adaptation in Malawi. 2. Evaluate major constraints in achieving expected project results. 3. Provide specific recommendations and lessons learned on strategies and approaches USAID/Malawi should continue in its future climate change mitigation activities. The final performance evaluation of PERFORM should, inter alia, determine whether the findings, recommendations, and lessons learned from the mid-term evaluation were taken into account during the remainder of the implementation period, and if so, what the impact was of any course corrections. EVALUATION QUESTIONS 1a. How effectively has PERFORM supported the GOM’s national climate change goals including the development of a GHG inventory, implementing its National REDD+ Action Plan, and developing and implementing its Intended Nationally Determined Contribution (INDC) under the UN Framework Convention on Climate Change? 1b. Do the pilot projects at the local level effectively support the national goals, and are they achieving meaningful livelihood benefits at the site level? 2. What lessons and implications does PERFORM offer for REDD+ readiness scaling up and likely impacts on REDD+ outcomes? This can also include a synthesis of lessons from the REDD+ Readiness Program. 3. How sustainable are the efforts at the local and national level? Are GOM officials and local communities likely to be able to continue these programs on their own when PERFORM ends? 4. How well is PERFORM integrated with: a) other USAID initiatives (e.g., Feed the Future): and b) other DPs efforts in GCC and forest conservation? 5. How are gender issues integrated into the activity? Have women been able to participate fully and benefit from it, including from income-generating activities? For cross cutting subjects such as gender and capacity building, how effectively do they contribute to PERFORM’s intended results? The midterm performance evaluation leveraged three distinct stages to examine the PERFORM activity and its intended outcomes, seeking to draw evidence of achievement and non-achievement from both 9 primary and secondary sources, including the direct beneficiaries of project interventions, and key stakeholders. These three technical stages of the process are: (1) A review of secondary (document) sources and initial fieldwork preparations: (2) the implementation of all In-country data collection and preliminary data analysis activities and; (3) the production of Final Evaluation Report. Across all three stages, the evaluation team focused on the specific contribution of PERFORM to serving the stated theory of change, without a deliberate attempt to demonstrate attribution to longer-term outcomes at this midpoint stage of its implementation. In conducting the research among village beneficiaries, the team’s data collection and analysis considered differences in land availability, agroecology, and gender roles (patrilineal vs. matrilineal societies). The evaluation team also collected supplementary data from external actors (e.g., opinion leaders, public sector officials, and peer implementer/development partner staff) who were able to offer an informed vantage point regarding the project implementation context, and specific factors affecting intended outcomes. Some key informants (KIs) were selected in light of their inherent linkages to the intended or achieved outcomes. Group interviews were used as means for triangulation, as well as opportunities for peer-to-peer interaction and open reflection on intended outcomes (whether present or future). In total, approximately 110 individuals contributed information to this report. In interviews with village farmer clubs, involving large numbers of individuals, each interview has been counted as a single entry, where the total number of actual participants is approximately 50. PROJECT BACKGROUND USAID’s Protecting Ecosystems and Restoring Forests in Malawi (referred to hereafter as PERFORM) was specifically designed to align with Malawi’s Growth and Development Strategy (MGDS) and to promote forest conservation and green growth. It is the main implementation vehicle for the low￾emissions partnership between the U.S. and GOM. The project period is 5 years, from 2 September 2014 to 1 September 2019. PERFORM is one of the key activities under “Development Objective 2 (DO2): Sustainable Livelihoods Increased” of USAID Malawi’s Country Development Cooperative Strategy (CDCS). PERFORM has completed approximately 2.5 years; therefore, USAID wishes to evaluate the performance of the project at the mid-point of its five-year program. Consistent with USAID’s 2011 Evaluation Policy and its more recent 2016 updates, the primary goal of this mid-term evaluation is to provide evidence so that USAID/Malawi can determine whether PERFORM is on track to achieve its desired results or not. Other goals of the evaluation are to inform what course corrections, if any, are needed, identifying specific project interventions that can be scaled up further and to determine best ways to ensure sustainability of the activities, institutions and capacities promoted by the project. Finally, this evaluation will provide strategic options for future USAID/Malawi engagement in climate change and offer lessons learned with wider applicability in Malawi or beyond. 10 EVALUATION DESIGN, METHODS & LIMITATIONS The midterm performance evaluation leveraged three distinct stages to examine the PERFORM activity and its intended outcomes, seeking to draw evidence of achievement and non-achievement from both primary and secondary sources, including the direct beneficiaries of project interventions, and key stakeholders. The evaluation team focused on the specific contribution of PERFORM to serving the stated theory of change, without a deliberate attempt to demonstrate attribution to longer-term outcomes at this midpoint stage of its implementation. Table 1: Evaluation Approach: Challenges/Opportunities First Stage Second Stage Third Stage Review of secondary (document) sources In-country data collection and preliminary data analysis Production of Final Evaluation Report 1. Initial review of context/program documents; 2. Document identification and review; 3. In-brief with USAID and kickoff with PERFORM; 4. Identification/contacting of evaluation sources and KIs; 5. Finalized schedule of visits; 6. Finalized data collection tools. 1. Field observation and data collection; 2. Ongoing data integration and preliminary analysis; 3. Data triangulation and final facts-checking; 4. Debrief to discuss preliminary findings and enable feedback; 5. Production of draft report; 6. Final in-country debriefing. 1. Final data analysis and final drafting of findings, conclusions and recommendations; 2. Production of draft report; 3. Review, comment and approval of draft report. Challenges/Opportunities in Implementation  Secondary sources and background documents originally identified had to be expanded to cover added inputs to the team’s understanding of both context and process.  Field visit schedule was significantly reduced due to unforeseen logistic hindrances;  Literacy and uncontrolled bias (field locations), as well as challenges in ensuring attendance of key participants.  Draft and final report production will benefit from advance/added comments and feedback included in the joint discussion of preliminary findings with IP participants. The evaluation team gathered, analyzed, and assessed a series of documents to contextualize the evaluation, and to gain as complete a description as possible of each project’s inputs, outputs, and outcomes. See Annex 6 for a list of documents reviewed. Key informants engaged by the evaluation constituted a purposive sample. The evaluation team consulted project implementer staff to identify and prioritize data sources, as well as opportunities or constraints affecting each source. KIs included implementer staff, local partner/counterpart organization staff, primary project beneficiaries, key GOM and other stakeholders, and donor representatives. The evaluation team collected supplementary data from external actors (e.g., opinion leaders, public sector officials, and peer implementer/development partner staff) who were able to offer an informed vantage point regarding the project implementation context, and specific factors affecting intended outcomes. Some KIs were selected in light of their inherent linkages to the intended or achieved outcomes. Overall, the mid-term evaluation of the PERFORM activity relied on primary sources that were cross-referenced with applicable secondary sources (Figure 1) to determine the achievement or non-achievement of its intended outcomes, while closely examining the quality and reliability of the data collected to date. In total, approximately 110 individuals contributed information to this report. In interviews with village farmer clubs, involving large numbers of individuals, each interview has been counted as a single entry, where the total number of actual participants is approximately 50. In conducting the research among village beneficiaries, the team’s 11 data collection and analysis considered differences in land availability, agroecology, and gender roles (patrilineal vs. matrilineal societies). Key Informant Interviews (KIIs) remained the core data source used to assessing progress, with a selected number of group interviews. Data from individual or group interviews was collected by utilizing semi-structured interview protocols, seeking greater flexibility in question/topic emphasis. Whenever possible and appropriate, direct field observation of ongoing activities was incorporated in tandem with individual and group interviews conducted for selected sites. Despite the team’s initial proposed use of focus group discussions (FGD), challenges in convening the right mix and relevance of participants eventually prevented their inclusion in the data collection activities. Although the main focus of most data collection activities was the evaluation questions formulated by USAID and included in the SOW, activities such as the group interviews were used as means of triangulation, as well as opportunities for peer-to-peer interaction and open reflection on intended outcomes (whether present or future). In this respect, group interviews were primarily intended to provide important feedback from the perspective of service providers, partners, and direct beneficiaries. DATA COLLECTION METHODS The evaluation team developed a standardized format to register the information collected across all interviews, directly coding all information relating to the evaluation questions and sub-questions from the semi-structured questionnaire by sub-question and thematic keywords. The information was compiled on a daily basis in an Excel matrix (Annex 5) that assisted the targeted review of coverage and the identification of information gaps. The approach of systematic ongoing compilation also was instrumental to enabling the Team Leader to track and follow, often remotely, the progress and depth of data collection activities, while attending to a parallel and separate USAID evaluation. Data drawn from a set of additional interviews (mainly context or topic-driven through a targeted approach) was enabled by splitting the team into a Lilongwe versus target beneficiary sites, has also been incorporated into the review and analysis of project reported data. In the case of issues/topics instrumental to gender gap analysis, the evaluation team aimed to identify, understand, and explain gaps between males and females, identify the relevance of gender norms and Secondary Sources Context, IP data compilation & reporting and useful technical references) throughout the evaluation process Key Informant Interviews & limited Group Interviews were conducted in both Lilongwe and site contexts FIGURE 1: METHODS Analysis & Triangulation  Identify information gaps (as per evaluation questions);  Conduct reference/expert interviews;  Consider keyword distribution of responses 12 power relations specific to the PERFORM implementation context(s). Wherever feasible and useful, self￾assessment approaches were used to draw additional feedback from KIs; however, the team decided to not structure this approach as a separate tool of segregate its use from the main interview process. Data analysis was also aimed to highlight any gender-specific practices that the PERFORM may have adopted. Any particularly useful data on PERFORM approaches has been highlighted to assist recommendations that may best support reflection on progress, after-action review and informed adaptation, extensive to future USAID program design. FIGURE 2. KEY INFORMANT NUMBER AND DISTRIBUTION OF INDIVIDUALS INTERVIEWED DATA COLLECTION TOOLS AND ADJUSTMENT TO CONTEXT The mid-term evaluation tools used to collect and analyze information from beneficiaries were developed to establish useful linkages between responses and the main evaluation questions. This core focus of the research on the evaluation questions was primarily supported by the semi-structured individual interview protocols used in a one-on-one setting (see Annex 4). A standardized template associated to the main research questions (SOW), and a predetermined set of keywords enabled the team to code responses on an ongoing basis. As noted, a compilation matrix assisted a targeted (question specific) review of responses, where gaps identified were supplemented by available secondary sources. Category Number of Individuals Interviewed Academia 2 Agriculture 2 DP 6 Energy 2 Forestry 10 Grantee 1 IP 5 Natural Resources 3 NGO 2 Parks 2 Planning 2 Reps. Of GVHs 5 Village Farmer Clubs* 3 Total 45 * Village Farmer Clubs counted as entities, not individuals, to avoid skewing the distribution of KIs. In total, the team contacted approximately 120 individuals (including approximately 75 village farmer club members). Academia 5% Agriculture 5% DP 13% Energy 5% Forestry 22% Grantee 2% IP 11% Natural Resources 7% NGO 4% Parks 4% Planning 4% Reps. Of GVHs 11% Village Farmer Clubs* 7% DISTRIBUTION OF INDIVIDUALS INTERVIEWED 13 However, the originally proposed semi-structured approach for the Group Interviews and the related use of questionnaires was not deemed feasible. Assessed hindrances on bias control and response contamination stemming from low English language literacy prevented the use of the mini-survey questionnaires previously developed by the evaluation team. Instead, the team made a decision, to verbally discuss the mini-survey questions with the group, thus eliminating any differences in participation related to English language literacy. As previously noted, although the team developed protocols and guidelines to conduct focus group discussions, their effective implementation was not considered feasible or reliable. Nonetheless, a limited set of reference interviews with expert informants, mostly associated with other development partners working in Malawi provided useful overarching feedback on both, progress to date and opportunities for the remaining implementation. In addition, although the team expected to have access to supporting data beyond reported PMEP values, the review of the PMEP provided by PERFORM could not be supplemented with sample data sets, typically not intended for testing, but for better rendering observed patterns that can enrich the team’s understanding data flows, or support limited clustering to flag data quality observations and/or support possible corrections or enhancements. In applying a more limited set of tools, the evaluation team has adopted a more conservative approach that translates into a more limited cross-site analysis. PMEP￾supported target accruals and annual reporting were the only secondary source for comparisons with observed field progress. SAMPLING In terms of the characteristics of the sample reached by the data collection activities, the evaluation team sought to cover a balanced sample but also consider specific social, cultural and demographic aspects that are contained in the geographic coverage covered by PERFORM. See Annex 5 for lists of the KIs interviewed. Sampling tracked the scope and nature of the activities undertaken by PERFORM with variable numbers of participants when using a group format. The selection of sites and KIs was based on information provided by PERFORM, including the team’s review of pilot site characteristics, annually reported target accruals, considerations of possible biases and other limitations that could skew the data collected. Associated logistics and deployment distances were an important factor in the final selection. See Annex 5 for a map of the locations where team members completed data collection. LIMITATIONS Data Availability. The evaluation team encountered some limitations related to data availability. None of the sites visited or KIIs enabled additional quantitative data or access to locally kept records. In addition, hindrances related to English language literacy hindered the application of mini-surveys or questionnaires limited the expected triangulation across the scheduled field visits. Despite multiple efforts to reach some KIs, the evaluation team was not always successful in interviewing some individuals (particularly in Lilongwe), translating into time lags or, in some cases, unavailability. Nonetheless, based on the large number of individuals/groups (50 or 110 including individual farm club members, as explained above), the evaluation team believes that its findings accurately reflect the focus of PERFORM￾supported activities. Recall Bias.The evaluation team anticipated that it would encounter some recall bias, given the two￾year plus period elapse and some instances of turnover in public sector counterparts. Although KIs consistently make a clear distinction between PERFORM activities and previous non-USAID implementation, there is natural overlap in the cause and effect recall, particularly with actors at the higher levels of central government. Response Bias. Some inputs offered by key informants may have been affected by response bias. Some KIs, predominantly at the village level, seemed to deliberately avoid direct criticism or controversy 14 regarding project implementation. Conflict or opposing views involving different institutional groups, personalities, and special interests also influenced the responses offered by primary sources. This trend appears to reflect the fairly common concern among stakeholders about sustaining donor funding. While such perspective could have impacted some of the feedback offered, the evaluation team found that public sector officials were able to point to specific project outcomes that provide a legitimate basis for their assessment of PERFORM interventions. The field visits also used numerous questioning techniques and opportunities to view interventions, to triangulate and confirm overly optimistic feedback (e.g., “everything provided by the project has worked perfectly”). Selection Bias. Selection bias is an inherent risk when implementers help to facilitate contact with project beneficiaries, as they may select the most active, responsive, or engaged beneficiaries—meaning that the evaluation team may only hear from KIs who report positive experiences. Typically, an evaluation team could observe and speak with a wider range of beneficiaries by attending project activities or using snowball sampling during the fieldwork period—asking KIs to themselves identify other potential KIs in their network. To mitigate selection bias, the evaluation team attempted to filter responses to interview protocol questions to avoid evident slant, as well as, to the extent possible, triangulated responses collected from various data sources. BRIEF OVERVIEW OF PMEP TARGETS AND DATA QUALITY In assessing PERFORM’s advancement at its mid-term point, the evaluation team has endeavored to make the most feasible and useful comparisons between progress reported (PMEP-based tracking) and on-the–ground observations where USAID-funded outputs were examined utilizing semi-structured interviews with target beneficiaries and other key stakeholders. Consistent with USAID’s own internally evolving process (ADS 201) performance indicators should assist the measurement of outcomes that are relevant and significant. Accordingly, at the activity level we look for relevance reflected in their direct link to the activities specific logic (theory of change) and conclude on their significance based on their contribution to a chain of results that is consistent with its purpose. Supported by this rationale, the evaluation team chose to highlight on a subset of six indicators that best relate to the findings presented in this report and also appear most useful when examining the contribution story under each of the evaluation questions, as described in the previous section. Accordingly, Table 2 provides a useful summary of the selected indicators, the measured change as reported in the PERFORM PMEP (2016) and the implications that more readily relate to the theory of change (TOC) as presented in Annex 9. In this respect, across the six indicators the main drivers of change are capacity building and a critical facilitation role that is consistent with the five outcomes and associated sub-outcomes of PERFORM’s results framework. Evidence, for example, presented in the sections to follow that relates to Indicators 1.1, 1.2 and 3.2, supports the advancement of PERFORM Outcomes (1 and 2) and is also directly relevant to the noted crosscutting result on policy and systems strengthened. Likewise, evidence linked to additional resources is supported by target achievements under indicators 4.1 and 5.2 which tracks USAID/Malawi CDCS priorities for purposeful alignment and overlap of interventions, partnerships and advancement of its 3-C strategy. Positive differences registered under all six indicators also readily associate to specific positive gains/achievements noted in the TOC comparisons included in Annex 9. 15 Table 2: Performance Indicators relevant to the TOC Comparisons PERFORM has addressed the recommendations of the 2016 Data Quality Assessment regarding deficiencies identified, which the technical staff are reportedly advancing on the ground. The practice instated of developing “Memorandums to the File,” further guides the use of data and enables a sound record of progress where tangible evidence can be more difficult to collect otherwise. In addition, beyond enhancements to its data capture instrumentation and protocols, paired with appropriately adjusted triangulation, PERFORM is also considering improvements to the use of data in the context of its learning activities. This task is not only consistent with USAID guidelines and policy, but it constitutes a natural extension of the capacity and presence that this activity has already established on the ground. In furthering the utilization of data collected by PERFORM, a remaining challenge is the relevance and usefulness of data collected under broader indicators (reported annually), which may have limited applicability in addressing specific (possibly junctural) challenges, as well as emerging opportunities. In this respect, planned activities such as a learning survey need to carefully consider the knowledge gaps                                                              2 Only national level is considered here, since no evidence was observed for local scale (Target 6, Result 7) Indicator Target 2016 Result 2016 Change Comparison to PMEP MTE Observations 1.1 Number of institutions with improved capacity to address sustainable landscapes issues as supported by USG assistance (EG.13-2) 8 10 Target exceeded by 25% Evidence for capacity building noted for DOF, EAD, CEPA, RExG and constituent bodies. 1.2 Number of laws, policies, regulations, or standards addressing sustainable landscapes formally proposed, adopted, or implemented as supported by USG assistance (EG.13-3) 22 3 Target exceeded by 50% Progress noted on GHG inventory, MRPAP, National Charcoal Strategy, NFLRS, and advancing the timeline for the National REDD+ Strategy 2.2 Number of farmers and others who have applied new technologies or management practices as a result of USG assistance (FACTS/FTF 4.5.2-5) 1,500 5,445 Target exceeded by 263% Beneficiaries indicated having been introduced by PERFORM to practices for forest conservation, alternating crops with trees, and water/soil conservation 3.2 Improved quality of GHG inventory (as measured by the EPA’s GHG Inventory System Score card inclusive of USEPA’s Inventory Project Progress Indicator) 1.53 1.52 Target essentially met Evidence suggested support to EAD and the sectoral ministries responsible for data provision, as well as the establishment of MOUs as critical underpinning to advance intended progress 4.1 Number of new USG supported public-private partnerships (PPPs) formed- Output/FACTS (FACTS/PPP 5) 1 2 Target doubled Field observations noted the highly successful KHP Corymbria out￾grower scheme 5.2 Number of discrete integration opportunities successfully pursed and implemented (Custom) 4 6 Target exceeded by 50% Evidence suggested integration with USAID FtF MISST and CIP, as well as BMZ, BMU, and DFID 16 which may have emerged in the course of the first half of the LOP. Different from the straight data paths that are required by traditional monitoring activities, the design of the surveys should consider using a layered approach that helps, for example, to build multiple expansion scenarios based on greater integration that may emerge with other development initiatives. The information layers crafted through learning activities center on a relevant and context-sensitive utilization focus. In this, a clear action framework (that includes specific capacity building among intended users) and a sustained review of the value questions (that specifically address the needs of intended users) will be essential to maximizing its contribution to reflection and adaptation, as per ADS 201 guidance. Data quality, beyond the 2016 DQA recommendations, should be directly linked to pre-identified and emerging user needs, further building on ADS mandated standards (e.g. Validity should focus on a range that covers both existing and potential implementation scenarios). After-action review should therefore benefit not only from an annual accrual or synthesis of strategic layering of knowledge accrued under each targeted outcome, but also from a tangible “commitment to use and expand” emerging knowledge by key stakeholders, which presently is not as evident in the data collected and could be better tracked by PERFORM. APPROACH TO TOC ANALYSIS In addressing the approved SOW and its stated evaluation purpose, the PERFORM midterm evaluation enabled a purposeful, albeit limited, snapshot of what the activity had set out to accomplish through the first half of its LOP. Guided by the evaluation questions posed by USAID, the evaluation team set out to collect data from primary and secondary sources that would contribute to the formulation of useful answers. Subsequent to the in-country data collection phase, the team rapidly organized the evidence gathered under each one of the five questions included in the SOW for initial consideration, both within a discussion of preliminary findings with USAID, which included feedback from the implementing partner, as well as in the more structured format of a draft report that included conclusions and recommendations. Based on the feedback provided and responsive to a subsequent request by reviewers that included contributors from USAID/Washington, IBTCI has incorporated an additional dimension to the final structure of the evaluation report. This added dimension is offered in the form of analytical tables (Annex 9) that examine the implications of the evidence collected to PERFORM’s theory of change, for each of the evaluation questions. The intention is to further enrich the analytical value of the findings and enhance the utilization potential of the team’s conclusions and recommendations. Contribution analysis is useful in situations where the program is not experimental—there is little or no scope for varying how the program is implemented—and the program has been funded on the basis of a theory of change. Mayne, ILAC 2008 17 BRIEF CONTEXT FOR THE CONSIDERATION OF PERFORM’S TOC In general, when introducing questions of cause and effect the fundamental determination of attribution, meaning: can we isolate and estimate accurately the particular contribution of an intervention? Attribution refers to ensuring that causality runs from the intervention to the outcome.3 Like most implementation contexts, PERFORM is implemented in a country and institutional landscape where the accurate determination attribution is not feasible within the parameters and limitations assumed by the evaluation SOW. Hence, contribution analysis provides a much more credible and useful assessment of cause and effect. In addition, while examining the theory of change adopted by PERFORM within the limitations of a retrospective desk review, and whenever applicable and feasible, the evaluation team also considered other external factors that could influence the outcomes, based on reasonable evidence about its contribution. To craft its theory of change, PERFORM initially convened project staff, USAID, partners, and targeted institutions to “build consensus” that would support its inherent logic, congruent with the applicable contractual requirements, its expected results and selected indicators. The resulting identification and consensus on causal linkages between PERFORM’s results and objectives was intended to guide the final development of the activity’s work plans and PMEP4, while identifying causal linkages between activity’s results and objectives. In addition, stakeholders were asked to assist the determination of external factors, positive and negative, which could influence achievement and non-achievement. Risks and critical assumptions were likewise considered assisted by local expert consultation. APPROACH TO TOC COMPARISONS Developed as a final analysis stage, the evaluation team considered useful (albeit limited) comparisons between observed and/or documented outputs and outcomes, and the causal logic that supports                                                              3 Non IE Guidance on Impact Evaluations, Addressing the Attribution Problem, World Bank Group, 2014.   4 USAID/Malawi’s Perform Activity: Performance Monitoring and Evaluation Plan, October 2016. PERFORM THEORY OF CHANGE If capacities of government, private sector, academia, civil society and local villages to analyze, implement and monitor sustainable land management practices, including REDD+; monitoring, reporting, and verification (MRV); and social and environmental safeguards (SES) are improved (Objective 1); and If local and national governance conditions are improved and provide an enabling environment for local and sustainable land management (Objective 1); and If our demonstration pilot sites demonstrate sustainable land management systems based on improved science and technologies that are scalable to the national level (Objective 2); and If economic incentives are efficiently, equitably, and sustainably aligned through REDD+ and SES to support these land management practices at the local and national levels, across stakeholder groups, including women and youth (Objective 3); and If additional resources are leveraged with other investments through public-private partnerships (PPPs), payment for environmental services (PES), co-location, coordination, and collaboration (Objective 4); then - Forests and soils will be managed more efficiently, equitably, and sustainably (PERFORM Goal); and - GHG emissions from land use will be reduced, resilience to climate change will be strengthened, and Malawians’ quality of life will be improved (CDCS Goal). 18 PERFORM’s theory of change. This final consideration was based solely on the available evidence, which was either gathered or reviewed by the team during its in-country data collection activities. Any causality inferred from the available evidence assumes that the key assumptions documented in PERFORM’s annual reporting (reviewed as evidence of implementation) through the first half of its LOP were sound, plausible, and agreed upon by the main stakeholders. Therefore, evidence collected by the evaluation team constitutes a valid (although partial) examination of the expected chain of results. Consistent with the limitations stated in this report, the evaluation team has also assumed that any and all significant contributions associated to outside factors (not directly resulting from PERFORM interventions) have been recognized by KIs and included in periodic project reporting. To the extent possible and applicable, the TOC comparisons offered under each of the five evaluation questions (compiled under Annex 9) track PERFORM’s outcomes and corresponding sub-outcomes identified in the Results Framework addressed in its Performance Monitoring and Evaluation Plan (PMEP). These comparisons reflect on the evidence collected by the evaluation team, following six steps: 1. Examine evidence collected: What evidence is collected directly applies to the occurrence of the various results? 2. Determination apparent links: What are the apparent links of the evidence collected to the theory of change that also need to be assessed? 3. Consider assumptions and risks: What evidence was available on the assumptions and risks behind these links? 4. Consider the strongest links: Which links are best represented by the available evidence, strongly adhering to the logic, and seemed to have wide acceptance among stakeholders? 5. Consider the weakest links: Which links appear weaker as per the available evidence, suggest weaker logic, or less convincing agreement among stakeholders? 6. Consider external factors: What evidence was available about other identified influencing factors and they contribution they made? Following the principles of contribution analysis, two specific cause-effect questions were applied consistently across the five evaluation questions.  The first question centers aspects observed related to coverage and depth: To what extent have the PERFORM outputs caused the observed outcomes?  The second question considers the status of the challenge(s) addressed by (or relevant to5) the TOC drawing from the collected data: Is it reasonable to conclude that PERFORM has made a difference to the problem? The logic snapshot provided in a table format under each evaluation question constitutes the contribution story that the evaluation team has chosen to highlight in each case. As such, each contribution story is intended to complement the analytical summaries that have been included in the main document as a lead into the recommendations offered by the evaluation team. However, it is important to keep in mind                                                              5 Some extrapolations were unavoidable when comparing implementation to the TOC. Gender issues or focus, for example, are not specifically addressed by the TOC. 19 that the relative weight of each story to the overall TOC will vary depending on the end use of the findings and the kinds of decisions that will be based on them. Since the evaluation SOW offered no specific direction in terms of a focus on causality in direct association to the main evaluation questions, the evaluation team chose to address the most representative and practical cross-section of arguments that could be drawn from the evidence collected and reviewed. In practical terms, this means that the assumptions guiding the comparisons to the TOC regarding the nature, extent and difference made by PERFORM’s contributions may have variable relevance and applicability across each of the five evaluation questions included in the approved SOW. 20 FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS BY EVALUATION QUESTION This section discusses the findings from the review of primary information gathered from interviews with KIs, meetings, group discussions, and site visits. In addition, secondary sources have been consulted to augment the primary evidence collected. The Findings are based upon a rigorous analysis and synthesis of the data gathered, which, as described above, was coded according to the evaluation questions, as well as thematic keywords. The data was then analyzed and triangulated against supporting or contradicting evidence. The findings presented in this section solely reflect the objective analysis and synthesis of the evidence collected and reviewed by the evaluation team. The Conclusions reflect the MTE team’s analysis and judgements of the finding(s), in order to best apply them towards answering as explicitly as possible the evaluation questions. Each set of conclusions includes a “wrap-up” paragraph which explicitly links the more specific individual conclusions to the evaluation questions. This wrap-up is then tied to the theory of change and log frame by an adjacent text box. Finally, Recommendations includes a set of proposed actions that can be taken to further improve the project’s attainment of its goals. The findings, conclusions, and recommendations are organized by the five evaluation questions defined in the evaluation’s Scope of Work. EVALUATION QUESTION IA – How effectively has PERFORM supported the GOM’s national climate change goals including the development of a GHG inventory, implementing its National REDD+ Action Plan, and developing and implementing its Intended Nationally Determined Contribution (INDC) under the UN Framework Convention on Climate Change? FINDINGS General Observations on National Level Support. A frequent observation of KIs was that the ministries contributing to the national level processes (GHG Inventory, REDD+, and INDC) tended to hire their own consultants. PERFORM building capacity for these ministries own staff to do so themselves was seen to be a preferential model. Noting this observation from the KIs, evidence from project documentation suggests that the project has indeed made efforts to build both short and long￾term capacity through support and trainings for EAD (e.g., all-sector, and sector-specific GHG inventory workshops for EAD and partner GOM institutions), CEPA (e.g., advocacy and communications strategies, and E-Information and Knowledge Management or E-IKM Platform), and DoF (for the NFMS – satellite land monitoring system, field-based forest inventory, including standard operating procedures for site level inventories); and has supported advanced academic degrees for four DoF personnel. GHG Inventory. PERFORM has ably built upon a foundation of GHG inventory efforts in Malawi, helping EAD establish MOUs with various sectoral institutions. Since the preparation of the Initial National Communications (INC) to the UNFCCC in 2003, various entities have supported GHG inventory efforts in Malawi, such as the U.S. Country Studies Program (USCSP), the Global Environment Facility (GEF)/UNDP, and UNFCCC National Communications Support.6 USFS IP was not involved in the GHG inventory development work, but instead was involved with DoF regarding REDD+ readiness.7                                                              6 As noted in the INC (2003) and the Second National Communications (2012) to the UNFCCC, Ministry of Natural Resources and Environmental Affairs. Lilongwe, Malawi. UNDP/GEF support to the Second National Communications is also documented here https://is.gd/pjeXE6 7 2017. Personal communication (John Kerkering, USFS IP). August 7th, 2017. 21  EAD is the GOM counterpart for the GHG inventory work under PERFORM; however, it is regarded as a challenging partner to work with, due to their ability to garner resources from multiple donors, and resulting shifting priorities. Evidence from KIs suggests that the lack of continuity from embedded PERFORM project staff due to the turnover of consultants hired by the project slowed progress. The project worked, through MOUs, with different ministries to gather data from the different emissions sources and sinks. In general, the sectoral ministries were enthusiastic participants in the inventory. REDD+. The key document for REDD+ policy in Malawi is the GOM REDD+ Action Plan 2014 – 2019 (MRPAP). It was developed with support from the USFS IP within MRRP (Malawi REDD Readiness Program, MRRP, 2012 – 2015), prior to support from PERFORM. Multiple KIs brought up the support from USFS and praised their involvement as a catalyst in the process. This action plan sets out multiple targets from 2014 to 2019, including finalizing Malawi’s National REDD+ Strategy by the end of 2018. The project is supporting RExG in accelerating the timeline for developing the strategy, currently aiming to have the strategy finalized by January 2018. However, KIs noted that the GOM has been too slow to develop what they referred to as the REDD+ action plan. A recipient of PERFORM support, indicated that their organization had been receiving support from the project to meet the Green Climate Fund’s (GCF) assessment criteria for environmental and social safeguards policy, as part of the GCF accreditation process.  A key element of the MRPAP is the GOM “no-regrets” approach, which states that the country will pursue REDD+ regardless of its uncertainty within the UNFCCC, given Malawi’s commitment to climate change adaptation and mitigation, maintaining and increasing forest ecosystems, and ancillary benefits (such as clearer resource tenure, enhanced technical capacity, improved forest governance, and the development of data and data standards).  Prior to PERFORM, REDD+ readiness in Malawi evolved considerably with support from USAID MRPP and UN-REDD. The Kulera Landscape REDD+ Program was a result of the USAID/Malawi-funded Kulera Biodiversity Project, and has successfully sold carbon credits. PERFORM builds upon this foundation to further advance REDD+ readiness, through local scale forest inventories and carbon stock assessments, capacity building at DoF, operationalizing the MRPAP, and improving policy and legal frameworks. The local forestry inventories and C stock assessments were widely lauded, and their importance to the National Forestry Inventory (NFI) recognized and appreciated by KIs.  As for REDD+ readiness, there appears to be a lack of clarity even among stakeholders close to the project about the distinction between the REDD+ Action Plan (completed) and Strategy (in progress). This has caused some frustration, especially among those interested in going beyond the no-regrets benefits of REDD+, and wishing to see more operational activities such as Kulera. It was unclear whether this lack of clarity reflects the capacity and level of engagement of the KIs, or insufficient communication. INDC – PERFORM. PERFORM supported EAD on the preparation of the INDC, through the development of an INDC Options Assessment report, which summarized existing UNFCCC and third 22 party guidance, as well as providing suggested approaches to Malawi’s INDC.8 Additionally, the project supported consultants that worked within the different ministries, in particular relating to the AFOLU aspects. These ministries included DoF, MAIWD, and MNREM. One of the project’s strengths was building a partnership around the INDC consisting of GOM, academia, the private sector, civil society organizations (CSO), and development partners (DPs). The project supported EAD on determining the investments needed to support different GHG mitigation scenarios. Analytical Summary. The project has contributed significantly to the GHG Inventory, MRPAP, and INDC. In addition, building upon its GHG Inventory support activities, PERFORM has also supported the Third National Communication to the UNFCCC. All evidence from KIs and secondary sources indicate that the level of progress on these national policies and plans has been good. However, at a more detailed level of analysis, GOM institutions beyond EAD and DOF could benefit from technical support related to the analysis and use of land use data (e.g., agricultural sector). Some concerns were expressed regarding the use of external consultants versus GOM staff, but it was unclear to the MTE team whether or not this was related to the common phenomenon of government staff attempting to secure development resources. PERFORM appears to be using a wide range of well accepted capacity development strategies to achieve project goals. With respect to REDD+, there exists a significant degree of confusion even among stakeholders close to the project regarding the difference between the MRPAP and REDD+ National Strategy. This requires clarification from the project and DOF such that stakeholder expectations and the GOM "no-regrets" approach to REDD+ are well aligned. EVALUATION QUESTION IA – RECOMMENDATIONS  The project should consider expanding technical capacity beyond that related to obtaining and analyzing forest data, and RS/GIS, beyond DoF, to ensure both that other GOM REDD+ counterparts to DoF are at a similar level of technical capacity, and that policymakers can understand and effectively use geospatial data products from DoF and other GOM institutions (e.g., for sectors such as agriculture and water resources). Doing so would further operationalize REDD+, and bring Malawi closer to the possibility of eventual revenue generating REDD+ projects. There is also an important need to boost the capacity of national and district level policymakers to use these geospatial data products. To this end, staff at GOM institutions beyond DoF and EAD may require added capacity building in order to more effectively analyze land use data and create geospatial products to move REDD+ readiness policy and strategy forward.  PERFORM, DoF, and other GOM counterparts should clarify for stakeholders the difference between the MRPAP and the REDD+ national strategy, and its implications for eventual REDD+ projects, to avoid misunderstandings and frustrations among those stakeholders who are mainly interested in the revenue generation aspects of REDD+ projects. This effort could be complemented by an explanation of the no-regrets approach of the GOM, such that when the strategy is completed, stakeholders, such as communities around forest areas and national REDD+ related NGOs, will have realistic expectations. EVALUATION QUESTION IB – Do the pilot projects at the local level effectively support the national goals, and are they achieving meaningful livelihood benefits at the site level? FINDINGS                                                              8 2015. Makungwa, S. et al. Options Assessment for Malawi’s INDC Development Process. Winrock International and Tetra Tech (for USAID). 23  The project’s efforts at the district and local scale can be divided into a package of activities aimed at benefiting beneficiary villages; and the forestry inventories and C stock assessments.  The village level activities are aimed at addressing local drivers of deforestation and degradation, and enhancing rural livelihoods in the districts of Mzimba (Perekezi Forest Reserve), Ntchisi (Ntchisi Forest Reserve), and Machinga (Liwonde Forest Reserve), focusing on villages adjacent to protected areas.9 Parallel aims include decreasing the rate of expansion of the agricultural frontier into forested areas by increasing productivity, increasing food security, and stimulating sustainable livelihoods as a potential alternative to illegal charcoal production. Secondary sources confirm that PERFORM organized village clubs as entry points for these interventions, and trained them in the dynamics of group projects. Specific activities can be divided into irrigation – providing treadle pumps for small scale irrigation; agriculture – providing seeds for sunflower, improved varieties of soybeans, maize, and potatoes; chickens with a pass-on replication program for vulnerable households; the introduction of OFSP; and a variety of conservation agriculture technologies; energy/forestry – community woodlots and improved cookstoves; and non-timber forest products such as honey beekeeping. In Mzimba, team members were shown a few examples of off-farm regeneration.  VNRMCs are part of participatory forest management efforts in Malawi. Registered with the DFO, they provide a potential link between village level participatory activities and the government structure. Discussions in Machinga did not reveal a tight integration between the DFO and VNRMC. VNRMC members here and elsewhere appear to overlap closely with the village famer “clubs” used as entry points by the project.  While most of the crops provided were already being grown by the farmers before PERFORM interventions, the project provided improved varieties with higher yield and/or early maturing times. In Mzimba and Machinga, qualitative evidence obtained from extensive group interviews with village clubs, as well as a group discussion with representatives of six GVHs, indicated that most of the increased production was used for consumption. Project documents indicate that in the 2016 winter cropping season, the project supported additional beneficiaries through treadle pumps and more extensive OFSP distribution. The evaluation team was unable to confirm this through the interviews and discussions with beneficiaries. Crops which were mainly sold include potatoes (Irish and OFSP), pigeon peas (a lentil), soya, and sunflower seeds.  With respect to improved cookstoves, penetration in the target villages visited was high, ranging from 70 to 100%. Beneficiaries mentioned the faster cooking times and ability to use smaller branches as significant advantages. The acacias in the woodlots visited by the evaluation team were quite small in height (more like saplings than trees per se), and contradicted the beneficiaries’ assertions that they would be providing fuel wood in a year or two from the present. It is possible that their growth was impeded by the drought associated with a strong El Niño in the 2015-2016 growing season. The improved cookstoves and woodlots, assuming sufficient growth, have the potential to reduce fuelwood extraction.  In terms of conservation agriculture (CA), beneficiaries mentioned that they had been sensitized by PERFORM to specific technologies, for example, maintaining some trees and shrubs when                                                              9 While not mentioned by KIs, the second edition of the PERFORM FY 2017 AWP mentions that the project will add two new sites in 2017, including the eastern buffer of the Liwonde National Park (Machinga) and the Thuma Forest Reserve, which spans four districts – Salima, Dowa, Dedza, and Lilongwe Rural. 24 clearing fields, alternating rows of crops with trees, and pits for water and soil conservation. While honey beekeeping was already known to the beneficiaries, and is only practiced by a few beneficiaries, anecdotal evidence suggests that the use of larger hives received from the project increased production and sales.  Another element of PERFORM local level work was the forest inventory efforts. PERFORM conducted site based mapping of the drivers of deforestation and forest degradation and identified associated hotspots. The local forest inventory work was an effort to develop standard operating procedures (SOP) at the local level. It was conducted by five teams working in Perekezi, Ntchisi, and Liwonde. PERFORM staff worked directly with District Forestry Officers (DFOs) who supplied substantial staff to conduct the activities. The project informed the district councils (DCs) about these activities. Interviews with district level authorities did not reveal that they were in possession of any of the forest inventory or C stock assessment data. At the district level, there exists a Local Assembly Management Information System (LAMIS) to potentially house such information. DFO typically keeps records of tree planting, though not always in digital form.  Involved with PERFORM as a grantee, the work done by Kawandama Hills Plantations (KHP) includes sustainable charcoal as well as alternative livelihoods elements. KHP plants trees such as Corymbria citriodora, other eucalyptus, rose geranium and lemongrass to produce essential oils for local and export markets. KHP uses the trunks of Corymbria trees which can no longer self￾coppice to make sustainable charcoal, which is sold in retail markets, and represents about 2% of charcoal production nationally. KHP received a grant from PERFORM to support beneficiary communities planting 10 hectares of this tree annually. 121 farmers organized into a cooperative have planted trees in the surrounding communities. The project also supports KHP’s own planting 30 hectares of these trees annually on their own plantation. Participating farmers earn significant income from growing the Corymbria trees, with anecdotal evidence of farmers dedicating themselves exclusively to growing these trees. EVALUATION QUESTION IB – CONCLUSIONS  VNRMCs are part of participatory forest management efforts in Malawi. Registered with the DFO, they provide a potential link between village level participatory activities and the government structure. As such, noting that the linkage between the VNRMCs and DFO may not be robust in the target areas, there may be little added value besides formal district level registration. Also, as a practical matter, the membership of the farmer clubs overlap closely with the VNRMC.  In addition to meeting the important need of food security, especially in light of the El Niño drought during the 2015-2016 growing season, some of the crops improved or introduced by the project provide farmers with the opportunity to earn much needed cash. Increasing the cash inflows from improved crop productivity, and the introduction of the new crops (e.g., OFSP) would provide beneficiaries with extra income for basic needs and agricultural investments, and potentially reduce the need to expand into the surrounding forest areas.  In rural areas, reducing the pressure on the forests for energy involves decreasing both charcoal production (for sale) and fuelwood collection (for cooking). The former remains a lucrative but illegal source of income for cash strapped families. No evidence was obtained to suggest that increased cash earnings from any of the village interventions provided sufficient income substitution to defray that potentially earned from charcoal sales. 25  Woodlots observed in Mzimba and Machinga showed that growth of the acacia had been very slow, with most trees below 3 feet in height. While it is possible that the trees had been adversely impacted by the 2015 El Niño associated drought, the growth to date appeared to be incompatible to beneficiaries’ assertions that they would be harvesting fuel wood from them in a year to 2 years from the date of the evaluation visit. With respect to off-farm regeneration, the team was shown acacia around 20 feet in height, in a natural area adjacent to the woodlot. The difference in height was attributed to the 2015 drought, but the team’s opinion was that the trees were naturally occurring, and did not represent off-farm regeneration. Based on the sites visited, the woodlots and off-farm regeneration did not appear to be thriving, and their impact on reducing pressure on forests may therefore be limited.  The KHP model of engaging beneficiary communities in the production of a high value commodity appears to be very successful, and provides farmers with tangible cash earning potential. While data on payments was not available, anecdotal evidence such as farmers dedicating their land exclusively to Corymbria, suggest that this model shows high potential as a low emissions land use opportunities, and a potential replacement for income form illegal charcoal production. Analytical Summary. The pilot projects encompass both agriculture and energy/forestry, as well as livelihood activities aimed at benefiting beneficiary villages, together with the forestry inventories and C stock assessments. Their support for national climate change and REDD+ goals is mixed. With respect to the agricultural interventions, their main (and important) contribution has been to increase food security, especially in ameliorating the negative impacts of the 2015 El Niño drought. While some excess production of crops are sold in local markets, no evidence was obtained to show that the income obtained could replace the potential income from illegal charcoal sale. In terms of meaningful livelihood benefits, the KHP Corymbria out-grower scheme shows the greatest promise. Beekeeping does not seem to be adopted in any significantly increased way. The distribution and capacity building for production of cookstoves has been greatly successful, with significant penetration and potential for replication. This reduces the need for fuel wood from the adjacent forests, with important implications for lowering carbon emissions and forest conservation. The woodlots visited by the MTE team did not seem to be thriving and the trees appeared stunted, potentially related to the 2015 drought, and current unavailability of water. As such, it is difficult to see how they could contribute to providing fuel wood in any meaningful way within the life of the project. As for the local forestry inventories and carbon stock assessments, they have been extremely successful and recognized by DFO as well as national level GOM officials. Their contributions in terms of data, methodology, analysis, and outputs have been recognized, as well as their development of standard operating procedures. These certainly can and are influencing national level processes related to forestry inventories, plans, and strategies. This technical forestry contribution is significantly advancing GOM climate change and REDD+ goals. EVALUATION QUESTION IB – RECOMMENDATIONS  To increase cash income for farmers, the project should continue its efforts to increase the production of marketable commodities. PERFORM should consider significantly ramping up planned efforts to pool farmer production and sell in greater volume through other locally based initiatives. Preliminary efforts in this direction could include already planned activities (Ntchisi – coffee and macadamia; Machinga – Kilombero rice). Doing so will help instill low emissions land use opportunities at the local scale. 26  The project should consider expanding the use of the KHP Corymbria out-grower scheme model to other private sector partners, for other high value cash crops such as coffee and macadamia.10 EVALUATION QUESTION 2 – What lessons and implications does PERFORM offer for REDD+ readiness scaling up and likely impacts on REDD+ outcomes? This can also include a synthesis of lessons from the REDD+ Readiness Program. FINDINGS  The MRPAP, among other goals, stipulates the GOM aim to complete a long-term, national REDD+ strategy by the end of 2018. Its aim is to scale up REDD+ policy by mainstreaming it within relevant policy and legal frameworks within Malawi; to launch a National Forest Monitoring System (NFMS); and to draft and agree upon a roster of social and environmental safeguards that will inform all aspects of program design, implementation and management.  According to responses from KIs and project documents, in addition to PERFORM’s support of the RExG to draft the National REDD+ Strategy, the project also supported the National Charcoal Strategy and the National Forest Landscape Restoration Assessment and Strategy (NFLRS).  The project works actively with the RExG working groups, Technical and Steering Committees on Climate Change, and links both levels effectively. KIs indicate that the RExG has proven to be a particularly productive venue to engage a wide spectrum of stakeholders.  PERFORM supported the launch of the NFMS Roadmap in 2015 and improved capacities of stakeholders to monitor and analyze land use and land cover in Malawi. The support included the acquisition and analysis of forestry data, forest inventories at national and local project sites. The project has provided GIS/RS training (2 PhD and 2 MSc students and 7 in-country training sessions) for the DoF and other stakeholders.  The College of Forestry and Wildlife (Dedza) has solicited support from the project to develop a GIS laboratory and PERFORM awarded the College of Forestry an in-kind grant (hardware, software, and improving the lab conditions) for the GIS lab. Some of the products from these efforts were already been used by key stakeholders. For example, the Department of Forestry used some of the maps from PERFORM analyses to target sustainable land management interventions for other projects. However, it was not clear whether the use of the data and products will increase beyond the LOP.  Some stakeholders suggested that the archiving and ongoing use of open data platforms (e.g. MASDAP) can enhance the use of the data and products. At the district level, LAMIS may be a possibility, while noting the challenges related to capacity and the prevalence of paper records.  Both PERFORM and DoF staff members have lectured at the College of Forestry and Wildlife, as well as at other academic institutions such as MZUNI and LUANAR. This outreach as well as the future GIS lab at the College of Forestry and Wildlife is expected to support the creation of a cadre of young professionals that could contribute to REDD+ technical analyses in the future.                                                              10 Global macadamia prices are at an all-time high of between $16/kg and $17/kg and the market is expected to increase. Source: Agriorbit. Record US dollar prices expected for macadamia nuts in 2017. Feb. 28, 2017. https://agriorbit.com/record-us-dollar-prices-expected-macadamia-nuts-2017/ 27  Evidence from project beneficiaries indicated scaling up use of improved cookstoves. The evaluation team visited a few households using the cookstoves in the target sites. Other technologies mentioned by beneficiaries to be scaling up were the OFSP and minimum tillage. EVALUATION QUESTION 2 – CONCLUSIONS  PERFORM has provided land cover/use data to DoF, FRIM and academic institutions. The local forest inventories and carbon stock assessments have contributed to the NFMS. The data and enhanced analytical abilities allow GOM to move towards a baseline forest inventory key to REDD+ readiness.  A number of forest and REDD+ related instruments and strategies have been finalized in the past 2.5 years. PERFORM’s support to the RExG, positions the activity to assist in the dissemination of these strategies through both, the RExG and the Technical Working Groups members. This also points to its potential in assisting them to streamline REDD+ activities in their sectoral strategies.  The project’s support of a wide range of forestry related policies such as the National Charcoal Strategy and the National Forest Landscape Restoration Assessment and Strategy (NFLRS) further points to the mainstreaming, as well as the scaling up and out of REDD+ readiness efforts. Analytical Summary. PERFORM has been successful in contributing to multiple aspects of REDD+ readiness, through activities including support to the NFMS, National Charcoal Strategy, and NFLRS. The project has actively engaged RExG and its constituent bodies, and effectively used it as a venue for outreach and ensuring buy-in from a wide range of stakeholders, including GOM, academia, civil society, and the media. Through lecturing at academic institutions, the project is helping ensure a future cadre of trained experts in REDD+ related issues. The ongoing support towards the development of a GIS Lab at the College of Wildlife and Forestry (Dedza) further advances the goal of institutionalizing sustainable capacity development. The storage and management of the forestry data could be improved, but is subject to the limitations of the national context regarding reliable electricity supply and sufficiently rapid internet connectivity. EVALUATION QUESTION 2 – RECOMMENDATIONS The project should consider an assessment of the current data handling and sharing infrastructure at DoF and FRIM as key REDD+ related institutions, including limitations related to stable electricity provision and internet connectivity. Identifying an effective and strategically positioned data repository could provide a more sustainable and consistent source of updated data to support REDD+ scaling. The connectivity needed for the transfer of large quantities of data will be a challenge to these efforts. EVALUATION QUESTION 3 – How sustainable are the efforts at the local and national level? Are GOM officials and local communities likely to be able to continue these programs on their own when PERFORM ends? FINDINGS At the national level:  Evidence collected across over 25 interviews at the national level, KIs from both GOM and other development partners referred to the importance of PERFORM-supported achievement of a 28 GHG inventory, reflecting its inherent value as a fundamental requirement of the UNFCCC framework. This also pointed to an expectation of sustainable progress in meeting additional commitments, harbored in the visibly enhanced capabilities built through PERFOM support, strategically hosted inside the EAD.  KIIs also pointed to multi-donor funding that overlaps PERFORM’s contributions, which has also translated into support from the International Energy Agency and GIZ. However, donor funding is only an intermediate step to the sustainability of progress that will require added GOM investments. In this respect, discussions with PERFORM suggested that items on watershed management were being incorporated into the National Resilience Plan. Likewise, KIs indicated that GHG Inventory was being included in the national budgets for EAD and that climate change and forestry were being prioritized in the upcoming Malawi Growth and Development Strategy (MGDS 2017-2022). At the sub-national and community levels:  PERFORM has provided hands-on trainings on technologies introduced to beneficiaries. In this respect, added sustainability may be inferred from an enhance use and ownership of appropriate technologies. For example, the evaluation team observed that improved cookstoves were being locally built by women in the target villages. The team also saw established nurseries for OFSP in the target sites to ensure the availability of planting materials in the next season.  PERFORM hosted collaboration with FtF MISST and CIP, possibly ensuring the continuation of some of the agricultural interventions such as the OFSP and improved legume varieties.  Although clubs are an evident and successful vehicle to introduce PERFORM interventions among a predominantly female population, there is no evidence to support that the PERFORM clubs can evolve or replicate independently beyond the specific communities selected and supported by USAID funding. None of the interviews mentioned concrete access to alternate sources of funding. EVALUATION QUESTION 3 – CONCLUSIONS  PERFORM efforts will likely continue after the USAID-funded project life based on both, additional technical and financial resources from other development partners such as the German Government and DFID at the national level, as well as FtF MISST and CIP inputs and enhancements at the site level.  GOM resource allocations can also be expected to be a contributing factor in the continuation of USAID-funded activities. The GOM has incorporated the GHG inventory as a line item expenditure into its national budget. Moreover, the expectation of climate change and forest conservation to be one of the priorities in the upcoming medium and long-term GOM strategies was alluded to by the PERFORM specialists (KIs) embedded within the EAD and DoF.  At the local level, improved cookstoves showed great promise for uptake from villages adjacent to the target villages through word-of-mouth. The provisioning of local chicken used a built-in pass-on approach ensuring replication that may factor into its sustainability by supporting a multiplier effect. 29 Analytical Summary. There are two drivers to the sustainability of national level efforts, one institutional, and the other financial. With respect to the former, obligations under the UNFCCC will require that the updating of GHG inventories (for inclusion in the national communications), as will the INDC. This reflects a soft sustainability, essentially meaning that the GOM will have to continue these efforts using its own or donor resources. The other driver is financial, and some KIs reported that some of the national level efforts could receive financing via MGDS prioritization, noting that forestry was a GOM priority. Another source of intermediate sustainability would be other bi- and multi-lateral donors supporting the national level efforts. At the local level, cookstoves appeared to be the most enduring technology, and replication of crops such as OFSP show some promise of sustainability, as does the chicken pass-on program. Based on the evidence obtained, it is unlikely that woodlots will continue, as the trees did not appear to be growing very well, and it is difficult to conceive of farmers taking time away from crop production to irrigate the woodlot trees. EVALUATION QUESTION 3 – RECOMMENDATIONS  PERFORM should continue its efforts to introduce cookstoves and build the capacity to build them, which will ensure that benefits related to decreased time needed for cooking and fuel wood collection, as well as some level of diminished dependence on forest resources  The project should consider strengthening its linkages to National Technical Committee on Climate Change Technical Committee (NTCCC) to foster opportunities to influence national priorities on climate change and forest management.  The project should consider strengthening its linkages to the District Environmental Sub￾Committee (DESC) of the District Councils to enhance collaboration with other projects, to incorporate lessons learned into the District Development Planning Process and to take advantage of complementary financial opportunities for target communities.  There is a need for PERFORM to explore market-linkages for smallholder farmers in the target communities to stimulate adoption of the introduced technologies as well as those to be introduced in the remaining years. Fostering wide and spontaneous adoption can better ensure sustainability based on new market opportunities. EVALUATION QUESTION 4 – How well is PERFORM integrated with: a) other USAID initiatives (e.g., Feed the Future): and b) other DPs efforts in GCC and forest conservation? FINDINGS  Reports from PERFORM IPs and project documents indicate close collaboration with the German Federal Ministries of Economic Cooperation and Development (BMZ), the German Federal Ministry for the Environment, Nature Conservation and Nuclear Safety (BMU), and the United Kingdom’s Department for International Development (DFID) towards the National Forest Landscape Assessment. The Federal German Government and the International Energy Association also collaborated with PERFORM on the development and submission of the national INDC to the UNFCCC, while efforts towards the GHG Inventory were supported in collaboration with the UNEP. Furthermore, the reports indicated strong integration with a number of USAID funded activities in the Machinga District (e.g. FtF MISST and CIP), NJIRA, and Farm Radio Trust). This was confirmed by field visits.  Regarding collaboration with other DPs, the evaluation team observed close collaboration with the Shire River Basin Management Program (SRBMP) in Machinga District. Through the DoF, the 30 two projects chose target areas to avoid the duplication of effort. The key difference between PERFORM and SRBMP was that the latter provided Community Environment Conservation Revolving Funds to its target areas, whereas PERFORM provided direct inputs and capacity building.  The evaluation team found evidence of more possible opportunities for integration with other partners in the target areas and nationally on climate change and forest conservation in Malawi. KIs mentioned that the MCA grantee CICOD could potentially be a valuable partner in the Machinga district; the World Bank LDF MASAF IV on restoration activities in Mzimba, Ntchisi and Machinga; the UNDP ADAPT-PLAN project in Machinga district; The Shire River Basin Management Program (SRBMP); and the JICA Project for Conservation and Sustainable Management of Dzalanyama Forest Reserve (DFR) in Lilongwe. These projects could provide an opportunity for PERFORM to improve results and ensure sustainability while avoiding duplication. These were also deemed as opportunities to collaborate beyond traditional stakeholders, enable co-financing options, and leverage the implementation of successful interventions beyond the target sites. EVALUATION QUESTION 4 – CONCLUSIONS  PERFORM has collaborated and coordinated its efforts with other USAID funded projects, particularly in the Machinga district, on the provision of agricultural inputs. Opportunities exist for PERFORM to collaborate with other projects and programs at the national and local level.  PERFORM has leveraged coordination with other development partners to avoid the duplication of similar efforts in Machinga district. The World Bank funded Shire River Basin Management Project is an example of this good practice, involving bilateral/multilateral coordination. Analytical Summary. PERFORM’s implementation on the ground evidences important efforts to effectively integrate with other USAID-funded activities, and engage in substantive collaboration with other DPs. Field visits confirmed, for example, work done in concert with activities such as FtF MISST and CIP. This is also the case with other DPs in the context of the NFLRS and INDC. Likewise, intentional avoidance of duplication of efforts was observed with the World Bank SRBMP in Machinga. Future opportunities for additional collaboration with other DPs could include JICA’s DFR initiative, currently in its start-up phase. EVALUATION QUESTION 4 – RECOMMENDATIONS  PERFORM should continue its efforts with the district councils to explore opportunities for collaboration with other projects (e.g. MCA’s Circle for Community Development (CICOD), World Bank’s MASAF IV) to improve outcomes and ensure sustainability. EVALUATION QUESTION 5 – How are gender issues integrated into the activity? Have women been able to participate fully and benefit from it, including from income-generating activities? For cross cutting subjects such as gender and capacity building, how effectively do they contribute to PERFORM’s intended results? FINDINGS  Women are actively engaged at the national policy level in all project related fora. While, their actual numbers relative to men are low, this is a reflection of the gender distribution of staff in the GOM, and unrelated to project activities. KIs noted that women were actively involved and even 31 held leadership roles in national level project efforts, and that they brought gender related perspectives that would not have been discussed in their absence.  Women participation as evidence of integration – All pilot sites visited by the evaluation team offered high numbers of women participants (often above 90%) where women were clearly placed in leadership positions in terms of the group’s productive activities, as well as in organizational roles and decision-making functions. Most individuals that responded to questions on group structure, daily routines, specific tasks and resources allocation were women. Responses on crop selection and care, as well as household impact were also addressed by women, based on their own experience.  Women benefitting from Income-generating activities – All sites visited provided evidence to support avid and proactive participation of women and of women/households directly benefiting. Women respondents specifically referred to benefits in terms of additional cash income that was applied to household expenses such as school fees, medical services and transportation, where these cash funds were directly derived from income-generation activities supported by PERFORM.  Lack of advancement in policy/legal framework impacts sustainability of gains at the local level – Responses from DoF/EAD KIs at the national level (Lilongwe interviews) to questions about the integration of women into project activities mirror the conditions and gains observed and alluded to by beneficiaries at the local level. However, when asked about final decision-making, in households where men are regularly present they have the final authority on issues of land use and use available income. EVALUATION QUESTION 5 – CONCLUSIONS  PERFORM has achieved generation of new income through added productivity that directly (and predominantly) engages women participation in the selected sites. While there is evidence of limited commercial activity based on the crop yields most benefits linked to women mainly support individual household needs.  The gains and integration realized by women through PERFORM support in the pilot sites face important challenges in their expansion or replication beyond USAID-funding. The evident lack of a congruent advancement at the national level is a key factor in any long-term sustainable replication of PERFORM activities at the local level. Analytical Summary. At the level of participation and engagement both nationally and especially locally, women are well (usually over 80%) represented in project activities. Some women even drive key elements of national REDD+ readiness. At the local level, regardless of the intensive participation of women both in farmer clubs, and in the actual implementation of project activities within the villages; and irrespective of whether the societies were matrilineal or patrilineal, discussions with farmer groups, and representatives of GVHs, indicated that decisions regarding cash income and land use were strongly influenced or made exclusively by men. As such, while women have been able to participate in income￾generating activities, at the household level, their ability to benefit from them is controlled by men. As discussed in the appropriate sections above, capacity building has made significant contributions towards the national level goals. At the local level, women’s labor is contributing towards the implementation of the local scale activities, but, as discussed above in question 1b, the main impact of these has been on improving food security. EVALUATION QUESTION 5 – RECOMMENDATIONS 32  PERFORM should consider leveraging existing savings & loans groups, particularly those with have linkages to business interests and investments, to introduce new incentives/expansions to its initial income generation activities. On-the-ground experiences in Malawi have already assisted women to access capital to purchase quality seed, pay for seasonal labor and buy agricultural inputs that factor into added productivity and income generation. For example, PERFORM may want to explore lessons learned from existing “Women in Coffee” savings models at the village level, currently supported by other development partners (UNWomen) under the Mzuzu Coffee Cooperative. ADDITIONAL RECOMMENDATION ON DATA QUALITY AND USE Given the limited applicability of data collected under broader standard indicators in addressing specific (possibly junctural) challenges, as well as emerging opportunities, planned PERFORM activities such as a learning survey need to carefully consider the knowledge gaps which may have emerged in the course of the first half of the LOP. Moreover, learning surveys should be contextualized against USAID’s broader framework for “Collaborate, Learn and Adapt” (CLA / Learning Lab), as well as provide clearly identified opportunities to contribute to learning across the USAID/Malawi portfolio. ANNEXES ANNEX I: EVALUATION STATEMENT OF WORK ANNEX 2. EVALUATION DESIGN, METHODS AND SOURCES ANNEX 3. DATA COLLECTION TOOLS AND RESOURCES 3.1 QUESTIONNAIRE TRACKER 3.2. FOCUS GROUP DISCUSSION GUIDE 3.3 KEY INFORMANT NOTE TAKING TEMPLATE 3.4 KEY INFORMANT TRACKER TEMPLATE 3.5 PROCESS DOCUMENTATION RESOURCES (LINK TO) 3.6 INDIVIDUAL & GROUP QUESTIONNAIRE ANNEX 4. CONFLICT OF INTEREST FORMS FOR TEAM MEMBERS ANNEX 5. STATEMENT OF DIFFERENCES ANNEX 6. TOC COMPARISONS 1 Annex I: Evaluation Statement of Work Mid-term Performance Evaluation of USAID’s Protecting Ecosystems and Restoring Forests in Malawi (PERFORM) Activity United States Agency for International Development (USAID)/Malawi, Office of Sustainable Economic Growth (EG) March 2017 POINT OF CONTACT MADALITSO KAFERAWANTHU COR, PERFORM Project Office of Sustainable Economic Growth USAID/Malawi Abel Nyoni COR, Monitoring, Evaluation, and Learning Support (MELS) Project Office of Sustainable Economic Growth USAID/Malawi Tel: 265 (0) 177-2455, Ext xxx Email: anyoni@usaid.gov Project Information Project Name Protecting Ecosystems and Restoring Forest in Malawi (PERFORM) Award Number T.O. # AID-612-TO-14-00003 Original Project Dates 2 September 2014 – 1 September 2019 Original Funding Implementing Partner TetraTech ARD 2 CONTENTS I. Background II. Program Components and Description III. Evaluation Purpose IV. Evaluation Questions V. Evaluation Methodology VI. Existing Sources of Information VII. Deliverables VIII. Evaluation Team Composition IX. Level of Effort X. Scheduling and Logistics XI. Budget 3 I. Background USAID’s Protecting Ecosystems and Restoring Forests in Malawi (referred to hereafter as PERFORM) was specifically designed to align with Malawi’s Growth and Development Strategy (MGDS) and to promote forest conservation and green growth. It is the main implementation vehicle for the low￾emissions partnership between the U.S. and GOM. The project period is 5 years, from 2 September 2014 to 1 September 2019. PERFORM is one of the key activities under “Development Objective 2 (DO2): Sustainable Livelihoods Increased” of USAID Malawi’s Country Development Cooperative Strategy (CDCS). PERFORM has completed approximately 2.5 years; therefore, USAID wishes to evaluate the performance of the project at the mid-point of its five-year program. Consistent with USAID’s 2011 Evaluation Policy and its more recent 2016 updates, the primary goal of this mid-term evaluation is to provide evidence so that USAID/Malawi can determine whether PERFORM is on track to achieve its desired results or not. Other goals of the evaluation are to inform what course corrections, if any, are needed, identifying specific project interventions that can be scaled up further and to determine best ways to ensure sustainability of the activities, institutions and capacities promoted by the project. Finally, this evaluation will provide strategic options for future USAID/Malawi engagement in climate change and offer lessons learned with wider applicability to other activities in Malawi or beyond. II. Program Components and Description Malawi is a Least-Developed Country (LDC), whose population is predominantly rural and heavily dependent on smallholder agriculture. The limited ability of this population to afford commercial fuels has led to a heavy dependence on fuelwood and charcoal for fuel, with the result that Malawi suffers from severe deforestation. Deforestation, agriculture, and land use change are also the leading sources of GHG emissions in Malawi. PERFORM seeks to address the drivers of deforestation and provide alternative livelihood opportunities, including through the potential payment for environmental services associated with the global REDD+ framework. It builds capacity at the national and local level to implement REDD+ and meet the GOM’s commitments to the UNFCCC. Component 1 (C1): Advance Reducing Emissions from Deforestation and Degradation (REDD+) readiness. PERFORM builds capacity in the GOM to advance readiness for REDD+, a framework under the UN Framework Convention for Climate Change (UNFCCC) for reducing emissions from the forest, agriculture, and land use sectors, including through potential payments for GHG emissions avoided/sequestered. Activities under this component build GOM capacity for data collection concerning forest carbon stock and greenhouse gas (GHG) emissions from forests, improve the enabling framework related to REDD+, and support implementation of the GOM’s REDD+ Program (MRP) Action Plan. Component 2 (C2): Increase low-emissions land use opportunities in targeted geographies. The focus of Objective 2 is to directly improve rural Malawian’s access to “land use opportunities that increase the economic and livelihood returns to conservation,.” as part of a REDD+ approach in which initial subnational policies and measures inform the decision on those to be adopted and implemented at a national level. PERFORM’s REDD+ site-based pilot activities, carried out in the districts of Machinga, Mzimba, and Ntchisi, use a suite of tailored interventions that address the supply and demand side of the drivers of deforestation. Key partner institutions are the regional and district forestry offices, local forestry organizations, the district assemblies, and the Traditional Authorities, Group Village Heads and Village Heads. Component 3(C3): Improve low-emissions development capacities. Objective 3 complements work in Objectives 1 and 2 by focusing on building Malawian capacity to collect, analyze, and report on GHG emissions at the national level. The intent is to support Malawi, as a least developed country, to 4 make informed decisions about nationally appropriate mitigation actions and access development assistance tied to climate change mitigation. The Malawian focal institution for Objective 3 is the Environmental Affairs Department Objective 4 and 5 are cross-cutting, supporting the first three objectives. Component 4 (C4) Institute pathways for sustainability. Under Objective 4, PERFORM increases sustainability of its USG-financed activities by enhancing the capacity of Malawian institutions, building public-private partnerships, and helping secure additional donor/private funds. Component 5 (C5): Advance Country Development. Objective 5 (CDCS priorities for integration and institutional strengthening advanced) focuses on finding and leveraging synergy between PERFORM and other USAID funded projects in Malawi, particularly in the focus districts for the CDCS. PERFORM’s link to USAID / Malawi’s Country Development Cooperation Strategy. USAID Malawi supports the GOM’s Malawi Growth and Development Strategy (MGDS II) through programming in the areas of agriculture/food security, health, environment/climate change, and democracy and governance. USAID’s CDCS hypothesize that if its assistance efforts are integrated across sectors, concentrated geographically, and coordinated better with other development partners (DPs), development results will be enhanced, more sustainable, and lead to achievement of its CDCS goal: Malawians’ quality of life improved. The results framework for the Country Development Cooperation Strategy (CDCS) 2013-2018 contains three Development Objectives (DOs) that contribute to USAID / Malawi’s overall Development Goal of Malawians’ “Quality of Life Improved.” PERFORM contributes to DO2: Sustainable Livelihoods Increased. PERFORM was specifically designed to align with MGDSII and to promote sustainable livelihoods through forest conservation and REDD+ PES activities at the local level, combined with national REDD+ policies that will help combat deforestation and its attendant erosion and watershed/soil degradation PERFORM Results Framework and Development Hypothesis. USAID has identified the development hypothesis under DO2 as: if sustainable livelihoods are increased, then Malawians’ Quality of life will be improved. Strengthening resilience to climate change (IR 2.1), adding value to agricultural production (IR 2.2), better nutrition (IR 2.3), and expanding agricultural trade (IR 2.4) together are means to achieve increased sustainable livelihoods. PERFORM’s basic hypothesis is that if the drivers of deforestation and forest degradation are addressed, forest conservation and green growth will be promoted, contributing to sustainable livelihoods and thus improved quality of life for Malawians. The PERFORM Results Framework is shown in Figure 2, below. 5 Figure 1: PERFORM Results Framework (RF) PERFORM Implementing Partners and Target Areas PERFORM is implemented by Tetra Tech ARD, in association with five subcontractors: Total Land Care (TLC), Center for Environmental Policy and Advocacy (CEPA), Michigan State University (MSU), Winrock International (Winrock), and World Resources Institute (WRI). PERFORM targets three districts in which pilot site- based activities are implemented and these include: Machinga, Mzimba, and Ntchisi. Within these districts, PERFORM’s activities are anchored by the Liwonde Forest Reserve (Machinga), Perekezi Forest Reserve (Mzimba), and Ntchisi Forest Reserve (Ntchisi). The Map below provides a general picture of where SEG activities are implemented including the three PERFORM districts. 6 Figure 2. Map of Target PERFORM Districts 7 III. Evaluation Purpose The primary purpose of this mid-term performance evaluation is to assess the effectiveness of the PERFORM design and implementation processes. The evaluation will also inform management of course corrections, if any, are needed. Additionally, through this evaluation, USAID seeks strategic and programmatic options for future engagement to support climate change mitigation and adaptation in Malawi. Major objectives of this evaluation are to: 1) Review, analyze, and evaluate the effectiveness of the PERFORM activity in achieving program objectives and contributing to USAID/Malawi’s efforts on climate change mitigation and adaptation in Malawi. 2) Evaluate major constraints in achieving expected project results. 3) Provide specific recommendations and lessons learned on strategies and approaches USAID/Malawi should continue in its future climate change mitigation activities. The final performance evaluation of PERFORM should, inter alia, determine whether the findings, recommendations, and lessons learned from the mid-term evaluation were taken into account during the remainder of the implementation period, and if so, what the impact was of any course corrections. IV. Evaluation Questions The contractor’s evaluation of PERFORM must be comprehensive. Whenever possible, the evaluation team should ensure that data is gender disaggregated and subject to gender analysis. The contractor shall assess PERFORM performance and achievements against the performance indicators, targets, reporting requirements, outputs and deliverables described in the approved M&E plan, Annual Work Plans and the agreement. In doing so, the team should pay particular attention to required climate change and biodiversity indicators. While this evaluation is not a data quality audit, the contractor shall assess and describe the quality and use of performance monitoring data and information generated during PERFORM implementation. Evidence-based conclusions and recommendations on how PERFORM used performance monitoring data and information in performance management are required when describing PERFORM effectiveness in component management. The following questions should be addressed: 1. How well is PERFORM progressing toward its high-level objectives of measurably advancing national REDD+ readiness, reducing deforestation relative to business as usual, and creating low-emission livelihoods opportunities? a) How effectively has PERFORM supported the GOM’s national climate change goals including the development of a GHG inventory, implementing its National REDD+ Action Plan, and developing and implementing its Intended Nationally Determined Contribution (INDC) under the UN Framework Convention on Climate Change? b) Do the pilot projects at the local level effectively support the national goals, and are they achieving meaningful livelihood benefits at the site level? 2. What lessons and implications does PERFORM offer for REDD+ readiness scaling up and likely 8 impacts on REDD+ outcomes? This can also include a synthesis of lessons from the REDD+ Readiness Program. 3. How sustainable are the efforts at the local and national level? Are GOM officials and local communities likely to be able to continue these programs on their own when PERFORM ends? 4. How well is PERFORM integrated with: a) other USAID initiatives (e.g., Feed the Future): and b) other DPs efforts in GCC and forest conservation 5. How are gender issues integrated into the activity? Have women been able to participate fully and benefit from it, including from income-generating activities? For cross cutting subjects such as gender and capacity building, how effectively do they contribute to PERFORM’s intended results? V. Evaluation Methodology The Contractor must propose a robust evaluation methodology to answer the key evaluation questions identified above. The evaluation methodology must follow a mixed–method approach using both quantitative and qualitative techniques in gathering reliable data and valid evidence of project outcomes or impacts including review of relevant PERFORM, USAID, GOM, and other secondary data sources. The contractor must develop a Mixed Methods Evaluation Design Matrix that will detail the data source, data collection methods, data collection instruments and analysis of data to answer each of the evaluation questions. By using a mixed-method approach, the evaluation team is expected to gain insight on the progress of PERFORM project activities (mostly from quantitative data collected by the project and others) and the processes (mostly qualitative information provided by the project staff that may be validated and enhanced by key informants) that lead to those impacts. It should generate gender￾disaggregated data and reflect attention to gender relations such as the participation of women in training. Sequential (whenever applicable and advantageous) and iterative approaches should be used to integrate the mixture of methods at various stages of the evaluation. The evaluators should utilize several different, yet complementary and inter-related forms of gathering information/data such as those described below. The contractor is expected to utilize its expert judgment and evaluation best practices in selecting which methodological components to include in the evaluation design. Document Review. Evaluation team members will review documents throughout the evaluation process including program reports, relevant studies to ensure that comprehensive and grounded best practices will be identified. Key Informant Interviews. The team will conduct one-on-one interviews with a variety of stakeholders including the most relevant GOM ministries and agencies, local government authorities, private partners and community leaders, bi- and multilateral development partners supporting GOM’s REDD+, forest conservation, and climate change mitigation efforts This will be done administering a semi-structured or structured questionnaire. Self-assessment. The IPs will respond to a self-assessment either through a questionnaire or standard interview checklist put together by the evaluation team and approved by USAID before use. Expert Opinion Survey. Utilizing expert opinion is a technique used increasingly in evaluation. The evaluation team, with approval of USAID, can apply this method as well. Focus Group Discussions (FGD). FGD (small group of 6 to 10 people) will be used to lead open discussion through a skilled moderator to gather semi-structured qualitative data. The pre-selected participants will discuss issues and concerns based on a list of key themes drawn up by the moderator. 9 No more than 10 questions will be addressed by a group. These sessions will encourage free flowing discussion about the activity. Mini Survey. This type of survey is small (30-40 participants) and can be performed rapidly in the field without analytical software such as SPSS or a large questionnaire. The sample size is not statistically significant; however, this type of analysis can be used to triangulate with other methods. Because of the small sample size this type of survey can be implemented quickly when time and resources are constrained. All the methodological strengths and weaknesses should be explicitly described in the evaluation report. Prior to the start of the evaluation, the evaluation team shall meet with SEG office to refine the evaluation questions and methodology, and address any other concerns the SEG office may have. Limitations to Data Collection. No current limitations exist that IBTCI is aware of. VI. DELIVERABLES All deliverables are internal to USAID, MELS and the Evaluation Team unless otherwise instructed by USAID. Evaluation deliverables include: Evaluation Team Planning Meeting. Essential in organizing the team’s efforts. During the meeting, the team should review and discuss the SOW in its entirety, clarify team members’ role and responsibilities, work plan, develop data collection methods, review and clarify any logistical and administrative procedures for the assignment and instruments and to prepare for the in-brief with USAID/Malawi. In-brief Meeting with USAID/Malawi. Within two working days of international team members’ arrival in Malawi; Inception Report: At the In-brief meeting, the Evaluation team will provide an Inception report that will outline key aspects of the Evaluation, including the Work Plan, evaluation Design Matrix, and proposed Data Collection Instruments (e.g., interview guides). See more detail below.  Work Plan. The Contractor will prepare a detailed work plan that includes task timeline, methodology outlining approach to be used in answering each evaluation question, team responsibility, document review, key informant and stakeholder meetings, site visits, survey implementation, travel time, debriefings (for USAID, implementing partner and, if decided, the GOM), draft and final report writing. The work plan will include a data analysis plan. The work plan will be submitted to the MELS and PERFORM CORs at USAID/Malawi for approval no later than the fifth day the Evaluation team arrives in Malawi.  Evaluation Design Matrix. A table that lists each evaluation question and the corresponding information sought, information sources, data collection methods, data analysis methods, and limitations. The matrix should be finalized and shared with USAID/Malawi before evaluation field work starts. It should also be included as an annex in the evaluation report.  Data Collection Instruments. Development and submission of data collection instruments to USAID/Malawi during the design phase and after the evaluation is completed; Regular Updates. The Evaluation Team Leader will brief the MELS and PERFORM CORs, on progress with the evaluation on at least a weekly basis, in person or by electronic communication. Any delays or complications must be quickly communicated to USAID/Malawi as early as possible to allow quick resolution and to minimize any disruptions to the evaluation. Emerging opportunities to strengthen the evaluation should also be discussed with USAID/Malawi as they arise. Debriefing with USAID. The Contractor will present the major preliminary evaluation findings to 10 USAID/Malawi through a PowerPoint presentation before the team’s departure from country. The debriefing will include a discussion of achievements and issues as well as any preliminary recommendations. The team will consider USAID and MELS comments and incorporate them in the Draft Evaluation Report. Stakeholders Workshop. The team will present the major findings from the evaluation to key stakeholders (as appropriate and as defined by USAID) through a PowerPoint presentation prior to the team’s departure from the country. The debriefing will include a discussion of achievements and activities only, with no recommendations for possible modifications to project approaches, results, or activities. The team will consider key stakeholder comments and incorporate them appropriately in drafting the evaluation report. Draft Evaluation Report. A draft report on the findings and recommendations should be submitted to USAID/Malawi and MELS 10 working days after departure of international team members from Malawi. The report must be no more than 50 pages in length (excluding annexes) and comply with the Checklist for Assessing USAID Evaluation Reports (see annexes). The written report should clearly describe findings, conclusions, and recommendations. The draft report must be of high quality with no grammatical errors or typos. A report is high quality when it represents a thoughtful, well-researched and well organized effort to objectively evaluate what worked in the project, what did not and why. The draft report must have well-constructed sentences that are presented in a way that clearly presents findings, conclusions and recommendations. The report should answer all the evaluation questions and the structure of the report should make it clear how the questions were answered. The draft report must meet the criteria set forth under the Final Report section below. USAID will provide comments on the draft report within 10 working days of submission. Final Evaluation Report. The Contractor will submit two versions of the Final Evaluation Report that incorporate Mission comments and suggestions no later than five working days after USAID/Malawi provides written comments on the Draft Evaluation Report. The format of the final reports is provided below. The report will be submitted in English, electronically. The public version of the final report that will be uploaded to the Development Experience Clearinghouse (DEC) will not include the annex on “Strategic Options for Future Programming” – evaluation objective 3. The USAID-only, procurement sensitive version of the final report will include the aforementioned annex on evaluation objective 3. The Contractor must ensure that Appendix I of the USAID Evaluation Policy – Criteria to Ensure the Quality of the Evaluation Report is followed. The final report should at a minimum meet the following criteria to ensure the quality of the report: ▪ The evaluation report should represent a thoughtful, well-researched and well organized effort to objectively evaluate what worked in the project, what did not and why. ▪ Evaluation report shall address all evaluation questions included in the scope of work. ▪ The evaluation report should include the scope of work as an annex. All modifications to the scope of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology or timeline need to be agreed upon in writing by the MELS and PERFORM CORs. ▪ The evaluation report must include a separate annex to answer objective 3 “Specific recommendations and lessons learned on strategies and approaches USAID/Malawi should continue in its future climate change mitigation activities.” ▪ As this annex is procurement sensitive, it will not be shared with any other entity other than USAID/Malawi. ▪ Evaluation methodology shall be explained in detail and all tools used in conducting the 11 evaluation such as questionnaires, checklists and discussion guides will be included in an Annex in the final report. ▪ Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, etc.). ▪ Evaluation findings should be presented as analyzed facts, evidence and data and not based on anecdotes, hearsay or the compilation of people’s opinions. Findings should be specific, concise and supported by strong quantitative or qualitative evidence. ▪ Sources of information need to be properly identified and listed in an annex. ▪ Recommendations need to be supported by a specific set of findings. ▪ Recommendations should be action-oriented, practical and specific, with defined responsibility for the action. The format of the final evaluation report should strike a balance between depth and length. The report will include a table of contents, table of figures (as appropriate), acronyms, executive summary, introduction, purpose of the evaluation, research design and methodology, findings, conclusions, lessons learned and recommendations. Where appropriate, the evaluation should utilize tables and graphs to link with data and other relevant information. The report should include, in the annex, any dissenting views by any team member or by USAID on any of the findings or recommendations. The report should not exceed 30 pages, excluding annexes. A second version of this report excluding any potentially procurement-sensitive information will be submitted (also electronically, in English) to Development Experience Clearinghouse (DEC) and for dissemination among implementing partners and other stakeholders. All quantitative data, if gathered, should be (1) provided in an electronic file in easily readable format; (2) organized and fully documented for use by those not fully familiar with the project or the evaluation; (3) owned by USAID and made available to the public barring rare exceptions. A thumb drive with all the data should be provided to the MELS COR and MELS Project Director. The final report will be edited and formatted by the Contractor and provided to USAID/Malawi 5 working days after the Mission has reviewed the content and approved the final revised version of the report. VII. EVALUATION TEAM COMPOSITION The team will include two international and three local consultants. The former should include specialists with the following areas of expertise: project evaluations and assessments (ideally natural resource management evaluations), forest conservation, climate change, greenhouse gas inventories, REDD+ and/or related scientific fields. The local consultants should have a background in forestry and/or natural resource conservation. Team Leader/Evaluation Expert (International). This expert will serve as Team Leader for both the FISH evaluation and the concurrent PERFORM evaluation. The Team Leader/EE will provide overall leadership for the team, and s/he will finalize the evaluation design, coordinate activities, arrange periodic meetings, consolidate individual input from team members, and coordinate the process of assembling the final findings and recommendations into a high quality document. S/he will lead the preparation and presentation of the key evaluation findings and recommendations to the USAID / Malawi team and other major partners. The TL should have the following attributes/qualifications and experience: ● S/he should have a postgraduate degree in an appropriate field, with at least 10 years of 12 international experience leading evaluation teams, ideally for natural resources management (NRM) and/or climate change projects. Relevant experience in Malawi or Eastern/Southern Africa preferred. ● S/he should have extensive experience in conducting quantitative and qualitative evaluations. ● The Team Leader/EE must be familiar with USAID regulations and systems including performance monitoring, gender policies and guidance, project management, budgeting and financial analysis, and reporting. ● Experience in international donor development program management and overseeing multiple program areas simultaneously is preferred. ● Should be experienced in preparing documents that are objective, evidence-based, and well organized. ● Excellent oral and written skills in English are required. ● A separate scope of work for this position will define its role, including tasks and LOE, in the two concurrent evaluations. The LOE shown in the table below for this position is for the combined workload. Senior GCC/REDD+ Specialist (International): The Senior GCC/REDD+ Specialist will be responsible for assessing the effectiveness of project implementation according to the M&E plan, while addressing the evaluation questions above. S/he will provide technical leadership with respect to GHG emissions/inventories and REDD+ policy implementation at both the national and local level. S/he will also provide technical leadership on the evaluation question concerning USAID’s strategic options for future climate change programming in Malawi. S/he will participate in team meetings, key informant interviews, group meetings, site visits, and draft the sections of the report relevant to his/her expertise and role in the team. S/he will also participate in presenting the report to USAID or other stakeholders and be responsible for addressing pertinent comments provided by USAID/Malawi or other stakeholders. The Senior GCC/REDD+ Specialist should have the following attributes/qualifications and experience: ● S/he will have a postgraduate degree in environment, climate change, forestry, or a related environmental science. ● S/he should have a minimum of 8 years of international experience in climate change/REDD+, including field work, with an understanding of issues related to management of GHG emissions. ● S/he should have M&E experience with at least one similar USAID project, or that of another international donor, and have familiarity with USAID regulations and systems. Evaluation Specialist (Local): The local Evaluation Specialist will participate in team meetings, key informant interviews, group meetings, site visits, and draft the sections of the report relevant to his/her expertise and role in the team. S/he also will participate in presenting the report to USAID or other stakeholders and be responsible for addressing pertinent comments provided by USAID/Malawi or other stakeholders. The local Evaluation Specialist should: ● Have a postgraduate degree in monitoring and evaluation, international development, forestry, natural resources, environment, or a related field. ● Be a Malawi national with at least 4 years of field experience in evaluation preferably with some sectoral experience in REDD+ and/or forest conservation. Logistics Specialist (Local). The local Logistics Specialist will be responsible for providing program logistics support, arranging appointments and maintaining the schedule, providing interpretation and 13 translation, and assisting with the preparation of project reports, as needed. S/he will serve both the FISH and PERFORM evaluations, and the LOE shown in the table below is for the combined workload. VIII. Existing Sources of Information The evaluation team should consult a broad range of background documents apart from project documents provided by USAID /Malawi. These should include, but are not limited to, documents on GCC, REDD+, and forest conservation, both in and outside of Malawi. The evaluation team also should review relevant GoM national strategies and policies, as well as relevant projects and strategies of other aid agencies active in Malawi. USAID, MELS, and the PERFORM project will provide the assessment team with a package of briefing materials, including: ● The agreement for PERFORM activity; ● M&E plan for PERFORM; ● Project quarterly and annual reports, work plans and management reviews developed as part of routine monitoring; ● Training reports; ● DQA reports; ● USAID/Malawi Country Development Cooperation Strategy 2014-19 (Public version); ● USAID Malawi DO:2 PMP; ● GOM INDC to UNFCCC; ● GOM National REDD+ Action Plan; ● Performance Indicator Tracking Table (PITT); ● M&E tools; ● Training & Beneficiary database. ● PERFORM Baseline Report ● USAID Evaluation Policy ● Checklist for Assessing USAID Evaluation Policy Conflict of Interest. All evaluation team members will provide a signed statement attesting to a lack of conflict of interest, or describing an existing conflict of interest relative to the project being evaluated. USAID will provide the conflict of interest forms. See Annex 1 for the Template. 14 IX. Level of Effort (LOE) of Study Team by Task Deliverables Below is an estimate of the evaluation level of effort (LOE). Level of Efforts of Team Members by Task Deliverables Task/Deliverable Duration / LOE in Days Team Leader / EE Sr. GCC/REDD+ Specialist Evaluation Specialist Logistics Specialist Review background documents and home￾based preparation work 5 5 5 2 Travel to Malawi 2 2 0 5 Team planning meeting and meeting with USAID 2 2 2 0 Development of Evaluation Work Plan (concurrent with document review and initial meetings) 2 2 2 2 Development of data collection instruments 2 2 2 7 Information and data collection. Includes interviews with key informants (stakeholders and USAID staff) and site visits 14 14 14 0 Discussion, analysis, and preliminary draft evaluation report in country including discussion with USAID 5 5 5 0 Debrief meetings with USAID (preliminary draft report due to USAID) 1 1 1 0 Debrief meetings with key stakeholders 1 1 1 2 Team Leader meets with Technical Specialists and USAID to synthesize findings/discussion 1 1 1 1 Depart Malawi/Travel to U.S. 2 2 0 1 Finalization of draft and internal review (IBTCI) remote 4 4 4 0 USAID provides comments on draft report 0 0 0 0 Team revises draft report and submits final to USAID (out of country) 10 4 4 0 USAID completes final review 0 0 0 0 Editing and formatting of report 0 0 0 0 Total Estimated LOE 51 45 41 20 X. Scheduling and Logistics Funding and Logistical Support. USAID/Malawi’s MELS project will be responsible for all off-shore and in-country administrative and logistical support, including identification and fielding appropriate local staff. They will take care of arranging and scheduling meetings, international and local travel, hotel 15 bookings, working/office spaces, computers, printing, and photocopying. The Logistics Specialist will arrange field visits, local travel, hotel, and appointments with stakeholders and provide translation services. Scheduling (see attached table). Work is to be carried out over a period of approximately 8 weeks (9 weeks for the team Leader because of his dual responsibilities). At this point in time, we anticipate that the evaluation would begin on/about May 1st, with work in Malawi commencing on/about May 10th, field work completed in early June and final report and close out concluding end-June. The anticipated May 1 start date provides over one month to finalize, clear the SOW, recruit and approve the consultants, and complete background research prior to arrival. See Annex-1 for Malawi Mission Holiday Schedule. A six-day work week (Monday-Saturday) is authorized for the evaluation team while in Malawi, however, no overtime or premium pay is authorized. The evaluation team will submit a work plan with timeline as part of the evaluation methodology proposal and develop a GANTT chart displaying the time periods during which activities occur. Team mobilization will include: travel approval; airline tickets; visa; lodging; work facility and vehicle transport arrangements; dates for meetings with USAID/Malawi SEG staff and key contacts; in-country travel agenda; and accommodations. 16 ANNEX 2. EVALUATION DESIGN AND METHODS Type of Answer/Evidence Needed Description, Comparison, Cause and Effect (and notes on any special requirements or sources of data) Methods for Data Collection Records, Structured Observation, KIIs, Mini-Survey Sampling or Selection Approach (if one is needed) Data Analysis Methods Frequency Distributions, Trend Analysis, Cross￾Tabulations, Content Analysis Method Data Source Evaluation Question 1: How well is PERFORM progressing toward its high-level objectives of measurably advancing national REDD+ readiness, reducing deforestation relative to business as usual, and creating low-emission livelihoods opportunities? 1. Evidence from project records to GOM goals tracking data available across four official counterparts and Focal Points to UNFCCC and UN-REDD+; 2. Relevant data or stakeholder feedback on major factors influencing achievement or non-achievement. 1. Research/data collection from secondary sources; 2. Interviews with central government officials at various levels; 3. Interviews with government officials at the district level; 4. Interviews with civil society (opinion leaders and NGOs/CSOs); 1. Ministry of Natural Resources, Energy and Mining (MNREM) responsible for environmental matters in Malawi; 2. Department of Forestry (DOF); 3. Environmental Affairs Department (EAD); 4. Local councils, community-based organizations and special interest groups; 5. Secondary sources including as available: Official GoM reporting and quantitative data collected by PERFORM, as well as other USAID, Development Partners, IP or academic assessments or studies; 6. Secondary sources will also consider as may be available, possibly including assessments or reporting at the regional level; overlapping sector-focused data such as agriculture (e.g. ag extension), planning (e.g. land use), Health (e.g. household surveys). 1. Data from secondary sources must be applicable to PERFORM´s current total period of performance; 1. Analysis and comparison of available data on deforestation reduction across a useful and relevant period of time. Period may not necessarily cover total elapsed LOP but should be comparable to periods determined for baseline data); 2. Comparison of state of advancement of policy of REDD+ as per UN REDD readiness roadmap. 3. Context specific discussion of low emissions livelihoods policy as reflected in PERFORM interventions. How (if any) has the needle moved 17 Type of Answer/Evidence Needed Description, Comparison, Cause and Effect (and notes on any special requirements or sources of data) Methods for Data Collection Records, Structured Observation, KIIs, Mini-Survey Sampling or Selection Approach (if one is needed) Data Analysis Methods Frequency Distributions, Trend Analysis, Cross￾Tabulations, Content Analysis Method Data Source Evaluation Question 1a: How effectively has PERFORM supported the GOM’s national climate change goals including the development of a GHG inventory, implementing its National REDD+ Action Plan, and developing and implementing its Intended Nationally Determined Contribution (INDC) under the UN Framework Convention on Climate Change? 1. Evidence of support (objectives/results) and comparison of progress reported by PERFORM to milestones & indicator targets; 2. Data and stakeholder feedback collected on major factors influencing achievement or non￾achievement. 3. Relevant data or stakeholder feedback on major factors influencing achievement or non￾achievement. 1. Research/data collection from secondary sources; 2. Interviews with central government officials at various levels; 3. Interviews with government officials at the district level; 4. Interviews with civil society (opinion leaders & NGOs/CSOs); 1. Ministry of Natural Resources, Energy and Mining (MNREM) responsible for environmental matters in Malawi; 2. Department of Forestry; 3. Secondary sources include: Official GoM reporting and quantitative data collected/reported by PERFORM, as well as any available reporting by Development Partners; 1. Data from secondary sources must be applicable to PERFORM´s period of performance; 2. KI selection and number should seek balance between different national level institutions, ensure triangulation and clearly identify risks of bias or undue influence. 1. Analysis and comparison of PERFORM work plan and reported progress against GoM reported completed milestones, completion markers and/or instruments in place; 2. Analysis and comparison of of available data on deforestation reduction across the relevant period, covered annually by PERFORM. 18 Type of Answer/Evidence Needed Description, Comparison, Cause and Effect (and notes on any special requirements or sources of data) Methods for Data Collection Records, Structured Observation, KIIs, Mini-Survey Sampling or Selection Approach (if one is needed) Data Analysis Methods Frequency Distributions, Trend Analysis, Cross￾Tabulations, Content Method Data Source Analysis Evaluation Question 1.b: Do the pilot projects at the local level effectively support the national goals, and are they achieving meaningful livelihood benefits at the site level? 1. Evidence from milestones and targets reported by PERFORM and specific relevant contributions to national goals; 2. Evidence collected by PERFORM on local livelihood benefits, threats & challenges associated with site-level targets tracked & reported by the project; Relevant data or stakeholder feedback on major factors influencing achievement or non￾achievement.; 3. Relevant data or stakeholder feedback on major factors influencing achievement or non￾achievement. 1. Research/data collection from secondary sources; 2. Interviews with central government officials at various levels; 3. Interviews with government officials at the district level; 4. Interviews with civil society (opinion leaders and NGOs/CSOs); 1. Key partner institutions including the regional and district forestry offices, local forestry organizations, the district assemblies, Zonal Representatives, the Traditional Authorities, Natural Resource Committees (NRCs) (at Group Village level), Group Village Heads and Village Heads. 2. Select primary sources in terms of their ability to address one or more of four areas of improvement: 1. Strengthening resiliency to climate change (IR 2.1), 2. Adding value to agricultural production (IR 2.2), 3. Better nutrition (IR 2.3), and 4. expanding agricultural trade (IR 2.4) 1. Consider any differences in demographic, social and economic makeup of target populations in the North and South; 2. Consider a limited (feasible) focus, covering both North & South, on specific agricultural value chains linked to low emissions development. 1. Content and trend analysis of data on performance of pilots as per milestones, targets and contribution to national goals; 2. Content and trend analysis of data on performance of pilots that compares outcomes across the selected districts; 3. Cross-cutting focus on gender relations and their impact on capacity building and participation in decision-making. 19 Type of Answer/Evidence Needed Description, Comparison, Cause and Effect (and notes on any special requirements or sources of data) Methods for Data Collection Records, Structured Observation, KIIs, Mini-Survey Sampling or Selection Approach (if one is needed) Data Analysis Methods Frequency Distributions, Trend Analysis, Cross￾Tabulations, Content Analysis Method Data Source Evaluation Question 2: What lessons and implications does PERFORM offer for REDD+ readiness scaling up and likely impacts on REDD+ outcomes? This can also include a synthesis of lessons from the REDD+ Readiness Program. 1. Data that clarifies critical factors in scaling-up identified by PERFORM interventions; 2. Data should reveal specific implications/limitations of scaling up and income generation strategies addressed by PERFORM interventions; 1. Research/data collection from secondary sources; 2. Interviews with central government officials working at the district levels; 3. Interviews with civil society (opinion leaders) and special interest groups; 4. Group interviews in Lilongwe and selected sites (North/South); 1. Ministry of Natural Resources, Energy and Mining (MNREM) responsible for environmental matters in Malawi; 2. Department of Forestry; 3. Key partner institutions including the regional and district forestry offices, local forestry organizations, the district assemblies, and the Traditional Authorities, Group Village Heads 1. Consider Group Interviews to enable a better stakeholder balance (age/gender/education) in the feedback collected. 1. Content analysis on processes covering qualitative information compared against reported outcomes; 2. Frequencies on coded (keyword driven) responses from individual and group interviews; 3. Analysis on registered perception and behavior (Individual/Group interviews) validated and enhanced by key informants; 20 Type of Answer/Evidence Needed Description, Comparison, Cause and Effect (and notes on any special requirements or sources of data) Methods for Data Collection Records, Structured Observation, KIIs, Mini-Survey Sampling or Selection Approach (if one is needed) Data Analysis Methods Frequency Distributions, Trend Analysis, Cross￾Tabulations, Content Analysis Method Data Source Evaluation Question 3: How sustainable are the efforts at the local and national level? Are GOM officials and local communities likely to be able to continue these programs on their own when PERFORM ends? 1. Evidence to support comparisons of PERFORM-supported interventions to other similar ongoing work; 2. Evidence to support the discussion of both triggers and hindrances to non-USAID funding of these activities; 3. Evidence to enable a comparative mapping of these efforts (actual or projected) in terms of their focus and coverage of PERFORM activities to evidence gaps and voids or gains and extensions; 1. Research/data collection from secondary sources; 2. Interviews with central government officials working at the district levels; 3. Interviews with civil society (opinion leaders) and special interest groups; 4. Group interviews in Lilongwe and selected sites (North/South); 1. Ministry of Natural Resources, Energy and Mining (MNREM) responsible for environmental matters in Malawi; 2. Department of Forestry; 3. Key partner institutions including the district forestry offices, local forestry organizations, the district assemblies, and the Traditional Authorities, Group Village Heads and Village Heads. 4. Other development partners 1. To the extent possible, gender-disaggregated data will be linked to observed participation of women in leadership/decision￾making roles associated with increased sustainability 2. Group Interviews will be used where feasible to enable a better stakeholder balance (age/gender/education) in the feedback collected. 1. Content analysis on processes (mostly qualitative information provided by project staff compared against reported outcomes; 2. Analysis on registered perception and behavior validated and enhanced by key informants 21 Type of Answer/Evidence Needed Description, Comparison, Cause and Effect (and notes on any special requirements or sources of data) Methods for Data Collection Records, Structured Observation, KIIs, Mini￾Survey Sampling or Selection Approach (if one is needed) Data Analysis Methods Frequency Distributions, Trend Analysis, Cross￾Tabulations, Content Analysis Method Data Source Evaluation Question 4: How well is PERFORM integrated with: a) other USAID initiatives (e.g., Feed the Future): and b) other DPs efforts in GCC and forest conservation? 1. Data should support associations of PERFORM-supported interventions to relevant (complementary or supplementary) ongoing work, including USAID and other DP funding; 2. PERFORM outcomes in terms of the USAID/Malawi 3-C approach: Co-locating, Coordinating & Collaborating. 1. Research/data collection from secondary sources; 2. Interviews with central government officials working at the district levels; 3. Interviews with civil society (opinion leaders) and special interest groups; 4. Group interviews in Lilongwe and selected sites (North/South); 1. Ministry of Natural Resources, Energy and Mining (MNREM) responsible for environmental matters in Malawi; 2. Department of Forestry; 3. Key partner institutions including the regional and district forestry offices, local forestry organizations, the district assemblies, and the Traditional Authorities, Group Village Heads and Village Heads. 1. Gender-disaggregated data will be examined to identify participation of women in leadership/decision￾making roles associated with increased sustainability; 2. Consider Group Interviews to enable a better stakeholder balance (age/gender/education) in the feedback collected. 1. Content analysis on processes (mostly qualitative information provided by project staff compared against reported outcomes; 2. Trend analysis on registered perception and behavior validated and enhanced by key informants. 22 Type of Answer/Evidence Needed Description, Comparison, Cause and Effect (and notes on any special requirements or sources of data) Methods for Data Collection Records, Structured Observation, KIIs, Mini-Survey Sampling or Selection Approach (if one is needed) Data Analysis Methods Frequency Distributions, Trend Analysis, Cross￾Tabulations, Content Analysis Method Data Source Evaluation Question 5: How are gender issues integrated into the activity? Have women been able to participate fully and benefit from it, including from income-generating activities? 1. Data available (emphasis on project-collected data) should provide an understanding of key gender relations affecting male and female participation in six domains that guide sampling & selection; 2. Evidence from data on gender related challenges (consider applicable gaps identified in participation, remuneration & advancement) collected as a baseline and then tracked & reported by the project; 3. Relevant secondary sources will be examined to enhance the analysis on major factors influencing achievement or non-achievement. 1. Research/data collection from secondary sources; 2. Interviews with central government officials working at the district levels; 3. Interviews with civil society (opinion leaders) and special interest groups; 4. Group interviews in Lilongwe and selected sites (North/South); 1. Key partner institutions including the regional and district forestry offices, local forestry organizations, the district assemblies, and the Traditional Authorities, Group Village Heads and Village Heads. 2. A (context-specific) balance of sources in the visited districts where pilot site- based activities are implemented. 3. Data/context from secondary sources covering any of the three districts. 1. Sampling/selection must consider any differences in demographic, social and economic makeup of target populations in the North and South; 2. Sampling/selection should best inform a common set of domains of gender analysis; 3. KI selection and number should seek balance between official views, NGO advocates and a grassroots perspective. 1. Content and trend analysis of data on performance of pilots as per key gender relations affecting male and female participation in six domains that guide sampling & selection; 2. Content and trend analysis of data on performance of pilots that compares outcomes across the three districts; 3. Cross-cutting focus on gender relations impact on capacity building. 23 Type of Answer/Evidence Needed Description, Comparison, Cause and Effect (and notes on any special requirements or sources of data) Methods for Data Collection Records, Structured Observation, KIIs, Mini-Survey Sampling or Selection Approach (if one is needed) Data Analysis Methods Frequency Distributions, Trend Analysis, Cross￾Tabulations, Content Method Data Source Analysis Evaluation Question 5a: For cross cutting subjects such as gender and capacity building, how effectively do they contribute to PERFORM’s intended results? 1. Evidence of outcomes collected by PERFORM on capacity levels of target communities and institutions; 2. Evidence of outcomes data collected by PERFORM on gender dynamics and challenges (focus on applicable gaps identified in participation, remuneration & advancement), also tracked & reported by the project; 3. Evidence that links gender & capacity building data/analysis to the consideration of major factors influencing achievement or non￾achievement; 4. Relevant secondary sources will be examined to enhance the analysis on major factors influencing achievement or non￾achievement. 1. Research/data collection from secondary sources; 2. Interviews with central government officials working at the district levels; 3. Interviews with civil society (opinion leaders) and special interest groups; 4. Group interviews in Lilongwe and selected sites (North/South); 1. Ministry of Natural Resources, Energy and Mining (MNREM) responsible for environmental matters in Malawi; 2. Department of Forestry; 3. Key partner institutions including the district forestry offices, local forestry organizations, the district assemblies, and the Traditional Authorities, Group Village Heads and Village Heads; 4. Data/context from secondary sources covering any of the three districts. 1. Consider any differences in demographic, social and economic makeup of target populations in the North and South. 1. Content and trend analysis of data on performance of pilots as per milestones, targets and contribution to national goals; 2. Content and trend analysis of data on performance of pilots that compares outcomes across the three districts; 3. Analysis on the focus on gender relations and their impact on capacity building and participation in decision-making. 24 ANNEX 3. DATA COLLECTION TOOLS AND RESOURCES 3.1. Questionnaire Tracker Evaluation Questions for Mid-Term Evaluation USAID PERFORM Project Question No. Questions Status (C / NC) Q1a How effectively has PERFORM supported the GOM’s national climate change goals including the development of a GHG inventory, implementing its National REDD+ Action Plan, and developing and implementing its Intended Nationally Determined Contribution (INDC) under the UN Framework Convention on Climate Change? SQ1 Have you been involved or are you aware of the national GHG inventory? Do you know if the institutions that created it received any assistance? If so, from whom? (Steer to USAID, and ask about PERFORM if they don’t mention it) SQ2 Have you been involved or are you aware of the National REDD+ Action Plan? Do you know if the institutions that created it received any assistance? If so, from whom? (Steer to USAID, and ask about PERFORM if they don’t mention it) SQ3 Have you been involved or are you aware of the INDC? Do you know if the institutions that created it received any assistance? If so, from whom? (Steer to USAID, and ask about PERFORM if they don’t mention it) Q1b Do the pilot projects at the local level effectively support the national goals, and are they achieving meaningful livelihood benefits at the site level? SQ1 Are you aware of the USAID PERFORM project sites in Machinga/Mzimba/Ntchisi? SQ2 Do you know anything about the details of the work that USAID PERFORM has done in Machinga/Mzimba/Ntchisi? SQ3 Can you speak about the new economic opportunities brought by PERFORM to the local residents in these areas? SQ4 Are you aware of forest inventory work being done by USAID PERFORM at these sites? 25 Q2 What lessons and implications does PERFORM offer for REDD+ readiness scaling up and likely impacts on REDD+ outcomes?  This can also include a synthesis of lessons from the REDD+ Readiness Program. SQ1 Related to the work at the site level, do you think that it could be replicated in neighboring or other districts? Are you aware of interest in any such replication? Q3 How sustainable are the efforts at the local and national level?  Are GOM officials and local communities likely to be able to continue these programs on their own when PERFORM ends? SQ1 Do you think that the work being supported by PERFORM will continue when in 2.5 years, the project ends? If so, how? Are there any institutions in particular that you think could best continue the work? SQ2 Have you incorporated any lessons learned from PERFORM into your planning process? If so, what? SQ3 (For local) Are you aware of new economic opportunities in this area supported by PERFORM? Do you think that they will continue after the project ends? If so/if not so, why? Q4 How well is PERFORM integrated with: a) other USAID initiatives (e.g., Feed the Future): and b) other DPs efforts in GCC and forest conservation? SQ1 Besides PERFORM, are there any other USAID or other donor projects in this area? Do you know if they work together? If so, do you feel that their working together has provided additional benefits to the local communities. Q5 How are gender issues integrated into the activity?  Have women been able to participate fully and benefit from it, including from income‐ generating activities? SQ1 Do you know who participates in the work being done by PERFORM in the project sites? What kind of work do the participants do? Is this work done only by men/women or both? Mostly which? SQ2 Are the participants mostly men? If women are also involved, do you know roughly what is the number of men versus women? SQ3 Of the participants who earn more money through the work of PERFORM, is there a difference between the money earned by the work that men and women do? 26 Q5a For cross cutting subjects such as gender and capacity building, how effectively do they contribute to PERFORM’s intended results? SQ1 Has your institution or community received any training from PERFORM? How has it helped? NOTE Q1 : Question 1 SQ1 : Sub Question 1 C : Checked NC : Not Checked 27 3.2. Focus Group Discussion Guide 28 29 3.3 Key Informant Note Taking Template 3.4 Key Informant Tracker Template 3.4 KI Master Template 3.5 Process Documentation Resources (LINK TO) Meeting: (Date, Time, Name of Individual and Institution) Text Evaluation Question Keyword 30 3.6 Individual & Group Questionnaire 31 32 ANNEX 4. CONFLICT OF INTEREST FORMS FOR TEAM MEMBERS 33 34 35 ANNEX 5. STATEMENT OF DIFFERENCES There are no statements of differences from any of the evaluation team members. 36 ANNEX 6. TOC COMPARISONS EVALUATION QUESTION1a: How effectively has PERFORM supported the GOM’s national climate change goals including the development of a GHG inventory, implementing its National REDD+ Action Plan, and developing and implementing its Intended Nationally Determined Contribution (INDC) under the UN Framework Convention on Climate Change? To what extent have the PERFORM outputs caused the observed outcomes? TOC elements considered:  local and national governance conditions are improved  Sustainable and scalable land management systems based on improved science and technologies Observed Outputs Linked to TOC  Capacity building has involved public and private sector, including academic and civil society targets  Direct institutional support alongside training with cross-sector and sector-specific focus  Multiple assistance mechanisms contribute to improvement of governance conditions and advancement of an enabling environment  Support to the RExG in accelerating the timeline, compared to that in the MRPAP, for developing the Malawi’s National REDD+ Strategy  Engagement of multiple ministries to collect data from different emissions sources  Support to EAD on the preparation of the INDC, through the development of an Options Assessment Report, and on determining the investments needed to support different GHG mitigation scenarios  Building on the foundation established by MRRP and UN-REDD to advance REDD+ readiness  Local scale forest inventories and carbon stock assessments, and capacity building to operationalize MRPAP and improve policy and legal frameworks Outcome 1: REDD+ Readiness Advanced Sub-Outcomes 1.1 Capacities for REDD+ data collection, analysis, use increased 1.2 Core priorities of the Malawi REDD+ Strategy operationalized 1.3 REDD+ -relevant policy/legal frameworks improved Outcome 3: Low Emission Development Capacities Improved Sub-Outcomes 3.1 GHG data management capabilities improved 3.2 A GOM-owned GHG inventory management system operationalized 3.3 Capacity for implementing mitigation options increased Is it reasonable to conclude that PERFORM has made a difference to the problem? Challenges Affecting PERFORM/National Goals Positive/Negative Factors for Achieved Difference GOM has been slow in advancing REDD+ readiness Positive: Added resources have supported visible progress of Malawi’s National REDD+ Strategy led by the RExG Negative: Low levels of capacity across GOM institutions to effectively obtain, analyze and use land use data, and create geospatial products EAD is regarded as a challenging partner to work with, associated to public-sector efforts to secure additional resources and resulting in shifting priorities Positive: New mechanisms have contributed to improvement of governance conditions and advancing an enabling environment Negative: Lack of clarity among stakeholders about the distinction between the REDD+ Action Plan (completed) and Strategy (in progress) 37 EVALUATION QUESTION 1b. Do the pilot projects at the local level effectively support the national goals, and are they achieving meaningful livelihood benefits at the site level? To what extent have the PERFORM outputs caused the observed outcomes? TOC elements considered:  improved capacities to analyze, implement and monitor sustainable land management practices  local and national governance conditions are improved  demonstration of sustainable land management systems based on improved science and technologies  economic incentives are efficiently, equitably, and sustainably aligned through REDD+ & SES  additional resources are leveraged with other investments through public-private partnerships Observed Outputs Linked to TOC  Village clubs organized as entry points for interventions and trained in group dynamics for project implementation  Increased means at local level covering irrigation, crop/sustenance improvements, woodlots and successful introduction of cookstoves  Sensitization and capacity building focused on improved practices and specific technologies in land management and conservation  Site based mapping of deforestation and forest degradation drivers including associated hotspots  Kawandama Hills Plantations (KHP) includes sustainable charcoal as well as alternative livelihoods elements  Local forestry inventories and carbon stock assessments have been highly successful and recognized as useful by DFO, as well as national level GOM officials Outcome 2: Low emission Land Use Opportunities Increased Sub-Outcomes 2.1 Decentralized forest/land management improved 2.2 Livelihood opportunities from conservation expanded 2.3 Forest cover increased Outcome 4: Pathways for Sustainability Instituted Sub-Outcomes 4..1 Capacities of targeted partner organizations developed 4.2 PPPs facilitated 4.3 Sources of additional financial/technical support leveraged Is it reasonable to conclude that PERFORM has made a difference to the problem? Challenges Affecting PERFORM/National Goals Positive/Negative Factors Lack of stronger/effective links and accreditation between village level participatory activities and GOM to assist the advancement of decentralized management Positive: VNRMC members here and elsewhere appear to overlap closely with the village farmer “clubs” used as entry points by the project Negative: Continued low levels of integration between the DFO and VNRMC At the village level, potential income from commercializing excess production of crops does not replace the potential income from illegal charcoal sale Positive: Village-level agricultural interventions are a major contribution, especially to increased food security Negative: While some is sold in local markets, no evidence obtained suggests that increased cash earnings from any of the village interventions provided sufficient income substitution to defray that potentially earned from charcoal sales 38 EVALUATION QUESTION 2. What lessons and implications does PERFORM offer for REDD+ readiness scaling up and likely impacts on REDD+ outcomes? This can also include a synthesis of lessons from the REDD+ Readiness Program. To what extent have the PERFORM outputs caused the observed outcomes? TOC elements considered:  improved capacities to analyze, implement and monitor sustainable land management practices  local and national governance conditions are improved  demonstration of sustainable land management systems based on improved science and technologies  economic incentives are efficiently, equitably, and sustainably aligned through REDD+ & SES  additional resources are leveraged with other investments through public-private partnerships Observed Outputs Linked to TOC  Support to a wide range of forestry related policies such as the National Charcoal Strategy and the National Forest Landscape Restoration Assessment and Strategy (NFLRS) points to further mainstreaming and scaling up/out of REDD+ readiness efforts  Support for the launch of the NFMS Roadmap in 2015  Improvements in stakeholder capacity to obtain and analyze land use and land cover data  The local forest inventories and carbon stock assessments have made a significant contribution to the NFMS  Support to develop a GIS laboratory at the College of Forestry and Wildlife (Dedza) Outcome 2: Low emission Land Use Opportunities Increased Sub-Outcomes 2.1 Decentralized forest/land management improved 2.2 Livelihood opportunities from conservation expanded 2.3 Forest cover increased Outcome 4: Pathways for Sustainability Instituted Sub-Outcomes 4..1 Capacities of targeted partner organizations developed 4.2 PPPs facilitated 4.3 Sources of additional financial/technical support leveraged 39 Is it reasonable to conclude that PERFORM has made a difference to the problem? Challenges Affecting PERFORM/National Goals Positive/Negative Factors Absence of a cohesive and effective data repository strategy and limited use of open data platforms (e.g. MASDAP) to promote/enhance the use of the data and digital products Positive: New and enhanced flows of land cover/use data have been supported and promoted while engaging DoF, FRIM and academic institutions, thus bolstering analytical abilities that allow GOM to move towards a baseline forest inventory key to REDD+ readiness Positive: Digital products stemming from enhanced capabilities have already been used by key stakeholders Negative: Limitations of the national context regarding reliable electricity supply and sufficiently adequate internet connectivity Negative: It remains unclear whether the use of the data and digital products can be sustainably increased beyond the LOP, contributing to the goal of institutionalizing and maintaining capacity Assessed need to further demonstrate specific REDD+ principles, themes and goals, concurrently incorporating community-based consultation and support Positive: Active engagement of RExG and its constituent bodies, has provided a venue for outreach and ensuring buy-in from a wide range of stakeholders Positive: Direct involvement and support to academic institutions is helping to ensure a future cadre of trained experts in REDD+ related issues Negative: REDD+ advancement has faced misunderstandings and frustrations among those stakeholders who are mainly interested in revenue generation aspects Negative: Lack of clarity even among key stakeholders about the distinction between the REDD+ Action Plan (completed) and Strategy (in progress) has caused frustration, especially among those interested in going beyond the no-regrets benefits of REDD+, as in tangible on-the-ground activities 40 EVALUATION QUESTION 3. How sustainable are the efforts at the local and national level? Are GOM officials and local communities likely to be able to continue these programs on their own when PERFORM ends? To what extent have the PERFORM outputs caused the observed outcomes? TOC elements considered:  improved capacities to analyze, implement and monitor sustainable land management practices  local and national governance conditions are improved  demonstration of sustainable land management systems based on improved science and technologies  economic incentives are efficiently, equitably, and sustainably aligned through REDD+ & SES  additional resources are leveraged with other investments through public-private partnerships Observed Outputs Linked to TOC  Enhanced capabilities to meet additional commitments are strategically hosted inside EAD  Sustained and ongoing collaboration with local government counterparts  Hands-on trainings on technologies introduced at the local level, enabling potentially enhanced use and ownership of appropriate technologies  Anticipated continuation of agricultural interventions through FtF MISST and CIP Outcome 2: Low emission Land Use Opportunities Increased Sub-Outcomes 2.1 Decentralized forest/land management improved 2.2 Livelihood opportunities from conservation expanded 2.3 Forest cover increased Outcome 4: Pathways for Sustainability Instituted Sub-Outcomes 4..1 Capacities of targeted partner organizations developed 4.2 PPPs facilitated 4.3 Sources of additional financial/technical support leveraged Outcome 5: CDCS priorities for integration and institutional strengthening enhanced Is it reasonable to conclude that PERFORM has made a difference to the problem? Challenges Affecting PERFORM/National Goals Positive/Negative Factors Funding by multiple international donor agencies overlaps USAID contributions as an intermediate step and sustained GOM progress will require considerable investment by the public sector Positive: Watershed management is being incorporated into the longer￾term GOM efforts under the National Resilience Plan Positive: GHG Inventory is expected to be included in the national budgets for EAD and climate change and forestry were being prioritized in the upcoming Malawi Growth and Development Strategy (MGDS 2017-2022) Negative: Without parallel structured commitments, added technical and financial resources from development partners may slow down increased GOM investments from internal sources No evidence to support that village clubs can evolve or replicate independently beyond the specific communities selected and supported by USAID Positive: Cookstoves appeared to be the most enduring technology, and replication of crops show some promise of sustainability, as does the pass￾on approach for chickens Negative: None of the activities supported at village level provide levels of income generation to support sustainability and expansion through feasible community-based financing and/or independent investments 41 EVALUATION QUESTION 4: How well is PERFORM integrated with: a) other USAID initiatives (e.g., Feed the Future): and b) other DPs efforts in GCC and forest conservation? To what extent have the PERFORM outputs caused the observed outcomes? TOC elements considered:  improved capacities to analyze, implement and monitor sustainable land management practices  local and national governance conditions are improved  demonstration of sustainable land management systems based on improved science and technologies  economic incentives are efficiently, equitably, and sustainably aligned through REDD+ & SES  additional resources are leveraged with other investments through public-private partnerships Observed Outputs Linked to TOC  Collaboration with multiple international agencies that significantly advanced the National Forest Landscape Assessment, the development and submission of the national INDC and the GHG Inventory  Collaboration that ensured cost-effective geographic coverage with the Shire River Basin Management Program (SRBMP) in Machinga District  Integration with a number of USAID funded activities in the Machinga District, benefitting program content and coverage Outcome 2: Low emission Land Use Opportunities Increased Sub-Outcomes 2.1 Decentralized forest/land management improved 2.2 Livelihood opportunities from conservation expanded 2.3 Forest cover increased Outcome 4: Pathways for Sustainability Instituted Sub-Outcomes 4..1 Capacities of targeted partner organizations developed 4.2 PPPs facilitated 4.3 Sources of additional financial/technical support leveraged Outcome 5: CDCS priorities for integration and institutional strengthening enhanced Is it reasonable to conclude that PERFORM has made a difference to the problem? Challenges Affecting PERFORM/National Goals Positive/Negative Factors Funding by multiple international donor agencies overlaps USAID contributions that may cause unintended and sometimes harmful redundancies Positive: Planned avoidance of duplication was observed with the World Bank SRBMP in Machinga Positive: Planned implementation conducted in coordination with FtF MISST and CIP Collaboration needs to move beyond traditional stakeholders and consider expanded approaches that tap other resource pools, leveraging the success achieved by interventions to beyond the target sites. 42 EVALUATION QUESTION 5: How are gender issues integrated into the activity? Have women been able to participate fully and benefit from it, including from income-generating activities? For cross cutting subjects such as gender and capacity building, how effectively do they contribute to PERFORM’s intended results? To what extent have the PERFORM outputs caused the observed outcomes? TOC elements considered:  improved capacities to analyze, implement and monitor sustainable land management practices  demonstration of sustainable land management systems based on improved science and technologies  economic incentives are efficiently, equitably, and sustainably aligned through REDD+ & SES Observed Outputs Linked to TOC  Women are actively involved and even held leadership roles in national level efforts supported by PERFORM, driving key elements of national REDD+ readiness  Capacity building at the local level has effectively involved women participation  Generation of new income through added productivity that directly (and predominantly) engages women participation in the target sites  Additional cash derived from income-generation activities is applied to household expenses such as school fees, medical services and transportation Outcome 2: Low emission Land Use Opportunities Increased Sub-Outcomes 2.1 Decentralized forest/land management improved 2.2 Livelihood opportunities from conservation expanded 2.3 Forest cover increased Outcome 4: Pathways for Sustainability Instituted Sub-Outcomes 4..1 Capacities of targeted partner organizations developed 4.2 PPPs facilitated 4.3 Sources of additional financial/technical support leveraged Is it reasonable to conclude that PERFORM has made a difference to the problem? Challenges Affecting PERFORM/National Goals Positive/Negative Factors At the local level, regardless of the intensive participation of women, both in farmer clubs and in the actual implementation of project activities within the villages, decisions regarding cash income and land use remain strongly influenced or made exclusively by men. Positive: Women are clearly placed in leadership positions in terms of the group’s productive activities Negative: Women’s labor is vastly driving the implementation of local scale activities, while at the household level, their ability to benefit from them is controlled by men