1 MEASURING IMPACT – PRODUCT TITLE DOCUMENT SECTION TITLE MEASURING IMPACT US Forest Service Participating Agency Program Agreement (PAPA) Evaluation Report March 21, 2014 This publication was produced for review by the United States Agency for International Development. It was prepared by Environmental Incentives, LLC, Foundations of Success and ICF International. DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. MEASURING IMPACT CONTRACT INFORMATION This program is made possible by the generous support of the American people through the United States Agency for International Development (USAID) under the terms of its requisition number REQ￾EGAT-12-000014 (Measuring Impact) implemented by prime recipients Environmental Incentives, LLC, Foundations of Success and ICF International. The Measuring Impact project has been issued under contract number AID-OAA-C-12-00078 and supports the same program objectives as described in RFP number SOL-OAA-000050. The Measuring Impact project is funded and managed by the USAID Office of Forestry and Biodiversity, and Bureau for Economic Growth, Education and the Environment. PREPARED BY Paul Cowles (Evaluation Specialist & Lead Evaluator) & Elaine Sabourin (Compliance Expert), with support from the Measuring Impact Project Team: Environmental Incentives, Foundations of Success and ICF International SUBMITTED BY Elizabeth Lauck Environmental Incentives, LLC SUBMITTED TO Rebecca Butterfield, Contracting Officer Representative Office of Forestry and Biodiversity / Bureau for Economic Growth, Education and the Environment United States Agency for International Development FOR MORE INFORMATION Environmental Incentives, LLC 1606 20th Street, N.W. Washington, D.C. 20009 E-mail: elauck@enviroincentives.com / Web site: www.enviroincentives.com DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. 3 MEASURING IMPACT – PAPA EVALUATION REPORT ACRONYMS AND ABBREVIATIONS ACRONYMS & ABBREVIATIONS AED Academy for Educational Development CCA Climate Change Adaptation CIFOR Center for International Forest Research EGSSAA Environmental Guidelines for Small-scale Activities in Africa ETD Environmental Threshold Decision ETOA Environmental Threats and Opportunity Assessment E3/FAB USAID/E3/Forestry and Biodiversity Office EU European Union FAA Foreign Assistance Act FAO Food and Agriculture Organization FEC Forest Enterprise Cluster (Brazil) FMMI Financial Management Modernization Initiative FRM Forest Resources Management Project GCC Global Climate Change GEF Global Environment Facility IAA Inter-Agency Agreement ICS Incident Command System IEE Initial Environmental Examination IPCC Intergovernmental Panel on Climate Change LEAD Low Emissions Asian Development Program LEDS Low Emissions Development Strategies METI Management and Engineering Technologies International MI Measuring Impact Project NDVI Normalized Difference Vegetation Index NGO Non-Governmental Organization NRM Natural Resource Management PAPA Participating Agency Program Agreement PMFAS Project Management and Financial Accountability System RDMA Regional Development Mission for Asia REDD Reduced Emissions from Degradation and Destruction RPP REDD Preparedness Plan SDN Specially Designated National STAR Sustainable Tourism in the Albertine Rift STEWARD Sustainable and Thriving Environments for West African Regional Development SWAMP Sustainable Wetlands Adaptation and Mitigation Program T/A Technical Assistance USAID United States Agency for International Development USDA United States Department of Agriculture USFS United States Forest Service USFS-IP US Forest Service Office of International Programs USG United States Government WB World Bank MEASURING IMPACT – PAPA EVALUATION REPORT 4 TABLE OF CONTENTS CONTENTS ACRONYMS & ABBREVIATIONS ............................................................................................... 3 EXECUTIVE SUMMARY .............................................................................................................. 5 Evaluation Methodology .............................................................................................................................. 5 Findings and Conclusions ........................................................................................................................... 5 Recommendations ...................................................................................................................................... 8 PURPOSE OF THE EVALUATION ............................................................................................10 BACKGROUND ON THE USFS/USAID RELATIONSHIP ........................................................11 METHODOLOGY .......................................................................................................................13 Evaluation Methodology ............................................................................................................................13 FINDINGS ...................................................................................................................................17 Task 1: Improving Our Understanding of How The PAPA is Being Used. ...............................................17 Task 2: Assessing the Technical, Program Management, and Financial Performance of the PAPA. ....28 Task 3: Assessing the Contribution of the USFS PAPA to the Purpose and Objectives... .....................39 CONCLUSIONS AND RECOMMENDATIONS FOR BEST PRACTICES ................................. 41 Task 4: Identifying Key Recommendations/Best Practices for the Continued Use of the PAPA ............41 ANNEXES ...................................................................................................................................46 Annex I: Evaluation Scope of Work ..........................................................................................................47 Annex II: Examples of Survey Instruments Used .....................................................................................59 Annex III: List of Persons Interviewed .......................................................................................................63 Annex IV: List of Documents Reviewed ....................................................................................................64 Annex V: Compliance Desk Review..........................................................................................................67 Annex VI: Branding Compliance Table .....................................................................................................80 Annex VIi: Database Fields .......................................................................................................................89 5 MEASURING IMPACT – PAPA EVALUATION REPORT EXECUTIVE SUMMARY EXECUTIVE SUMMARY USAID/E3/Forestry and Biodiversity Office (E3/FAB) has engaged with the Measuring Impact project (MI) to evaluate the United States Forest Service (USFS) Participating Agency Program Agreement (PAPA). The goal of the evaluation is to determine to what extent the USFS PAPA is being used and performing towards its intended objectives, and to provide recommendations that increase overall PAPA effectiveness. Inter-Agency Agreements (IAA) between USAID and the USFS date back over 32 years. The current PAPA with the US Forest Service Office of International Programs (USFS-IP), which is the subject of this evaluation, was established in 2007 and ends in 2017. This PAPA has the broadest technical range of any of the previous agreements. EVALUATION METHODOLOGY The evaluation was primarily executed through development of a funding database, selection of five priority buy-ins, a desk study of diverse documents and databases, and more than 40 interviews of agency staff. The MI evaluation team was led by Paul Cowles and included Elaine Sabourin, federal compliance expert. The evaluation was implemented using a phased approach: Phase 1: Construct a database of buy-ins. Phase 2: Analyze the database. Phase 3: Refine evaluation questions and approach. Phase 4: Select priority buy-ins and develop survey instruments. Phase 5: Collect data, analyze and report. FINDINGS AND CONCLUSIONS The findings and conclusions are structured around the key evaluation questions answered: 1. How is the PAPA being used? 2. What do PAPA buy-ins look like up close? 3. What is the quality and how effective is the T/A provided under the PAPA? 4. What is the USFS International Programs performance in managing the PAPA? HOW IS THE PAPA BEING USED? As of August 2012, there had been a total of 173 buy-ins made to the PAPA for a total obligation level of $113,753,747. The value of individual buy-ins varied from $15,577 to $6,000,000, with a mean value of $657,536. The distribution of buy-ins was skewed below the mean as shown by the statistical mode (most frequently appearing value) of $100,000. Use of PAPA has grown steadily since 2007. (See Figure 1.) Funding by Source Other Climate Change Disaster Biodiversity Funding by Beneficiary Other Local NGO Communities Host Government Figure 1: PAPA summary information related to funding source, funding magnitude and beneficiaries as of August 2012.   MEASURING IMPACT – PAPA EVALUATION REPORT 6 EXECUTIVE SUMMARY Biodiversity-related funding represents the greatest proportion (53%) of the PAPA’s funding, followed by Global Climate Change (26%) funding which has been increasing steadily over time. Host country governments are the primary beneficiary of the PAPA buy-ins (46%), followed by all other beneficiary types combined (28%), then communities (14%) and local NGOs (12%). Host governments receive slightly more support from climate change funding, whereas most other groups receive support primarily through biodiversity-themed T/A. Figure 2: PAPA funding analysis by funding source and beneficiary type. WHAT DO PAPA BUY-INS LOOK LIKE UP CLOSE? PAPA buy-ins provide a wide range of services globally as represented by examples from South America, Asia and other continents. Each of these locations cited results and challenges that provide insight into the PAPA. The Brazil Forest Enterprise Cluster (FEC) project was funded through a series of buy-ins to the PAPA under which the USFS-IP manages grants to four Brazilian NGOs and provides T/A to those partners and various government entities to enhance capacity building for improved forest conservation and management. USAID/Brazil is extremely satisfied with the performance of the USFS-IP under this project.   Results achieved - Training and capacity building of personnel and the development of the 2006 Brazilian law that created the Brazilian Forest Service; IMAZON-developed forest monitoring system that is now used by government and private sector; and communities that received training in sustainable forest management practices are receiving certification. Challenges cited – Distribution of funding to grantees on a timely basis; communicating successes and accomplishments to a broader audience; and a lack of analysis of the long term impacts of training. In many ways, the USAID Regional Development Mission for Asia (RDMA) relationship with USFS-IP is representative of an ideal example of PAPA engagement. RDMA plays a strong technical lead with implementation partners and operates a range of activities, while the USFS plays an advisory and capacity building role to all. Results achieved – Efforts to reduce wildlife trafficking and improve timber trade; support to Low Emissions Asian Development (LEAD) program; improved adaptation and mitigation of impacts of climate change using Payment for Ecosystem Services. Challenges cited – Need for engagement on selection of detailers, self-promotion of USFS programs and appropriateness of the agency for support related to wildlife trafficking. 7 MEASURING IMPACT – PAPA EVALUATION REPORT EXECUTIVE SUMMARY WHAT IS THE QUALITY AND HOW EFFECTIVE IS THE T/A PROVIDED UNDER THE PAPA? Interviewees and online survey respondents are highly satisfied with the quality of USFS detailers and the technical assistance (T/A) they deliver, based on detailer experience, expertise, ability to target the specific needs of beneficiaries, and understanding of relevant natural resource issues. The few concerns about the quality of detailers were related to inexperience and inability to apply technical knowledge to the local context. Similarly, respondents were satisfied with the effectiveness of USFS detailers and the T/A they provide. Respondents feel that USFS provides important and relevant inputs to USAID programs that result in better capacity among stakeholders. While rare, the evaluation did identify three inter-related areas of dissatisfaction among USAID staff: misalignment between USAID expectations and needs, and USFS expertise; the drive to generate more buy-ins and expand USFS-IP programs; and personality conflicts between USFS and USAID or partner staff. WHAT IS THE USFS INTERNATIONAL PROGRAM’S PERFORMANCE IN MANAGING THE PAPA? During the evaluation process five themes emerged related to USFS-IP’s performance in managing the PAPA.     Selecting effective detailers – Based on respondent feedback, it is clear that successful PAPA T/A depends on selecting, preparing and managing detailers effectively, particularly in a way that accounts for local contexts. Long term USFS-IP staff placement overseas – Placing or hiring USFS-IP staff overseas to help manage PAPA buy-ins is viewed, for the most part, very positively by USAID Mission staff and partners. These USFS-IP staff are considered key technical resources who help simplify communication between USFS-IP and Mission staff and partners, which improves interaction and collaboration in the development and implementation of buy-in scopes of work. Some USAID Missions did express concerns, however, that these staff are for the most part consultants, not long term USFS (i.e. USG) employees, which can complicate their roles within Missions. Perceived value by Missions – Mission personnel were satisfied with the way funds were expended, and believed PAPA buy-ins to be a very good value. The funds were used according to the buy-in SOW, unless mutually agreed upon changes were made. Mission personnel also appreciated the relative ease of use and management of a PAPA buy-in, compared to a competitive procurement, which raises the concern that PAPA is used in lieu of procurement even in instances where NGOs or contractors may be more appropriate for managing activities. Financial Management – USFS uses financial management databases and reports to track PAPA financial information at the transaction level, the project level and the buy-in level. In accordance with the PAPA agreement, USFS provides quarterly financial reports, for each buy-in, to E3/FAB. OTHER FINDINGS AND CONCLUSIONS In addition to the primary questions, the evaluation revealed several other findings and conclusions related to PAPA use, use of standard and custom indicators, and gender issues.    Based on ADS 103.3.8.4, IAAs are interpreted to include “inherently government functions or technical assistance.” Nearly everything that the USFS provides under the PAPA is some form of “technical assistance.” The Foreign Assistance Act stipulation that T/A should not be competitive with the private enterprise is more problematic because it is not clear that this rationale is consistently met when establishing buy-ins. All of the priority buy-ins examined are producing data on USAID standard indicators and Missions are satisfied with reporting on these indicators and data quality. However, improvements could be made in the collection of monitoring and evaluation (M&E) data beyond the standard indicators. Only the STEWARD buy-in (from USAID/West Africa) demonstrates significant positive outcomes for women based on USFS-IP Performance Management Plan (PMP) data. All USFS-IP managers stated that it is a priority for them to ensure women are included in workshops and training. RDMA support included developing a gender strategy to ensure women and disadvantaged groups received equitable benefits from buy-in activities. The STEWARD Program is currently finalizing a gender strategy. MEASURING IMPACT – PAPA EVALUATION REPORT 8 EXECUTIVE SUMMARY RECOMMENDATIONS Recommendations for improving compliance and managerial performance became clear based on the evaluation findings and conclusions. Each finding is presented in the context of its related issue, the responsible party and timeframe (current vs future PAPA agreements). IMPROVING COMPLIANCE WITH THE PAPA AND USAID REGULATIONS Although USFS implementation of PAPA buy-ins complies with USAID rules and regulations for the most part, there were several areas where improvements could be made, such as the timing and content of program reports, ensuring compliance with the audit provisions of the PAPA, increasing awareness of USFS personnel about the environmental compliance requirements of the PAPA, and ensuring that grantees are fully informed about branding of communications products financed with USAID funds. The recommendations in Table 1 below are intended to address existing weaknesses under this theme. Table 1: Improving Compliance: Recommendations, Responsibility and Timeframe # ISSUE RECOMMENDATION RESPONSIBILITY TIMEFRAME USFS Action USAID Action Current PAPA Future PAPA 1 PAPA template out-of-date Update PAPA template and Standard Provisions (p. 41) X X 2 Compliance Monitoring Develop a compliance tool to monitor each key area (p. 41) X X 3 Reporting provisions not clear and concise Revise reporting provisions in the PAPA to clearly define timing and content of reports (p. 41) X X 4 Content of annual program reports Develop a template for annual reports (p. 41) X X X 5 Non-compliance with audit provisions Prepare and implement a plan of action for completing all required financial audits (p. 41) X X 6 Non-compliance with audit provisions Revise the USFS grant template to lower the audit threshold to $300,000 in annual funding (p. 41) X X 7 Knowledge of IEE conditions Raise USFS staff awareness of PAPA IEE conditions(p. 42) X X X 8 Branding compliance by sub￾grantees and other 3rd parties Develop USAID branding guidelines for USFS activities under the PAPA (p. 42) X X 9 PAPA branding provisions not current Update the branding provisions to include references to the USAID Graphic Standards Manual (p. 42) X X 10 Anti-terrorism screening clause not up-to-date Update the anti-terrorism clause in the PAPA template (p. 42) X X 11 Financial reports not reconciled to each other Conduct quarterly variance analysis among various financial reports to ensure accuracy (p. 42) X X 12 Field finance risk mitigation Implement field operations manual best practices and procedures in PAPA-funded offices (p. 42) X X 13 Unexpended advances Carry out the November 2013 plan for expending advances by the second quarter of FY2015 (p. 43) X X 9 MEASURING IMPACT – PAPA EVALUATION REPORT EXECUTIVE SUMMARY IMPROVING THE TECHNICAL AND MANAGERIAL PERFORMANCE OF THE PAPA USAID staff are overall quite satisfied with USFS-IP technical performance under the PAPA. The recommendations described in Table 2 below are designed to address current challenges and improve performance over the long term by addressing both technical and managerial challenges. Table 2: Improving Technical and Managerial Performance: Recommendations, Responsibilities and Timeframe # ISSUE RECOMMENDATION RESPONSIBILITY TIMEFRAME USFS Action USAID Action Current PAPA Future PAPA 14 Selection of detailers. Continue to use and adapt internal best practices. Explore potential for training/orientation that would confront detailers with the realities of assignments in developing countries. (p. 43) X X X 15 Lack of monitoring and evaluation of outcomes and impacts of T/A. Develop mechanism, either internal to USFS-IP or externally contracted by USAID, to carry out structured monitoring and evaluation of T/A provided under the PAPA. (p. 44) X X X 16 Inadequate planning and coordination at Mission level. Develop a best practice model detailing a more collaborative form of planning between USAID Missions, USFS and implementing partners at the Mission level. The model should be flexible allowing for multiple scenarios (one-off buy-ins, urgent needs, and multiple USG funding sources. (p. 45) X X X 17 Integration of concerns for women and disadvantaged groups. Integrate existing information into buy-in SOWs where possible and develop more detailed gender audits and strategies where needed. (p. 45) X X X 18 Competition with Private Sector Develop a standard by which USAID Missions and bureaus can judge whether or not a potential buy-in should be implemented by the private sector or the USFS. (p. 44-45) X X X 19 Tracking quality of T/A provided Design and require use of an online survey that would provide standardized input on Mission staff satisfaction with buy-in implementation. (p. 45) X X X X 20 Sharing knowledge of the scope of PAPA activities. Produce more detailed annual reports that include bullet statements of the major activities implemented under the PAPA by region and/or technical theme. Periodic thematic and/or region focused meetings between USFS-IP and interested USAID technical staff to share information about the activities being implemented under the PAPA. (p. 45) X X X X 21 Size of PAPA unmanageable and time needed to get buy-ins approved. Develop thresholds or standards that could be used to trigger discussions with Missions or bureaus to encourage them to shift towards a direct PAPA agreement with the USFS-IP. (p. 45) X X X MEASURING IMPACT – PAPA EVALUATION REPORT 10 PURPOSE OF THE EVALUATION PURPOSE OF THE EVALUATION The Evaluation Policy commits USAID to measure and document project achievements and shortcomings so that the Agency’s multiple stakeholders gain an understanding of the return on investment in development activities. The policy states that evaluation at USAID has two primary purposes: accountability to stakeholders and learning to improve effectiveness. To this end USAID/E3/Forestry and Biodiversity Office (E3/FAB) has engaged with the Measuring Impact project (MI) to evaluate the United States Forest Service (USFS) Participating Agency Program Agreement (PAPA). The Measuring Impact project (MI) is an effort to enhance the impact of biodiversity and forestry programs throughout USAID through improved knowledge, evidence-based programming and adaptive management. The goal of the evaluation is to determine the extent to which the USFS PAPA is being used and performing towards its intended objectives, and to provide recommendations on how its use and performance may be improved to increase the overall effectiveness of this Inter-Agency Agreement (IAA). The evaluation has 4 key tasks or objectives: 1. Improve our understanding of how the PAPA is being used by USAID Operating Units to meet their development objectives. 2. Assess the technical, program management, and financial performance of the current PAPA. 3. Assess the contribution of the USFS PAPA to the purpose and objectives of both E3/FAB and USFS-IP. Is the PAPA meeting the purpose and scope of the IAA? 4. Based on the findings from the assessment of use and performance from above, identify key recommendations/best practices for the continued use of the PAPA to maximize the value to USAID Operating Units and ensure compliance with applicable regulations. 11 MEASURING IMPACT – PAPA EVALUATION REPORT BACKGROUND ON THE USFS/USAID RELATIONSHIP BACKGROUND ON THE USFS/USAID RELATIONSHIP * The current Participating Agency Program Agreement (PAPA) agreement between USAID and the United States Forest Service (USFS) is the latest iteration of Inter-Agency Agreements (IAA) between these agencies that date back over 32 years. The first collaboration between USAID and the USFS began in 1981 under an IAA with the Department of Agriculture under support of the Forest Resources Management Project (#936-5519) (FRM), a global project for supporting forestry activities that included the USFS and other agencies. At that time the Forestry Support Program began operation as the predecessor of today’s International Program. This phase ran for 10 years and worked to respond to Mission and host country needs for information and research. The USFS provided short and long term (including three regional advisors) technical assistance to Africa, Latin America, and Asia/Near East regions. This greatly strengthened USAID’s technical capacity to support Missions in forestry-related activities: reforestation, seed production, nurseries and forest research, and laid the groundwork for the expansion of the IAA in the second phase. The second phase of the FRM ran from 1991 to 1997. While it operated much like the previous phase, Diamond notes that there was a greater emphasis on strengthening institutions, technology transfer and supporting commercial forestry activities. During this period the USFS began developing and implementing international support activities independent of the USAID funded activities. Budget cuts at USAID and USFS were a common challenge during this period, forcing reductions in staffing. In 1997 USAID’s relationship with the USFS shifted to a direct IAA, categorized as a PAPA, with the USFS. This shift reduced the overhead paid by USAID from 32% to 18% and gave USAID more direct management authority. This first PAPA began in 1997 and ran through 2007 after being extended twice. This agreement was interpreted (apparently through a reading of ADS 103.3.8.4, governing IAAs, under the Foreign Assistance Act (FAA)) to require that the IAA be used to carry out “inherently government functions.” This interpretation is often mentioned as a sort of litmus test for whether or not an activity is appropriate to be undertaken though the PAPA. Priorities for this agreement included the following:        Urban, social and agroforestry in support of sustainable agriculture and biodiversity, Participatory natural forest management, Institution-building, Outreach and research, Collaboration with nongovernmental organizations, Global climate change issues, and Remote sensing, including geographic information systems and global positioning systems.” The current PAPA, and subject of this evaluation, was established in 2007 and is due to end in 2017. It has the broadest technical range of any of the previous agreements, covering:              Sustainable forest management policies and practices Protected area management and forest biodiversity conservation Fire prevention and fire response Forest monitoring, remote sensing and geographic information systems Global climate change analysis and mitigation, including carbon sequestration Tree-based biofuels production Community forestry Agro forestry Smallholder wood production systems Regional forest planning Invasive species and forest pest/disease management Disaster planning and mitigation Governance of natural resources * Information for this section was gleaned from interviews with Alex Moad (USFS IP) and Tim Resch (USAID), summary reports of the Forestry Support Program (1985 and 1989) and from the excellent history presented in Nancy Diamond’s Final Evaluation of the previous PAPA agreement (Diamond 2006). MEASURING IMPACT – PAPA EVALUATION REPORT 12 BACKGROUND ON THE USFS/USAID RELATIONSHIP In addition, the agreement states that other technical support activities may be added through mutual agreement of USAID and the USFS. The current PAPA ceiling now stands at $200 million, from an initial ceiling of $50 million (the ultimate ceiling of the previous PAPA), and approximately $140 million has already committed. While the Scope of Work for this evaluation suggests that the “inherently government function” still stands as a requirement for this IAA, it is not specifically mentioned in the 2007 Action Memo authorizing this PAPA. In fact, that memo refers to a different section of the FAA: “Justification: Section 621 (a) of the foreign assistance act (FAA) provides that: “In such fields as education, health, housing, or agriculture the facilities and resources of other Federal agency shall be utilized when such facilities are particularly or uniquely suitable for technical assistance, are not competitive with the private enterprise, and can be made available without interfering unduly with domestic programs.” The action memo does however give further clarification for the rationale for using the PAPA: “Under this PAPA, USFS will provide services that are particularly suitable and unique to their facilities and resources, are not competitive with private enterprise, and can be made available without unduly interfering with its domestic programs. The USFS is the premier forest management agency in the U.S. and throughout the world, having extensive responsibility to provide research and implement USG programs in forestry research and technology testing, training in forest management and service provision, and forest policy development. Its 30,000- person workforce provides expertise in forest policy, natural forest management, logging systems, forestry research, information management and technology, training, education, and forestry support to state and local governments. The Office of International Programs of the USFS complements USAID forestry program efforts and provides an opportunity for both agencies to leverage resources in promoting sustainable NRM worldwide.” Subsequent action memos do mention “inherently government function” from ADS 103.3.8.4 under the Authority section. The memos state that “the agreement is for inherently governmental functions or technical assistance” (emphasis added). The ability to fund either inherently government functions or technical assistance under the PAPA gives considerable leeway in the selection of activities. It seems that the more restrictive requirement is the stipulation in the FAA that Technical Assistance activities “are not competitive with the private enterprise.” To support implementation of the buy-ins under the current PAPA the USFS has contracted with a private sector partner, METI (Management & Engineering Technologies International, Inc.). The contract with METI allows the USFS-IP to deal with issues related to making international payments and hiring staff internationally that would normally be impossible under USFS rules and regulations (the USFS was not set up to operate internationally). In cases where USFS cannot field a USFS employee to fill a need overseas then METI is used to enter into a long term employment or short term contracting agreement with these staff. USFS-IP will also use METI to pass funds to overseas grantees and contractors in situations where the USFS is unable to do so due to USG rules related to registration and having approved NICRAs (Negotiated Indirect Cost Recovery Agreement). METI charges a 15 % overhead on funds paid to support employees and short term contractors and a relatively low 1.5% on pass through funds. 13 MEASURING IMPACT – PAPA EVALUATION REPORT METHODOLOGY METHODOLOGY EVALUATION METHODOLOGY MI and the USAID/E3/Forestry and Biodiversity Office (E3/FAB) created an initial Scope of Work (SOW) (Annex I) for the evaluation of the United States Forest Service (USFS) Participating Agency Program Agreement (PAPA) mechanism in early 2013. From the initial scoping stages, it was clear that there were specific challenges to designing and carrying out this evaluation. Challenges included the size of the mechanism (well over $100 million by the time evaluation activities got under way) and the complexity of conducting an evaluation across two agencies that have different ways of conceiving, tracking and reporting activities. The evaluation was implemented in a phased approach in order to best manage these challenges. Phase 1: Constructing a database of buy-ins. Phase 2: Analyzing the database. Phase 3: Refining evaluation questions and approach. Phase 4: Select priority buy-ins and develop survey instruments. Phase 5: Data collection, analysis and reporting. The MI team included Paul Cowles, lead evaluator and Elaine Sabourin, compliance expert. PHASE 1: CONSTRUCTING A DATABASE OF BUY-INS. To reduce complexity and to create a common language to evaluate activities, a database of the PAPA buy-ins was constructed. MI coordinated with USAID, USFS and a database developer at ICF International to design an Access database to capture and report information regarding buy-ins as of August 2012 (up to and including Amendment 14). The database captures information on technical assistance themes, the type of assistance provided, the delivery mode of technical assistance, the beneficiaries, funding levels, as well as other information (see Annex VII for full list of database fields). USFS coordinated data entry into the database from USFS-IP regional staff. The database was created to provide more detailed information on USAID Missions’ use of the PAPA, in order to prioritize which buy-ins to focus on in the evaluation, but it is also intended to be used as an ongoing tool for management of PAPA, after the evaluation. PHASE 2: ANALYZING THE DATABASE: The lead evaluator (engaged at the beginning of this phase) used the database to carry out a series of queries to better understand the predominant themes (biodiversity, climate change, disaster management, etc.) and technical activity areas being funded through the PAPA mechanism. The analysis also allowed a better understanding of the extent of use of a range of T/A delivery modes for supporting various types of beneficiaries. PHASE 3: REFINING EVALUATION QUESTIONS AND APPROACH. The initial analysis of the database allowed the evaluation to get a better handle on the scope (financial, technical and geographic) of the buy-ins funded through the PAPA mechanism. Based on this analysis, which was shared with E3/FAB and the United States Forest Service Office of International Programs (USFS-IP), the original SOW was discussed and USAID priorities were clarified in a meeting at the USFS￾IP offices on Sept. 20 2013. The meeting helped evaluators better understand the key areas under each Task where USAID wanted to “drill down” and arrive at best practices that would help future implementation of the PAPA and its follow-on agreement. These tasks include the following: MEASURING IMPACT – PAPA EVALUATION REPORT 14 METHODOLOGY Task 1: Improve our understanding of how the PAPA is being used by USAID operating units to meet their development objectives. Research questions for this Task did not change significantly, but maintained a focus on understanding the breakdown of buy-ins by funding amount, region, country, funding source, technical areas and mode of T/A delivery and gaining an understanding of the predominant technical themes and modes of delivering T/A to beneficiaries. This task was completed through an analysis of the PAPA buy-in database, described above, followed with discussions of analysis findings with USAID and USFS personnel. Task 2: Assess the technical, program management, and financial performance of the current PAPA. Evaluation questions under this task were categorized under Technical Performance or Program Management. Technical performance evaluation questions were given a high priority; they sought to better understand the quality and effectiveness of the detailers and T/A provided by USFS-IP to USAID operating units for the most relevant technical areas and delivery modes (identified above). Technical performance was assessed through key informant interviews with relevant USFS and USAID staff and T/A beneficiaries, combined with reviews of buy-in documentation and online surveys. Multiple sources of data were used to triangulate the analysis and ensure the broadest possible range of responses and experiences are included so that fruitful or divergent lines of inquiry can be followed. Program Management evaluation questions were discussed at the Sept. 20th meeting to improve formulation of some questions and to prioritize key questions. Priority program management evaluation questions included:    Are the tools the USFS-IP uses to manage the PAPA adequate (e.g. mechanisms to allow USFS to spend funds, track budgets and results)? Does the recent trend of placing or hiring USFS-IP staff overseas significantly improve project implementation, coordination and communication? Is the management of the PAPA in compliance with the rules and regulations applicable to USAID? Lower priority evaluation questions were related to developing a better understanding of funding and cost share:  How are funds used relative to budgets? What is the perceived value by Missions relative to the actual value? What are the requirements (or expectations) for cost share under the PAPA or specific buy-ins? What is the cost-share provided by USFS? How is it calculated? What are the funds leveraged from non-USAID sources and are those funds linked to buy-ins or to the program being supported by the buy-in? More emphasis was placed on collecting sufficient data to respond to the priority program management questions. Key informant interviews and online surveys included questions related to both high and lower priority program management research questions. In addition, a desk study that included a compliance review was carried out for five priority buy-ins. Scopes of work, work plans, financial reports, performance reports, activity reports, trip reports, communications products, grant agreement templates and procedural documents, and information obtained from online sources such as project websites and the USAID Environmental Compliance Database, were all used to carry out the desk study. Compliance assessment focused on five key areas of compliance: financial and program reporting; grants management; USAID environmental procedures and standards; branding; and anti-terrorism screening. Task 3: Assess the contribution of the USFS PAPA to the purpose and objectives of both E3/FAB and USFS-IP. Under this task, USAID placed a high priority on understanding if the “established rationale for implementing an IAA mechanism between USAID and the USFS had been met”. Essentially, USAID wanted to confirm that the PAPA mechanism is being used appropriately; i.e. does the use observed fit 15 MEASURING IMPACT – PAPA EVALUATION REPORT METHODOLOGY the US statutory and USAID administrative requirements of an IAA? In addition, USAID wanted to know what evidence exists that USFS T/A provided under the PAPA is achieving outcomes for intended recipients (including women and disadvantaged groups). Data was collected for Task 3 through detailed reviews of legislation and USAID regulations (ADS), the desk study of priority buy-ins, interviews with key informants, and online survey instruments. Task 4: Identify key recommendations/best practices for the continued use of the PAPA to maximize the value to USAID Operating Units and ensure compliance with applicable regulations. Key recommendations fall under three areas of performance that could be improved: compliance, management and technical. Research questions included: What technical areas and delivery modes most benefit USAID and its partners? How can we improve the management of subsequent buy-ins (or PAPA’s) to improve implementation? How can we improve the parameters for using the PAPA to ensure activities are appropriate? Recommendations were developed based on the evaluators’ knowledge of and research into best practices, reviews of legal and administrative guidance, and discussions with USAID and USFS staff. PHASE 4: SELECT PRIORITY BUY-INS Five priority buy-ins were identified in consultation with the E3/FAB Office and USFS-IP for further evaluation. These buy-ins were geographically and thematically diverse with representation across the predominant technical areas, delivery modes, beneficiary type, scope and structure (grant making vs. straightforward T/A). The buy-ins included the Brazil Forest Enterprise Cluster, STEWARD in West Africa, Uganda STAR project, RDMA support, and the global SWAMP program. A desk study that included a compliance review was carried out for these buy-ins. In addition, Liberia forestry programs and Peru buy￾ins were included for the review of environmental compliance activities only. After an initial look at these priority buy-ins it was decided to add a few “one-off” buy-ins to the mix to ensure the evaluation did not just focus on major investments, but also allowed the evaluation to triangulate and confirm findings based on data from smaller tasks implemented under the PAPA. These one-offs included: South Africa Incident Command Systems, Malawi REDD+ support, and Colombia Protected Areas support. PHASE 5: DATA COLLECTION, ANALYSIS AND REPORT DEVELOPMENT A set of discussion guides (both technical/managerial and compliance oriented) was developed focusing on the key groups involved with the PAPA. Interview guides were developed for USAID managers and buy-in technical leads, USFS-IP staff, recipients of T/A and USFS detailers. Online surveys were developed to supplement data collection and triangulate findings from buy-in technical leads and recipients of T/A. Interview and survey questions were identified based on the indicators (qualitative and quantitative) in the SOW. Not all indicators could be assessed but every effort was made to ensure instruments could respond to the key evaluation tasks. A targeted and opportunistic approach was taken to identifying key informants and survey respondents. Initial contacts with Mission personnel were facilitated by USFS-IP staff and then evaluators identified subsequent respondents. A total of 44 key informant interviews were carried out, and an additional 28 online surveys were completed. The lists of interview questions and interviewees are included in Annexes II and III, respectively. Quantitative and qualitative analysis of the interviews and survey data were carried out to identify key recurring themes and to identify topics for further exploration and discussion with USFS and USAID staff. Results of these analyses, along with data from the desk study and database analysis were used to compile this report. CHALLENGES AND LIMITATIONS The PAPA is a massive mechanism, with activities implemented all over the world and 218 buy-ins valued at over $145 million as of February 2014. Given limitations on time and resources, the evaluation team was only able to look at a small number of buy-ins and talk to a fraction of the persons involved with and benefiting from the PAPA. The pool of potential buy-ins for evaluation included those present before August 2012, or up to amendment 14. Although we believe that the general strengths and weaknesses of MEASURING IMPACT – PAPA EVALUATION REPORT 16 METHODOLOGY the PAPA mechanism identified in this evaluation are comprehensive, it should be noted that selection of priority buy-ins was not random and in fact was done in collaboration with the USFS-IP and E3/FAB, and therefore could include biases. Evaluators attempted to control for bias by opening online surveys to a much larger range of USAID staff and T/A recipients, with particular attention to issues raised in online surveys that differed from those expressed in live interviews. Given that the technical side of the evaluation was focused on USFS capacity, the technical performance of USFS grantees (under West Africa STEWARD, Brazil FEC and Uganda STAR Buy-ins) was not evaluated directly. Rather the evaluation focused on Mission and Bureau satisfaction with the overall work of the USFS, which included, in some cases, the work of its grantees. All information was gathered through interviews and online questionnaires from targeted and opportunistic (i.e. non-random) sampling so some opinions/experiences may be under or over represented in the analysis (particularly local beneficiaries). For this reason we do not simply count responses as a means to analyze data. Divergent or unique responses are followed up on where possible in order to better understand the context of the response and develop informed theories of the likely causes. While the possibility of over representation of some views is also possible, the evaluators made every effort to follow up on one-off comments to ensure they provided useful and relevant information. Comments in the report that are based on a single observation have been identified as such. The compliance component was limited in scope to only those buy-ins specifically identified by USAID as high risk for non-compliance. The compliance desk review is based solely on the documents provided by USFS in response to the evaluation team’s requests and on documents and information publicly available online, such as project Websites and the USAID Environmental Compliance Database. The desk review does not reflect information collected through interviews. See Annex V for the full Compliance Desk Review Report. 17 MEASURING IMPACT – PAPA EVALUATION REPORT FINDINGS FINDINGS The Findings section is organized by the four evaluation tasks. Under each task, a high priority evaluation question is introduced, followed by descriptions of the individual findings that pertain to that question. TASK 1: IMPROVING OUR UNDERSTANDING OF HOW THE PAPA IS BEING USED. HOW IS THE PAPA BEING USED? As of August 2012 (through Amendment 14), there had been a total of 173 buy-ins made to the Participating Agency Program Agreement (PAPA) for a total obligation level of $113,753,747. The value of individual buy-ins varied greatly: from $15,577 to $6,000,000. The arithmetic funding mean is $657,536 while both the median and mode are significantly lower ($300,000 and $100,000 respectively). This shows that the mechanism is more often used to fund activities on the lower end of the funding range. Fifty buy-ins were for $150,000 or less while only 23 were for a million dollars or more (only 12 of those were for more than $2,000,000). The remaining 100 buy-ins fall between $150,000 and $1 million. A “buy-in” is a specific activity established through an amendment to the PAPA agreement with a set scope of work and budget funded by a USAID operating unit (Mission, bureau or office). A buy-in may be a one-off T/A activity that addresses a specific need in USAID programing or it may be part of a longer term activity or project funded through multiple buy-ins over several years. This report will use the term buy-in to represent a one off activity or a longer term project that may actually be funded through multiple successive buy-ins. FUNDING LEVEL BY YEAR AND BY THEME As depicted in Figure 3 below, funding levels have increased steadily throughout the life of the PAPA. Anomalous timing of funding for Fiscal Year (F/Y) 2009 meant that most of the funds were obligated at the end of F/Y 2008, giving the appearance of much higher funding in 08 and much lower funding in 09. Four main types of funding can be identified, as depicted in Figure 4: Biodiversity (mainly Biodiversity Earmark funds), Climate Change (mainly Global Climate Change Earmark), Disaster management (often OFDA funding, usually focused on the supporting Incident Command Systems) and other sources of funding. The Other category can include activities like working on free trade agreements to address issues of illegal logging and other natural resource management practices and environmental compliance activities that may be funded from any funding source within a Mission. $- $5,000,000 $10,000,000 $15,000,000 $20,000,000 $25,000,000 $30,000,000 $35,000,000 $40,000,000 $45,000,000 2007 2008 2009 2010 2011 2012 PAPA Funding by Fiscal Year Figure 3: PAPA Funding by Fiscal Year 0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% Biodiversity Climate Change Disaster Other Percent of Total PAPA Funding by Theme Figure 4: Percent Funding by Theme MEASURING IMPACT – PAPA EVALUATION REPORT 18 FINDINGS Biodiversity funds are the most prevalent funding theme with over 50% of funds expended to advance biodiversity conservation. Climate Change funds are in second place, but as illustrated in the Figure 5 below, climate change has increased significantly and could well outpace Biodiversity funding before the current PAPA ends. $- $5,000,000 $10,000,000 $15,000,000 $20,000,000 2007 2008 2009 2010 2011 2012 PAPA Funding by Theme and Fiscal Year Biodiversity Other Disaster Climate Change FUNDING BY TECHNICAL AREAS Within the database, buy-in activities were assigned to the following technical areas: 1. Biodiversity/protected area management (biodiversity conservation, habitat management, wildlife assessments, illegal wildlife trafficking ) 2. Biodiversity/protected area use (tourism, trails, visitor centers, interpretation, conservation education) 3. Climate change/carbon monitoring (REDD+, carbon-focused inventory, remote sensing and GIS) 4. Climate change/other (LEDS, adaptation, GCC analyses) 5. Disaster management (non-fire ICS) 6. Fire management (fire ICS, ecology, prevention, response) 7. Illegal logging (Lacey Act, wood ID, forensics, genetics, log tracking) 8. Natural resources management practices (RIL, invasive species and pests, non-carbon forest inventories, forest monitoring) 9. Natural resources management policies (land-use planning; environmental, economic assessments; institutional capacity building) 10. Production forest use (community forestry, agroforestry, tree-based biofuels, smallholder wood production systems) 11. USAID Programming Support (Reg. 118/119 assessments, ETOAs, IEEs, institutional assessments) 12. Watershed management (ecosystem services) Figure 6 below shows the spending for these technical areas. The technical areas with the highest percentage of total PAPA funding are Climate Change/ Carbon Monitoring, Natural Resource Management Practices, and Biodiversity/Protected Area Management, in that order. Figure 5: Percent Funding by Theme and Fiscal Year 19 MEASURING IMPACT – PAPA EVALUATION REPORT FINDINGS 10.4% 6.7% 21.1% 7.8% 7.6% 3.6% 8.3% 7.9% 19.4% 4.4% 1.2% 1.7% 0.0% 5.0% 10.0% 15.0% 20.0% 25.0% BIODIVERSITY/PROTECTED AREA MANAGEMENT (BIODIVERSITY CONSERVATION, HABITAT MANAGEMENT, … BIODIVERSITY/PROTECTED AREA USE (TOURISM, TRAILS, VISITOR CENTERS, INTERPRETATION, CONSERVATION … CLIMATE CHANGE/CARBON MONITORING (REDD, CARBON-FOCUSED INVENTORY, REMOTE SENSING AND … CLIMATE CHANGE/OTHER (LEDS, ADAPTATION, GCC ANALYSES) DISASTER MANAGEMENT (ICS) FIRE MANAGEMENT (ECOLOGY, PREVENTION, RESPONSE) ILLEGAL LOGGING (LACEY ACT, WOOD ID, FORENSICS, GENETICS, LOG TRACKING) NATURAL RESOURCES MANAGEMENT POLICIES (LAND-USE PLANNING; ENVIRONMENTAL, ECONOMIC … NATURAL RESOURCES MANAGEMENT PRACTICES (RIL, INVASIVE SPECIES AND PESTS, NON-CARBON FOREST … PRODUCTION FOREST USE (COMMUNITY FORESTRY, AGROFORESTRY, TREE-BASED BIOFUELS, SMALLHOLDER … USAID PROGRAMMING SUPPORT (REG. 118/119 ASSESSMENTS, ETOAS, IEES) WATERSHED MANAGEMENT Percent of total PAPA funding by technical area Figure 6: Percent Funding by Technical Area ALLOCATION OF FUNDS BY USFS BY YEAR As illustrated in Figure 7, eighty percent of funds allocated to buy-ins are used directly by the USFS for implementation. About 20% of funds are passed through grants and contracts mainly to NGOs and contractors, although some funds are passed through to multilateral donors, and universities. Only about 2% of funds are transferred to other USG entities, such as US Fish and Wildlife Service or US Geological 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 2007 2008 2009 2010 2011 2012 ALL Percent allocation of funds by year Transferred to Others (NGOs, Universities, Multilaterals) Transferred to other USG Direct Implementation Figure 7: Percent Fund Allocation by Year MEASURING IMPACT – PAPA EVALUATION REPORT 20 FINDINGS Service. FUNDING BY REGION Latin American and African countries and regions provide the majority of buy-in funding under the PAPA (each over 30%, see Figure 8 below). Global Bureaus provide the next highest level of funding at a little over 20%. Figure 8: Percent Funding by Region 0% 5% 10% 15% 20% 25% 30% 35% 40% Africa Asia Pacific Global Latin America and the Caribbean Middle East Russia/E. Europe/Central Asia Regions by Percent of total PAPA funding 21 MEASURING IMPACT – PAPA EVALUATION REPORT FINDINGS Figure 9 below shows how funding from these regions has changed over time. The decline in Latin America and the Caribbean funding levels is due to a direct agreement established in Peru (i.e. that money is no longer passed through the global PAPA). Africa and Global funding has increased over time. Figure 9: Annual Funding by Region DIRECT BENEFICIARIES BY FUNDING AND TECHNICAL THEMES For the purposes of our analysis, and throughout this document, we will use the term beneficiary or recipient to indicate a group which receives or is targeted to receive the benefit of USFS T/A. USFS-IP staff identified the following categories of beneficiaries in the design of the database: 1. Host country government agencies 2. International NGOs 3. Multilateral organizations (CIFOR, FAO, WB, EU) 4. Local NGOs 5. Communities 6. International contractors 7. USAID Mission/Bureau 8. Private sector 9. USG agencies 10. Academic institutions MEASURING IMPACT – PAPA EVALUATION REPORT 22 FINDINGS Figure 10 below shows categories of beneficiaries by technical theme (Biodiversity, Climate Change, Disaster or Other) and the percent of funding directed at each beneficiary groups. Given the government oriented focus of most PAPA T/A, it is not surprising that Host country governments are the primary beneficiary of the PAPA buy-ins (45%). More surprising are the two next highest groups: communities (~14%) and local NGOs (~13%). While the Forest Service does provide some T/A direct to these groups, most of these activities are funded through grants and contracts to NGOs and contractors. Beneficiaries at the next level of funding include: International NGOs and Contractors (taken together), USAID Missions and Academic Institutions. In terms of technical themes, host governments receive slightly more support under the climate change theme, whereas most other groups receive support primarily through biodiversity-themed T/A. Figure 10: Percent Funding by Direct Beneficiaries and Themes 23 MEASURING IMPACT – PAPA EVALUATION REPORT FINDINGS HOW HAVE THE BENEFICIARIES OF THE PAPA INVESTMENTS CHANGED OVER TIME? As shown in Figure 11, over time there has been little change in support levels to most of the beneficiaries. The key exception are academic institutions that have sharply increased (begun really) in the last two years; this is entirely due to increases in climate change funding that are more likely to involve support to universities and research institutes. Figure 11: Percent Funding by Beneficiary and Fiscal Year WHAT ARE THE PRIMARY DELIVERY MODES FOR THE PAPA BUY-INS? The USFS has identified six Delivery Modes, depicted in Figure 12, that are used to provide technical assistance, including: 1. Field-based technical assistance within target country/region (direct interventions by USFS in support of USAID projects and programs). 2. Formal and informal training programs within target country/region (traditional training, workshops and seminars). 3. Policy and economic analysis (studies to support better understanding of institutional and economic context) 4. Environmental and program assessments, program design, program management, and monitoring and evaluations (supporting USAID 5. Study tours, workshop and seminars in US or third country 6. Applied research 0% 5% 10% 15% 20% 25% 30% 35% 40% Field Based Training Policy Anaylsis Env & Prog Workshops Applied Research Delivery Mode by percent of total PAPA funding MEASURING IMPACT – PAPA EVALUATION REPORT 24 FINDINGS          Figure 12: Delivery Mode by Percent Funding Most PAPA buy-ins are implemented as field-based activities (over 35%), followed closely by formal and informal training (~33%). Environmental and Program Assessments are third and represent just over 15% of buy-in investment. KEY TECHNICAL AREAS, DELIVERY MODES AND BENEFICIARIES OF THE PAPA: For the purposes of focusing the evaluation, the top technical areas, delivery modes and beneficiaries were identified from the data. These were used to support selection of the priority buy-ins (see below) for more detailed assessment. Top three technical areas receiving the most funding Climate change/carbon monitoring (REDD, carbon-focused inventory, remote sensing and GIS). Natural resources management practices (RIL, invasive species and pests, non-carbon forest inventories, forest monitoring). Biodiversity/protected area management (biodiversity conservation, habitat management, wildlife, assessments). Delivery modes receiving the most funding Direct Implementation Training Environmental and Program Assessments Beneficiaries receiving the most funding Host Country Governments Communities Local NGOs 25 MEASURING IMPACT – PAPA EVALUATION REPORT FINDINGS WHAT DO PAPA BUY-INS LOOK LIKE UP CLOSE? Each of the priority buy-ins that was reviewed by the evaluation team is discussed in detail below. BRAZIL FOREST ENTERPRISE CLUSTER (FEC) The Forest Enterprise Cluster project was funded through a series of buy-ins to the PAPA under which the USFS-IP manages grants to four Brazilian NGOs and provides T/A to those partners and various government entities to enhance capacity building for improved forest conservation and management. The FEC is the latest in a long tradition of USFS involvement in Brazil. The FEC ended last year although PAPA buy-ins in Sustainable Landscapes and potentially others will continue. USAID/Brazil is extremely satisfied with the performance of the USFS-IP under this project. Some of the results achieved include:     A grantee, IMAZON, developed a forest monitoring system that is now used by government and private sector. The USFS-IP contributed significantly to the training and capacity building of personnel and the development of the 2006 Brazilian law that created the Brazilian Forest Service and has since continued to aid capacity building of the BFS. PAPA funded grantees in Brazil have also advanced the development of local environmental policies allowing some municipalities to demonstrably improve environmental practice. Communities that received training in sustainable forest management practices for sustainability are receiving certification, an indication that T/A is effective. Some of the challenges cited include distributing funding to grantees on a timely basis (congressional notification slows the process considerably), communicating successes and accomplishments to a broader audience, and lacking analysis of long term impacts of training (no structured long term monitoring and evaluation). When asked why the PAPA was selected as a funding mechanism, respondents cited the ease of use and expertise of USFS, and the great relationship between USFS and the Government of Brazil as key motivators. It is noteworthy that the USFS holds the grant manager role in this context even though the Brazilian NGOs have considerable experience managing USAID funds and could potentially receive grants directly from USAID. The USFS prefers to be the grant manager so that they can have more control over the context of their T/A. USAID also prefers for USFS to manage the grant because they are a trusted partner. WEST AFRICA STEWARD The STEWARD project began in 2007 as a trans-boundary protected area conservation and livelihoods improvement project in a region emerging from significant conflicts. Now in its third phase of implementation (2011 – 2015), the project is funded through the PAPA mechanism under four different earmarks (Biodiversity, Sustainable Landscapes, Climate Change Adaptation and Water, Sanitation and Hygiene). USFS manages the grants, using METI to hire local staff and disburse funds to grantees (described above in background), of six different organizations. USFS technical responsibilities are primarily focused on monitoring and evaluation (M&E), and include some capacity building of key stakeholders in the region. Some of the results achieved to date include:    USFS grantee supported village savings and loan systems have been established with significant positive impacts on livelihoods for women. USFS grantee supporting land mapping around communities that are enhancing land tenure security. Significant improvements in community mobilization for forest management and conservation. Issues encountered include: embezzlement by a STEWARD (USFS/METI locally-contracted) field office employee (see details below); lack of oversight of grantee technical activities that has been improved with placement of a USFS manager/coordinator; Mission feels grantees are not as well coordinated as they MEASURING IMPACT – PAPA EVALUATION REPORT 26 FINDINGS could be; and that the program needs more collaborative planning and coordination that could lead to improved outcomes and impacts. Early on the USFS was chosen to manage STEWARD due to its ability to mobilize quickly in a region that was emerging from significant conflicts. In fact USFS was able to provide support to post conflict Liberia very quickly and played a significant role in establishing USAID support for Forest conservation there. The West Africa Bureau is moving to a direct PAPA agreement with the USFS-IP in order to continue implementation. Considerable progress in terms of peace building and development in the region begs the question as to whether or not STEWARD could now be run under a competitive procurement. UGANDA STAR PROJECT In Uganda the USFS was an implementation partner under the first phase of the STAR (Sustainable Tourism in the Albertine Rift) project, run by the Academy for Educational Development (AED). The suspension of AED by USAID in 2010 put the project into a crisis that could have disrupted activities for a year or more. Instead the Uganda Mission was able to use the PAPA to quickly move management of the STAR project to the USFS-IP as an interim measure. The focus of USFS work under STAR was to support trail development, interpretation materials and displays, train guides and fire control systems. After the suspension of AED the USFS managed grants to other organizations to carry out key components of the second phase of the STAR project, which lasted about 1 year. Some key results include:    A cadre of USFS trained individuals who can build trails and do fire management. There are trails in Uganda that have been built by USFS trained teams without any direct support from the USFS and those trails are now being used by tourists. Beginning to see USFS-trained personnel being shared between Parks to train others. New tourism products and interpretation materials were identified and developed in Uganda and they are now being used by tourists. Private sector workers have mountain rescue skills and equipment ensuring better safety for tourists. The only issues raised about the STAR project was that the lack of an on the ground USFS manager made communication and management follow-up more difficult and slow. The use of the PAPA to “rescue” this program is clearly an advantage of the mechanism. During this period USAID was able to carry out a procurement for the follow-on to STAR which is now being run by the African Wildlife Foundation. The USFS continues as a T/A provider (funded through the PAPA mechanism) under this latest iteration but is no longer managing grants. REGIONAL DEVELOPMENT MISSION FOR ASIA (RDMA) SUPPORT The USFS-IP has been providing support to the RDMA for many years. In recent years support has become more structured and integrated with regional activities. A regional USFS advisor is in place and a 5 year work plan guides USFS support, focused on:    Promoting the conservation of biological diversity and sustainable forest management working to reduce wildlife trafficking and improve timber trade in Asia. Lowering of Greenhouse Gas Emissions from the forest and land use sectors in Asia through support to the Sustainable Landscapes programing and the Low Emissions Asian Development program (LEAD). Improving Asian countries’ abilities to anticipate, adapt and mitigate the impacts of climate change building on USFS experience in Payment for Ecosystem Services and disaster management in the region. The relationship with the RDMA seems like the ideal way to engage and utilize the USFS-IP. RDMA plays a strong technical lead with implementation partners and operates a range of activities, while the USFS plays an advisory and capacity building role to all. RDMA staff and NGO Sustainable Landscapes partners/implementers praised the USFS-IP and felt they played a key role in forwarding the RDMA’s program in the region, particularly in the area of capacity building. Implementation partners underlined the need to engage very closely with the USFS on the selection of detailers, development of scopes of work and follow-up on reporting to ensure quality. 27 MEASURING IMPACT – PAPA EVALUATION REPORT FINDINGS The relationship between RDMA and USFS-IP is not without its issues though. Several references were made to USFS efforts to market itself in the region to both USAID and the State Department, sometimes creating the perception that they are trying to “go beyond where USAID is willing or able to go.” The fact that USFS has State Department funding for Climate Change activities independent of USAID funding, may contribute to this perception. Some USAID staff expressed skepticism that the USFS was coordinating on implementation of these activities because without a PAPA buy-in there is no way to ensure collaboration with USFS-IP. USFS maintains that it does coordinate with Missions on activities, regardless of funding sources, but also does “pitch” ideas to Missions, as appropriate. An additional concern that was raised, was the notion that USFS is not the right partner to provide support for reducing wildlife trafficking. Some issues were reported regarding the implementation of this portion of the RDMA support. A respondent from the RDMA felt that the USFS team consistently ignored requests to focus efforts on program priorities and instead were “pushing their own agenda.” USFS attributed the issue to personnel changes at RDMA that led to a lack of understanding of how USFS had been coordinating T/A from USFS, US Fish and Wildlife Service (USFWS), and the Justice Department in support of Mission programs. USAID relies heavily on an NGO partner to implement this activity, and issues between USFS and the partner are also contributing to the conflict. RDMA staff indicated that they were shifting these activities to a direct agreement with USFWS, the main recipient of pass-through funds from the buy-in in question. E3/GCC SWAMP PROGRAM The Sustainable Wetlands Adaptation and Mitigation Program (SWAMP) is jointly implemented by the USFS-IP and CIFOR (Center for International Forestry Research). USFS-IP, with funding from E3/GCC office via the PAPA, is working to improve the science and practice of measuring and conserving carbon in wetlands and peat lands worldwide. The PAPA is used to provide research, capacity building and equipment to government and research partners to enhance carbon measurement in wetlands in several countries (Indonesia, Kenya, Vietnam, India, Tanzania, Mozambique, Ghana, Colombia, Ecuador and Peru). Under SWAMP the PAPA has contributed to:    Detailed studies that have yielded clear data on the carbon content of wetlands and developed techniques for measuring that carbon. Training of and equipment provision for host government staff and university researchers in techniques for wetland carbon measurement. Trained teams are now in place and producing data. The Intergovernmental Panel on Climate Change (IPCC) is using the information developed under SWAMP to improve its’ assessments of the importance of carbon in mangroves and swamps. USFS has been able to use SWAMP to mobilize supplementary funding from individual Missions and the State Department. SWAMP is a global scientific partnership that has expanded the role of the US in tropical wetland and climate change issues and mobilized many US universities and USFS research labs to participate in these efforts. Issues for SWAMP have included reporting issues in terms of timeliness, content and general communication. USAID staff felt they had to push to get regular, concise reporting. In addition, USAID felt there was not good communication around the range of work being done by the USFS that could be related to SWAMP and could provide opportunities for synergy and coordination. One USAID respondent felt that the USFS “likes to keep the range of activities they are implementing to themselves.” USFS-IP identified problems with USAID fulfilling its expectation to coordinate with CIFOR,who is based in Indonesia, because of the time difference (12 hours) and lack of bandwidth; CIFOR only has 3-4 staff working on SWAMP. USFS-IP also were concerned about the lack of follow up activities with those who have received training and equipment to ensure that they are collecting data and uploading it to the global database. If field teams regularly followed up with trainees, it would help address issues with missing data. MEASURING IMPACT – PAPA EVALUATION REPORT 28 FINDINGS TASK 2: ASSESSING THE TECHNICAL, PROGRAM MANAGEMENT, AND FINANCIAL PERFORMANCE OF THE PAPA. WHAT IS THE QUALITY AND HOW EFFECTIVE IS THE T/A PROVIDED UNDER THE PAPA? USAID respondents and T/A recipients are consistently very satisfied with the quality and effectiveness of the T/A provided by USFS detailers. The USFS is able to provide experienced and highly qualified Technical Assistance Detailers from their more than 30,000 employees in the United States. The USFS is the largest land management agency in the United States and is respected internationally. The USFS’ mandate for “multiple use” natural resource management makes it a well suited partner for USAID in the developing country context where forests must serve many different kinds of beneficiaries. QUALITY OF TECHNICAL ASSISTANCE Interviewees and online survey respondents are highly satisfied with the quality of USFS detailers and the technical assistance (T/A) they deliver. USAID and T/A recipients give detailers high marks on their experience, expertise, and ability to target the specific technical needs of beneficiaries. Respondents also think detailers demonstrated good knowledge of local natural resource issues, suggesting that the USFS￾IP’s preparation of and technical support for detailers is working well. Indeed, the USFS-IP will no longer send solo detailers into a country that is new to them; they are accompanied either by another more experienced detailer or an USFS-IP staff member who has experience in the country. The USFS staff believe that this practice has reduced the number complaints about detailer quality. Comments about detailers often referred to their professionalism, competence, and personable nature. One buy-in AOR said: “More important than my opinion are the views of counterparts and beneficiaries and how highly they value the assistance provided by the USFS and the interaction they have with the US experts”. Host governments view the USFS as a counterpart agency rather than as a donor or contractor or NGO; several respondents believe this gives USFS personnel more latitude and influence in their interactions with these beneficiaries. Complaints about the quality of detailers generally were related to a lack of experience and ability to adapt technical knowledge in the context of a developing country. A lack of flexibility was cited as an occasional problem with some detailers. Respondents also questioned the quality of detailers based on the lack of international experience across the sub-set of USFS employees. Regardless of detailer technical expertise, respondents felt that the results of the T/A can be disappointing when detailers fail to present or use their knowledge in a way that is relevant to the developing country context. EFFECTIVENESS OF TECHNICAL ASSISTANCE Respondents were satisfied with the effectiveness of USFS detailers and the T/A they provide. Mission and Bureau personnel consistently feel that the T/A provided is relevant to their ongoing programs of activities. Respondents feel that the USFS provides important inputs to USAID programs that result in better capacity among stakeholders. Respondents believe USFS T/A provides actionable skills and recommendations. Despite occasional concerns about the flexibility and adaptability of detailers raised above, most felt that the T/A provided is appropriate to the context of the beneficiaries in terms of both level of technology and cost. The ability of the USFS to provide government to government technical cooperation is often cited as a key contributor to USFS effectiveness. It is often difficult for USAID implementation partners (NGOs and contractors) to build credible relationships with host government agencies that can deliver tangible outcomes. As another government agency, the USFS enters into these relationships as a counterpart with similar goals and objectives. Respondents felt that host government land management agencies are generally more open to cooperating with the USFS on issues of capacity building than with other institutions (including USAID). Some respondents focused on the ability of USFS to get things done: “the USFS is so ingenious, they know how to get things done.” Another Mission respondent said the “Forest Service is like a branch of USAID that can get things done for us.” Yet another underlined the key output that the PAPA provides: “the Forest Service really builds the skills of people and that is what we need.” The USFS-IP presents a 29 MEASURING IMPACT – PAPA EVALUATION REPORT FINDINGS very strong “can do” attitude that USAID Mission staff appreciate; they can mobilize quickly, understand how to work with Missions, and with the PAPA they are able to simplify procurement of needed technical assistance. Once there is an agreement in place the USFS can move through the bureaucracy and get to implementation quite readily in many cases. Early in the evaluation, both USAID and the USFS raised a concern about the selection of appropriate recipients, i.e. ensuring that those getting the training or T/A were the people who could actually use the skills being transferred. This concern arose from an issue in Latin America around a road-building workshop. This workshop was organized under a buy-in to the PAPA and was supposed to provide important road-building skills to beneficiaries. Those invited to participate by the in-country partner, however, were not actually in positions that would allow them to benefit from such training. The USFS-IP conceded that in this situation they should have more closely checked backgrounds and positions of the participants proposed for the training since, as was the case here, USFS-IP often depends on local government agencies and other partners to identify participants for training. They see this as a more significant problem when they are called in to carry out individual actions for Missions rather than larger projects or programs that give the USFS-IP more perspective on and control over how T/A is provided. In this particular case they felt they were overly dependent on other actors to ensure the appropriate people received the training. This evaluation did not find any other examples where this type of issue occurred, and interviewees and survey respondents consistently felt the right people were receiving support. However, the USFS-IP indicates that this type of issue does occur from time to time, which provides an important lesson for the overall process of how T/A is delivered under the PAPA. Clearly there is a need for USAID, implementation partners, and USFS-IP to understand the local context where the T/A takes place in order to avoid this problem. As noted above, most respondents from the priority buy-ins were able to identify some tangible outcomes from USFS T/A. E-survey respondents were also able to identify tangible outcomes and impacts from USFS T/A:          A coastal landscape project has been designed for Ghana. In Mexico: “an Incident Command System for forest fires established, the Climate Change Strategy for Protected Areas developed, the Intensively Monitored Forest field sites (with their CO2-flux measuring devices), and a method for estimating greenhouse gas emissions from forest fires in Mexico.” “A remote sensing/GIS unit has been created in the Forest Management Bureau in the Philippines.” “The forest cover change map that was developed by the USFS is now being used as a management tool for decision making by the Forestry Department in Zambia.” Through its support to the Central African Regional Program for the Environment (CARPE) the USFS has supported “management plans for national parks and protected areas, community￾based land use plans, analysis and recommendations for fire management within rangelands, technical tools for forest inventory, use of GIS and remote sensing products for national forest inventory design and implementation. Training in all of the above for staff in counterpart agencies, collaborating NGOS and local communities.” “Many forest rangers (beat officers) within the Bangladesh Forest Department have noticeably increased their skills in geospatial data collection and analysis. Previously, they did not have these skills or access to GPS devices or GIS software. The trainings provided have built a solid foundation that the FD can now build from. We plan on using this as we look to implement the REDD+ RPP, specifically the national forest inventory and satellite forest monitoring system.” In the Dominican Republic, “a complete national seed bank was established and Ministry of Environment staff trained to operate the seed bank. “ Also in the Dominican Republic, “Ecotourism business and activities were established in and around protected areas that better protected the biodiversity of the area. i.e. kayak tours within a protected area that brought local employment, extra funds to the park, and indirectly provided effective patrolling of the park decreasing illegal activities such as fishing.” NDVI (Normalized Difference Vegetation Index) Land Degradation maps for Northern Botswana have been used to target GEF funding for an upcoming sustainable land management program; MEASURING IMPACT – PAPA EVALUATION REPORT 30 FINDINGS Forest Conservation Botswana (an NGO) has used these same maps to better target its funding assistance to forest conservation activities in Botswana. While not all of the items listed above are outcomes or impacts that can be entirely attributed to the PAPA, it is clear that respondents understood what activities were being carried out and, in many cases, had anecdotal evidence of results from USFS T/A that go beyond simple output indicators (i.e. number of people trained). While rare, the evaluation did identify situations where USAID staff were not satisfied. In assessing negative comments and discussing them with USAID and USFS staff, there seem to be three inter-related points which can lead to dissatisfaction with a detailer assignment specifically, or USFS-IP T/A in general, as listed below:    A mismatch between USAID expectations and needs and USFS expertise The USFS-IP’s drive to generate more buy-ins and expand its programs Personality conflicts between USFS and USAID or partner (NGOs, contractors) staff. Not surprisingly, when the USFS attempts buy-in assignments that are outside of their normal range of expertise or where their expertise does not match USAID needs, there is a much higher potential for disappointing results. Climate Change Adaptation (CCA) presents an example of this potential mismatch. The USFS has very specific forest or ecosystem oriented CCA tools that have been adapted into training materials and used to build capacity of host country forest management institutions. While forest and ecosystem tools are important to CCA, particularly in relation to the vulnerability of human systems to climate change, they are not a central focus of CCA. This lack of focus on forests, exclusively, has resulted in some dissatisfaction for one USAID respondent. The situation described above underlines a basic cultural difference between the agencies; USAID is largely concerned with and approaches issues from a human development perspective, i.e. a perspective focused on how to improve the lives of present and future generations in the developing world. The USFS is made up largely of foresters and is rightly focused on the management of forests from a multiple use perspective. These misaligned perspectives lead to mismatched expectations on some assignments. The USFS-IP now has personnel with considerable development expertise and perspectives, which undoubtedly improves communication and expectations in discussions with USAID. However, most detailers still approach work as foresters and from a forest management perspective. Alternatively, it could be argued that USFS-IP is pushing the envelope of its technical expertise in order to generate more buy-ins. Many respondents, who were nonetheless very positive about the effectiveness of USFS T/A, referred to the USFS-IP tendency to “act like an NGO” in terms of self-promotion and continually working to expand its programs. One respondent referred to the USFS-IP as a “contractor” that needed to “recruit business” to support their business model. The USFS-IP is driven to find opportunities to generate new buy-ins and to diversify sources of funds to advance USFS-IP objectives. This is an important mandate for the USFS-IP that helps to justify its existence within the USFS. The USFS of course, has the authority to work independently at an international level and can accept funds from non-USAID sources. In some countries, the USFS has a long history of cooperation with local institutions. This has to lead occasional perceptions of conflict or competition among some USAID Missions and their partners who fear that USFS-IP’s agenda may differ from overall USAID objectives for a country or region. The ability of the USFS to mobilize State Department funds contributed to this perception because the USFS could potentially implement activities in a country without USAID. Some respondents felt that the USFS was developing its own ideas without considering Mission objectives. The USFS feels it does try to communicate with Missions independent of its funding sources. The last point concerns the potential for personality conflicts between USFS-IP personnel and/or detailers and USAID Mission staff or their implementing partners. It is clear that these kinds of conflicts do arise from time to time and can slow or halt implementation of buy-ins. Mission staff changes in the midst of a buy-in leading to reorientations in program direction or a simple lack of trust between actors may contribute to these kinds of conflicts. Mistrust and unresolved conflicts between Missions and Global 31 MEASURING IMPACT – PAPA EVALUATION REPORT FINDINGS USAID offices and bureaus may also fuel some of these conflicts. One respondent expressed a sense of inevitability that the Mission would need a buy-in because of pressures from the global office. WHAT IS THE USFS INTERNATIONAL PROGRAM’S PERFORMANCE IN MANAGING THE PAPA? The task of managing the PAPA is complex in that is requires a range of tools and internal processes that ensure compliance with USAID and USFS rules and regulations and the achievement of the results desired by Missions and bureaus. Overall the USFS adequately manages activities under the PAPA, and Mission and bureau personnel consistently expressed the belief that they receive good value for their investment. There are, however, weaknesses in the systems that need improvement. SELECTING EFFECTIVE DETAILERS The key to successful T/A under the PAPA is selecting, preparing and managing detailers effectively. As mentioned above most complaints about T/A under the PAPA are related to the ability of detailers to provide assistance that accounts for the local (developing world) context. They key to addressing this shortcoming will be to continue to build the pool of experienced detailers and taking steps to improve detailers and their understanding of local contexts over time. At this time, however, the pool of experienced detailers is limited and the time a given detailer can dedicate to the USFS-IP is also limited because of other commitments within the Forest Service. In addition, the USFS-IP is expected to expand opportunities for international experiences among USFS personnel in general, not just for those that already have international experience. Efforts to develop a roster of experienced detailers are further complicated by the fact that detailers are USG personnel and there are legal restrictions on documenting certain aspects of their performance (positive or negative). To address these limitations, the USFS-IP has developed a set of best practices to guide the referral, selection, preparation and in-country management of detailers. LONG TERM USFS-IP STAFF PLACEMENT OVERSEAS The relatively recent trend of placing or hiring USFS-IP staff overseas to help manage PAPA buy-ins is viewed very positively, for the most, by USAID Mission staff. These individuals, who are, in most cases, not long term USFS employees, are viewed as key technical resources by Mission staff and partners. They greatly simplify communication between USFS-IP and Mission staff and partners allowing for greater interaction and collaboration in the development and implementation of buy-in scopes of work. They also are sometimes placed as direct technical advisors to host government forest agencies (such as in Malawi). Regional staff are perceived as having very strong technical skills and able to contribute significantly to technical planning and implementation by both Missions and partners. Clear and specific scopes of work are a key component to ensuring successful technical assistance. These staff are viewed as good investments by the Missions in RDMA, West Africa and Malawi. Some USAID Missions have expressed concerns, however, that these staff are for the most part consultants with international experience, not long term USFS employees. Indeed the USFS has to provide orientation training to these overseas hires to integrate them into the USFS-IP. While the RDMA Mission views the regional advisor placed in Bangkok as an extended part of their team they also take steps to ensure he does not participate in procurement sensitive discussions and activities. In Missions or bureaus where lines are less clearly delineated there could be opportunities for conflicts of interest when decisions are made about additional buy-ins to the PAPA. In some Missions there is a feeling that these international hires are being portrayed as USG staff when they are actually contractors, this can be a source of potential conflict or at least confusion. PERCEIVED VALUE BY MISSIONS Mission personnel were satisfied with the way funds were expended under the buy-ins examined. The funds were used to implement the activities agreed to in the buy-in SOW unless mutually agreed changes to implementation were made. Mission personnel also consistently said that PAPA buy-ins were a very good value, and in some instances personnel indicated that the USFS PAPA is a more cost effective way to achieve results than NGOs or private sector firms. This perception seems to be due to three factors. First is the lower-than-industry standard overhead cost of 19% (most NGOs and contractors are over 30%) for PAPA activities. Secondly, many respondents were aware that at least some of the salary costs of detailers are covered by USFS budgets (even though the USFS-IP generally puts the full value into its buy-in budgets to be safe). Thirdly, and perhaps most significantly, was the perception of relative ease of MEASURING IMPACT – PAPA EVALUATION REPORT 32 FINDINGS use and management of a PAPA buy-in when compared with a procurement of an NGO or contractor. This third factor raises concerns that the PAPA is being used in lieu of a procurement even though NGOs or contractors exist may be better suited to manage the activities required. In terms of cost share and leveraging, actual numbers from the 2010 performance report provide a working example: A. Buy-ins from Missions and E3/FAB $16,306,705 B. FS contributions (Salary and other) $ 6,962,430 C. Other USG Contributions (DOS, MCC) $ 3,850,000 D. Other non-USG matching funds $ 1,505,000     Line A represents the total obligated buy-ins for the fiscal year. Line B Forest Service contributions are direct cash inputs to USAID programs/projects or objectives in USAID countries in two categories: program obligations and salary contributions.   Program obligations include USFS-IP funded grants to NGOs and Universities and (rarely) USFS-IP funded travel costs for T/A. According to USFS-IP this does not include expenditures in non-USAID countries but may include countries where there is no PAPA buy￾in but where USFS-IP funded activities are considered (by USFS-IP) to be contributing to USAID objectives. The expenditure estimates are derived from I-Web records for USFS-IP grants and PFMAS records for USAID related travel costs. Salary contributions are cash inputs covering USFS detailer and USFS-IP salaries. Contributions are calculated based on the "person-days of USFS technical assistance" provided, multiplied by the daily rate for a GS-13, Step 1 employee. Situations in which USAID paid the salaries of detailers are excluded from these calculations. This system has recently been upgraded so that managers can now identify situations where USAID paid salaries as they enter information about project activities into PFMAS. In addition to detailer salaries, USFS-funded salaries for USFS-IP managers of USAID programs are included as an in-kind contribution. On average, 50% of USFS-IP project manager salaries are paid by USFS-IP, and 50% by USAID (E3/FAB core funding of around $800,000 per year plus partial coverage of program management costs by Missions). USFS-IP salaries for the management of projects in non-USAID countries or unrelated to USAID programs are excluded from these calculations. USFS-IP keeps a separate database that records the annual salary charges for each USFS-IP manager against the appropriate category. Line C other USG contributions are external funds sourced from the State Department and Millennium Challenge Corporation for activities that support USAID projects or objectives. These transfers are documented in the I-Web system. The bulk of the funds are State Department transfers for the implementation of a USAID/State/USFS climate change program (SilvaCarbon). Line D other non-USG matching funds is the estimated cost share provided by recipients of the USFS-funded grants described under Line B, above. These organizations are not required to provide cost share so their estimates are not necessarily well documented. There is no requirement of cost share or match in the PAPA agreement (nor in any of the buy-ins that the evaluation team looked at) the “tradition” of including this information dates to around 10 years ago as a result of an informal agreement between USFS-IP and the former head of what is now the E3/FAB team. The agreement was needed to track expenditures that were being shared with E3/FAB to show relative contributions. The E3/FAB contributions to these activities have since stopped but the USFS-IP continues to track this information. Given that there is no cost share or match requirement and since most of these numbers are based on estimates and calculations (e.g. days X daily salary rate) it is probably more appropriate to consider these numbers as “leverage.” At the same time there are significant issues of un-expended funds within the PAPA. As of January 2014 $23.9 million remained unspent; $12.9 million of this is from buy-ins prior to August 2012, while the rest 33 MEASURING IMPACT – PAPA EVALUATION REPORT FINDINGS represents buy-ins received after February 2013. The USFS states that it is normal to have unexpended funds and that this has been common throughout its history with USAID. Most buy-ins cover activities for multiple years and so funds remain unspent from one year to the next. Changing circumstances in some countries (staff turnover, changing priorities, etc.) may cause delays in decision making by USAID Missions that control how funds are expended. USFS also concedes that contracting delays within the USFS-IP can also delay implementation. Regardless of the reason this is a serious concern for USAID and USFS as de-obligation of these funds (should they ultimately remain unspent) would mean that they return to the US Treasury, not to other USAID programming. The USFS believes it is on track to spend most of the funds in question as originally planned, despite difficulties with some projects. The USFS has now shifted to a cost reimbursement model that will no longer allow the agency to receive advances, so once the current unexpended balances are spent or de-obligated the issue will be resolved. THE TECHNICAL AREAS The evaluation did not assess performance in all the technical areas, but did include areas other than the priority technical areas in terms of funding. Brief critiques of the additional technical areas are included below, based on interviews and the analysis of reports. The evaluation does not recommend removing any of these technical areas from the USFS’ repertoire, but does suggest that some minor adjustments may be warranted. Biodiversity/protected area management (biodiversity conservation, habitat management, wildlife assessments, illegal wildlife trafficking) This is one of the top three technical areas for investment, representing about 10.6% of funding. USFS is very strong technically in this area, although issues of adaptability to developing country contexts can be very important for detailers. This is a very broad category of activities including all aspects of protected area management and planning for biodiversity conservation. Forest management is also a significant category of investment under this technical area. The evaluation did capture some negative comments on USFS involvement in illegal wildlife trafficking. The context for these comments is unclear, but the question is being raised of whether USFS is the best agency to support this work when compared with US Fish and Wildlife Service (USFWS), which has very strong credentials in this area. Indeed, the USFS has provided grants to the USFWS to work under past RDMA buy-ins, in which case it may make more sense to develop a direct agreement with USFWS. Biodiversity/protected area use (tourism, trails, visitor centers, interpretation, conservation education) Representing about 7% of buy-in investment these are exactly the kinds of areas where the USFS technical capacity is very strong. Buy-ins focused on this technical area are generally focused on improving and increasing tourism investments for the benefit of governments, communities and private sector. The excellent results under the STAR project in Uganda focused on building government and private sector capacity in trail construction and maintenance, interpretation/guiding and search and rescue are good examples of Forest Service strengths. Climate change/carbon monitoring (REDD+, carbon-focused inventory, remote sensing and GIS) The top area of investment under the PAPA (21% of total spending). These buy-ins are largely focused on forest inventory and monitoring (remote sensing, GIS, ground trothing) in relation to carbon and in many cases REDD+. Respondents had great respect for USFS technical expertise and experience in climate change mitigation activities. It is clear that the USFS is a leader in supporting USAID to develop clear strategies and activities to support the Sustainable Landscapes program. Major climate change efforts under the PAPA include the SWAMP program, as well as REDD+ readiness and carbon monitoring support in many countries and as parts of broader NRM programming. Climate change/other (EC-LEDS, adaptation, GCC analyses) Around 7% of buy-in funds are allocated to these types of activities that can include a broad range of assessments focused on better understanding forest and ecosystem vulnerabilities and potential responses to climate change. Respondents involved with the RDMA felt USFS is a strong contributor to MEASURING IMPACT – PAPA EVALUATION REPORT 34 FINDINGS EC-LEDS (Enhancing Capacity for Low Emissions Development Strategies) at country and regional levels. Adaptation was more mixed, as mentioned above. Adaptation focused on forests, watersheds and ecosystems is well suited to USFS tools and expertise. Fire management (fire ICS, ecology, prevention, response) Representing only about 4% of buy-in funding, fire management is an area in which the USFS has long experience and hard earned lessons in the US making it an excellent source for this type of expertise. The Forest Service Incident Command System, developed over many years in the US is a very effective and efficient system for identifying and dealing with wild fires. The South African ICS experience focused on building capacity for a range of actors (government, local NGO and private sector) in setting up and using ICS to manage wildfires. Similar long-term programs to train government and community firefighters in the application of the ICS system have been conducted by the USFS in Mexico, Brazil, and Indonesia. USAID Programming Support (Reg. 118/119 assessments, ETOAs, IEEs, institutional assessments) This is an area in which USFS involvement has increased over the past few years. There are strong reasons for involving the USFS in environmental compliance. Few, if any, US agencies perform more Environmental Impact assessment and analysis and they can be a key resource to support Reg. 216 implementation for Missions assuming detailers can adapt to the realities of doing environmental compliance in a developing country context. Participation in 118/119 assessments and Environmental Threats and Opportunities Assessments (ETOA) seem to be a more recent addition to the USFS repertoire that have in the past often been carried out through contracts with private enterprise (particularly through small enterprise set asides). The USFS has provided training to detailers in how to complete these kinds of analyses and USAID staff seem pleased with their involvement. However, given the considerable skills in the private sector in these areas it does seem these activities could conflict with restrictions on competing with the private enterprise. THE DELIVERY MODES Findings on delivery modes from interview observations, survey responses, and reviews of technical documentation are relayed below. Field-based technical assistance within target country/region This is the most often used delivery mode for T/A by the USFS-IP, with over 35% of PAPA funding used to support field based T/A. This most often involves a detailer(s) working with local T/A recipients (most often host government but can include other types of beneficiaries) to directly implement or build capacity for a specific task (such as trail building) or system (such as ICS). Respondents felt it was quite effective but depended on the quality and effectiveness of the detailer(s) involved. Formal and informal training programs within target country/region Training is the second most common delivery mode, with over 33% of PAPA funds. Training recipients and USAID Mission personnel are very satisfied with USFS skills in delivering quality training. However, there is no long term M&E system in place to assess the longer term impacts of USFS training. Therefore, while the training may be very effective, it is unclear whether it is achieving the desired outcomes of USAID. Study tours, workshop and seminars in US or third country. Representing about 8% of total buy-in funding, this mode is extremely popular with recipients and Mission personnel. While there is a concern that study tours, workshops and seminars may evolve tourism activities, they are viewed as valuable opportunities for learning, facilitating peer to peer exchanges and for building networks among practitioners. Staff of the Brazilian Forest Service credit these activities with significantly influencing the creation of the BFS. 35 MEASURING IMPACT – PAPA EVALUATION REPORT FINDINGS Applied Research Applied research, currently representing about 5% of buy-in funding, is increasing in use, particularly in relation to GCC activities. For example, under SWAMP (see above) applied research has proven very valuable to improving the science and practice of measuring carbon in wetlands. Greater involvement with universities and research institutes, as part of increased climate change activities, is anticipated to increase the use of this delivery mode. FINANCIAL MANAGEMENT USFS has two financial management databases used for capturing and reporting on financial data. The USDA central database, FMMI, tracks information at the transaction level, while the PMFAS database tracks information at the project expense level. Financial information in the PMFAS project financial management database are reconciled on a quarterly basis against FMMI, the new USDA centralized financial management system, which was implemented at the end of 2012. There is currently a four-to-six week lag in the PMFAS data from when the transaction occurs to when it appears in PFMAS. While this lag is not out of the ordinary in terms of reconciling financial data, USFS is working to reduce this gap even further with the eventual goal of having real-time data available. In addition to the quarterly reconciliation of financial data between PMFAS and FMMI, on a monthly basis project management personnel in Washington perform a financial review in which they assess the expenses to date and projections for the coming period for each project using the PMFAS data as the basis of these analyses. Quarterly reports submitted to E3/FAB track financial data by buy-in and were found to have significant variances with the quarterly financial reports produced by PMFAS for internal project management purposes, as well as with program reports that included financial information on budgets and expenditures and were submitted to the Missions for the STEWARD project and for PAPA activities under the RDMA. These variances could be explained by the lag time in data being recorded into the PMFAS system or by the different basis for each type of report, e.g. program/project level for PMFAS versus buy￾in level for quarterly reports to E3/FAB. USFS maintains a very limited number of field offices, namely in Peru for the Amazon Forest Sector Initiative, in West Africa to manage the STEWARD project, and a reforestation project in Lebanon. These offices initially were funded through the PAPA, but are now funded through other agreements with USAID Missions. The field offices are staffed by a combination of expatriate and host-country personnel contracted through METI. In this type of operating mode strong internal controls are essential for ensuring that funds advanced to support field operations in overseas locations are properly managed. A recent incident in which the STEWARD program field office fell victim to a bank fraud scheme that is not uncommon in the West Africa region, illustrates the risk of financial malfeasance in field office operations. The fraud was discovered through a periodic financial review conducted by METI, Inc., the contractor through which funding flowed to the field office. Upon discovery of the problem, the USFS immediately informed USAID and the international auditing firm KPMG was hired at USFS’ expense to perform an audit to determine the extent of the fraud. In addition, because the funds were USAID regional Mission funds, the USAID OIG conducted an investigation based on METI reports and the KPMG audit. The reports and audit revealed that three STEWARD local staff members (contracted through METI) working in collusion with a bank employee conspired to commit the fraud. Based on evidence derived from the U.S. investigations, the principal staff member involved is currently being prosecuted for embezzlement and the two other staff members have warrants out for their arrest, but remain at large. The loss incurred due to this fraud will be reimbursed by USFS from its own funds. MEASURING IMPACT – PAPA EVALUATION REPORT 36 FINDINGS USFS COMPLIANCE WITH THE PAPA AND USAID REGULATIONS Five key areas of compliance were assessed to determine to what extent the PAPA is managed in accordance with the applicable USAID rules and regulations. Perceived high risk buy-ins were used to sample compliance. REPORTING REQUIREMENTS In general, there are two sets of program reporting requirements for each task under the PAPA, the requirements established by the PAPA agreement for reporting to the agreement manager at E3/FAB and reporting requirements established by the USAID Missions through approved work plans for the buy-ins. Reports are submitted to the Missions at regular quarterly or semi-annual intervals as well as annually. Final reports (task completion reports) are submitted to the Missions at the completion of each task or program, with programs consisting of one or more buy-ins (i.e., tasks) funded over one or more fiscal years. Although USFS-IP submits program reports annually to E3/FAB in accordance with the PAPA agreement, the reports lack some of the required content. Annual program reports did not include task completion reports for each task completed during the reporting period as required by section D.9 of the PAPA Schedule. They also did not include identification of the five grants awarded for $300,000 or more per year, and an audit schedule for those grants as required by section F.2 of the PAPA Standard Provisions. The 2010 annual report and all annual reports submitted thereafter were lacking the following information required by section D.9 of the PAPA Schedule: “information on progress or completion of components, elements or activities against planned targets; description of overall program status; other accomplishments and major highlights of program implementation; major activities planned for the subsequent reporting period; and separate sections describing country-specific activities.” Financial reports are submitted to USAID in accordance with the PAPA quarterly financial reporting schedule. Although there were some late submissions in FY 2013 due to USDA’s transition to a new financial tracking system, the system is now fully implemented and the late reporting issue has been resolved. Quarterly financial reports contain the level of detail specified in the PAPA Schedule. USFS also provides information on budget and expenditure information in the quarterly programmatic reports to those Missions that request such information. It should be noted that due to the different treatment and presentation of data on obligations, accruals, and expenditures, the content of these reports varies by Mission and does not necessarily correlate to the data in the quarterly financial reports to E3/FAB. GRANTS/SUBGRANTS MANAGEMENT According to the PAPA, USFS will administer the funds under the PAPA in accordance with its own procedures; this includes grants management. USFS-IP personnel manage the grants under the PAPA in accordance with policies and procedures outlined in the USFS Grants and Agreements desk guide and the relevant sections of the Forest Service Manual. Grants are typically managed by USFS-IP at the headquarters level using a grants management database system to track compliance with reporting requirements, and field visits, phone calls, and email correspondence to monitor grantees. In cases with a field office, such as STEWARD, the field staff is also involved in monitoring grantee performance. USFS uses a grant agreement template that is approved by OMB and includes the necessary provisions to ensure compliance with 2 CFR 215, Uniform Administrative Requirements for Grants and Agreements with Institutions of Higher Education, Hospitals, and other Non-Profit Organizations (OMB Circular A-110). The USFS grant agreement template includes an audit clause with a threshold of $500,000 in annual expenditures of U.S. Forest Service awards rather than the $300,000 in “grant awards” threshold prescribed by section G.2 of the PAPA Standard Provisions. In practice, although five awards met or exceeded the $300,000 threshold and two exceeded even the higher $500,000 threshold, no independent financial audits of the funds provided under such grants have been performed. Grantees receiving more than $10,000 but less than $500,000 annually in U.S. Forest Service awards are required by the USFS grant agreement to make records available upon request for review by U.S. Forest Service officials or their designees. This is for the most part in compliance with the PAPA wherein grantees are subject to the requirement to make their books and records available upon request for 37 MEASURING IMPACT – PAPA EVALUATION REPORT FINDINGS review by authorized U.S. Government officials, except that the PAPA does not specify a minimum threshold. REGULATION 216, AGENCY ENVIRONMENTAL PROCEDURES Pursuant to section F.2 of the PAPA Schedule, USAID environmental regulations (22 CFR Section 216, "Regulation 216"), including the requirement for an Initial Environmental Examination (IEE), apply to the activities to be undertaken under the PAPA. The IEE for the PAPA resulted in a negative determination with the conditions that if any of the activities under the PAPA fall outside of those anticipated by the PAPA IEE or if any of the activities under the PAPA include recommendations that could result in some environmental impact, then USFS must advise the USAID bureau or Mission funding the activity of the need for a separate IEE or Environmental Assessment (EA). Of the six projects and the one regional Mission whose activities were examined for compliance with environmental procedures under the PAPA, four had IEEs that were supplementary to the PAPA IEE. The other three did not have any activities that warranted a separate IEE. USFS was not involved in identifying the need for a separate IEE in any of the four cases of supplementary IEEs. In two cases an IEE was done by the Mission for all environmental activities in the Mission area (Brazil and RDMA). The PAPA activities in Brazil and under RDMA were specifically identified in those IEEs and fell within either a categorical exclusion or a negative determination with conditions, including the condition that any activities involving the design of activities with the intended result of a direct impact on the environment would require further environmental review. In two cases, STEWARD and Uganda STAR II, a project-specific IEE was initiated by the Mission. For these projects a negative determination with conditions was issued and mitigation measures were mandated. In accordance with the IEE and associated mitigation plan, STEWARD carried out the required work adjustments and stipulated mitigation measures. STEWARD also maintained documentation to show compliance with the mitigation plan. Uganda STAR II also had a negative determination with conditions, all of which were met, including conducting training on the application of recommended practices for natural resources management and carrying out small scale construction in accordance with the principles for environmentally sound construction, as provided in Chapter 3, Small Scale Construction, of the USAID Environmental Guidelines for Small-scale Activities in Africa (EGSSAA). BRANDING The level and extent of compliance with USAID branding requirements varied from project to project (see Annex VI, Branding Compliance Tables), but was generally adequate. All focus areas showed at least an awareness of the branding requirements and an attempt to comply, albeit to a greater or lesser extent, depending on the project and, in some cases, depending on the level of involvement of lower tier recipients, or other third parties, in the development of communications products. Although the USAID Graphics Standards Manual advises against giving precedence to a project brand rather than the USAID Identity, it was noted that for at least two of the projects, STEWARD and SWAMP, the project brand was given precedence over the USAID brand. Other instances of non-compliance with branding standards involved failure to use the USAID logo, although in many of those cases USAID support was acknowledged in the text of the document or presentation. ANTI-TERRORISM SCREENING METI Inc., a USFS contractor, is responsible for performing anti-terrorism screening of all employees, consultants, and contractors providing services to, and on behalf of, USFS under the PAPA. METI’s written anti-terrorism screening policies and procedures are in accordance with the anti-terrorism clause of the PAPA. METI maintains adequate documentation to show compliance with its policies and procedures. Screening of grantees is conducted by USFS in accordance with its internal grant procedures, which require that all grantees are checked against the Specially Designated Nationals (SDN) list. Grantee responsibility for compliance with Executive Order 13224, Executive Order on Terrorist Financing, is established by a standard provision in the grant agreement. This provision includes a flow-down clause that requires grantees to apply E.O. 13224 compliance procedures to lower-tier subrecipients. MEASURING IMPACT – PAPA EVALUATION REPORT 38 FINDINGS The anti-terrorism clause in the PAPA Schedule is not the most recent version currently in use for other USAID funding instruments such as contracts, grants and cooperative agreements. The PAPA anti￾terrorism clause lacks the flow-down requirement for the clause to be included in all subawards. 39 MEASURING IMPACT – PAPA EVALUATION REPORT FINDINGS TASK 3: ASSESSING THE CONTRIBUTION OF THE USFS PAPA TO THE PURPOSE AND OBJECTIVES... It is clear through much of the above discussions that the USFS provides significant contributions to USAID’s purpose and objectives. The USFS has been a partner to USAID in improving Forest Management for over 30 years and with the recent shift to concerns over GCC and the role of forests in mitigation, it seems advantageous to continue this mutually beneficial partnership. The questions below and related findings are designed to identify the specific contributions of the USFS PAPA to USAID and USFS-IP purpose and objectives, in order to identify areas for improvement. HAS THE ESTABLISHED RATIONALE FOR USING THE IAA BEEN MET? As mentioned above, there are two aspects of justifying the rationale for using the PAPA. The first and most frequently cited requirement was that an IAA had to be used for “inherently government functions.” The ADS guiding IAAs (ADS 103.3.8.4) is interpreted such that an IAA can be for “inherently government functions or technical assistance.” While respondents seemed unclear about the meaning of “inherently government function” and the way it relates to USFS-IP work under the PAPA, it is clear that nearly everything the USFS provides under the PAPA is some form of “technical assistance.” The second aspect comes from the Foreign Assistance Act section 621(a) that provides that “the facilities and resources of other Federal agency shall be utilized when such facilities are particularly or uniquely suitable for technical assistance, are not competitive with the private enterprise, and can be made available without interfering unduly with domestic programs.” It is clear that in many forestry-related areas the USFS is “uniquely” suited to providing T/A and that there does not seem to be any interference with domestic forest work from participation in the PAPA. The other part of this restriction however does have repercussions for the PAPA: T/A provided should not be “competitive with the private enterprise.” It is not clear that this rationale is consistently met when establishing buy-ins. For example it seems difficult to justify the use of the USFS to manage grants to Brazilian NGOs under the Forest Enterprise Cluster program (under which the USFS also provides broad technical guidance, oversight and integration of programs). Certainly the expertise exists in Brazil and among international NGOs or for profit consulting companies to manage grants to these institutions. In West Africa the management of grants to international NGOs seems justified early on when there was significant instability in the region and there were no clear candidates to manage such a program. It is not clear this is still the case, however, and STEWARD could likely be managed through a more standard procurement process. It is likely that there are other buy-ins where this is the case since nearly 20% of buy-in funds are sub-granted out. Another area where the Forest Service might not be uniquely suited to providing T/A is in the delivery of 118/119 assessments and ETOAs (Environmental Threats and Opportunities Assessment). These are activities that are commonly carried out by private sector firms and are likely in direct competition with the private sector. EVIDENCE-BASED OUTCOMES AND IMPACTS: MONITORING, EVALUATION AND REPORTING The ability to disseminate information and findings, while satisfactory, was identified as an area in need of some improvement. Several respondents, including USFS-IP personnel, suggested that there was a need to improve reporting and dissemination of reports. Some USAID personnel specifically cited reporting as the only area where they felt USFS performance was weak. Other USAID staff felt they knew very little about the scope of USFS-IP PAPA activities and felt they could benefit by having access to information about the range of activities being carried out either within their geographic or technical areas of concentration. These respondents felt they could benefit from a better understanding of these activities in order to look for connections and opportunities for synergy. During the first few years of operation the current PAPA produced annual report narratives that included bullet statements describing all buy-in activities by region, this practice was discontinued by mutual agreement in 2011, although it does not seem to have been made official through a written amendment to the agreement. MEASURING IMPACT – PAPA EVALUATION REPORT 40 FINDINGS All of the priority buy-ins examined are producing data on USAID standard indicators and Missions are satisfied with reporting on these indicators and data quality. That said, only STEWARD, in West Africa was collecting M&E data beyond the standard indicators, and respondents were very aware that this makes it difficult to assess the real impact of the work being done under the PAPA on recipients of T/A. One respondent stated: “without baselines on the outcomes and impacts we cannot say for sure what the results are leading to.” This is a common difficulty with programs based largely on training. Evaluation research has shown that training can be well designed, well executed and successful in transferring knowledge but still not lead to the outcomes and impacts sought by development practitioners. The USFS-IP recognizes the need for better monitoring and evaluation, in particular as a means to improve learning and adaptive management. At the same time USAID is working, through the Evaluation Policy and USAID Forward, to hold other partners to a higher standard when it comes to providing evidence of the impact of their activities. It seems this would be a natural evolution for the next PAPA agreement. EVIDENCE OF OUTCOMES FOR WOMEN AND DISADVANTAGED GROUPS The USFS-IP provides disaggregated data in their PMP’s on the participation of women in their training activities. All USFS-IP managers stated that it is a priority for them to ensure women are included in workshops and training. RDMA support included developing a gender strategy to ensure women and disadvantaged groups received equitable benefits from project activities and supporting a gender specialist until that person was taken on by an NGO partner (LEAF). The STEWARD Program is currently finalizing a gender strategy. STEWARD was the only buy-in which was able to show significant outcomes for women with the creation of savings and loan associations that are directly targeting and benefiting women. It is widely recognized today that ensuring better outcomes for women and disadvantaged groups is key to long term development and NRM success and requires more than just encouraging participation in workshops and training. This is another area that USAID is stressing with NGOs and contractors that should probably be reflected in the next PAPA. At the same time it is worth noting that 2/3 of the USFS-IP Technical Cooperation staff, and all of the USFS-IP regional coordinators, are women. While this doesn’t guarantee that programs will necessarily provide significant outcomes for women it does provide a good role model for counterpart host-country agencies. PAPA CONTRIBUTIONS TO THE ACHIEVEMENT OF USFS-IP PURPOSE AND OBJECTIVES The International Forestry Cooperation Act of 1990 provides a legal basis for the USFS to play a role in international forest conservation. It provides “the Secretary of Agriculture with further authority to undertake various activities to promote sound management and conservation of forests and related natural resources in other countries…” A key aim of the legislation was to support tropical forest conservation through assessments and training. The USFS International Program (formerly the Forestry Support Program) was established to fulfill this mandate. The PAPA contributes significantly to the USFS-IP work internationally. The work done under the PAPA directly supports the improved management of forests and it provides USFS employees with exposure to new ideas that may be applicable to the US context. Some anecdotal examples:     Community Based Forest Management is an international approach which is not commonly used within the US but which allows USFS personnel to see different ways to interact with communities (a key part of the USFS work domestically). Work in tropical forest certification improves the market for legally harvested timber increasing returns for US timber companies. Learning better ways to integrate satellite and on the ground monitoring of forests. A better understanding of how to estimate the carbon content of wetlands (peat, etc.) that can apply to Alaska and other Northern States. There is no structured evaluation of the impacts of USFS-IP activities on the transfer of skills and other benefits to USFS staff domestically. That said overseas assignments to work on USFS-IP activities are coveted by domestic staff and may help boost morale and increase knowledge exchange. Many detailers provide “brown bags” in DC or back in their home offices after completion of assignments to bring lessons and experiences home. 41 MEASURING IMPACT – PAPA EVALUATION REPORT CONCLUSIONS AND RECOMMENDATIONS FOR BEST PRACTICES CONCLUSIONS AND RECOMMENDATIONS FOR BEST PRACTICES This section presents conclusions from the evaluation’s findings and analyses that are matched with recommendations for best practices, or simply actions that should be taken by USAID or the United States Forest Service (USFS). TASK 4: IDENTIFYING KEY RECOMMENDATIONS/BEST PRACTICES FOR THE CONTINUED USE OF THE PAPA IMPROVING COMPLIANCE WITH USAID RULES AND REGULATIONS The USFS is for the most part doing a good job of complying with USAID rules and regulations in the implementation of Participating Agency Program Agreement (PAPA) buy-ins. The following recommendations are intended to address existing weaknesses. GENERAL An out-of-date PAPA template and Standard Provisions make understanding compliance requirements unnecessarily challenging. USAID should revise the templates for the PAPA Schedule and Standard Provisions to reflect USAID’s current policies. USFS should develop a compliance monitoring tool that can be used to assess compliance for each project/buy-in. The tool would include questions on branding, Initial Environmental Examination (IEE) compliance, ADS 253 Participant Training for Capacity Development, and grantee compliance with applicable regulations such as anti-terrorism screening and branding. REPORTING REQUIREMENTS The reporting requirements in the PAPA Schedule for reporting to E3/FAB are not concise and comprehensive, thus they may result in reports not submitted on time and/or reports may lack the necessary content for tracking progress towards program objectives. USAID should revise the PAPA Schedule and Standard Provisions to clearly define reporting requirements in terms of both timing and content, for the quarterly, annual, and final reports that are submitted to E3/FAB. The Standard Provisions should be revised to specifically state that in addition to centralized reporting to E3/FAB, Missions may require quarterly or semi-annual reports as well as final reports, with the content and timing of the reports to be determined by the Missions for their respective projects/buy-ins. An annual program report template that incorporates all reporting requirements for reporting to E3/FAB should be developed jointly by USAID and USFS. Any changes to the template would require mutual written agreement of the parties. USAID and USFS should consider going back to a format that presents a bulleted summary of PAPA accomplishments by region. This would enhance coordination if the report was shared among interested parties in related offices and bureaus. This could be particularly helpful if the USFS-IP included hyperlinks in the document that would allow interested readers to reach out to the appropriate contact at USFS-IP to get more information. This could go a long way to helping improve information dissemination about the scope of activities to interested parties within USAID. GRANTS/SUBGRANTS MANAGEMENT In general, USFS program managers are providing proper oversight of grants and subgrants. However, USFS did not comply with the audit provisions of the PAPA by failing to include in the grant agreements with non-US recipients the USAID audit threshold mandated by the PAPA, and by failing to ensure that the required financial audits of non-US recipients were performed, USFS should prepare and implement a plan of action for completing all required financial audits for non￾U.S. nongovernmental organizations that received more than $300,000 annually in funding under the PAPA. MEASURING IMPACT – PAPA EVALUATION REPORT 42 CONCLUSIONS AND RECOMMENDATIONS FOR BEST PRACTICES USFS should consider revising the USFS grant template to lower the audit threshold to $300,000 in annual funding for non-US recipients of grant funds under the PAPA. REGULATION 216, AGENCY ENVIRONMENTAL PROCEDURES USFS complied with the mitigation measures that were stipulated by the supplementary IEEs issued for the STEWARD (West Africa) and STAR (Uganda) programs. However, USFS program management staff were not aware of the PAPA IEE conditions that require USFS to advise USAID when a separate IEE is needed for a specific project or buy-in under the PAPA. USAID should raise awareness of USFS program management staff regarding the compliance requirements of the PAPA IEE. BRANDING USFS implementing partners that are first-tier grantees or contractors are knowledgeable about the branding requirements under the PAPA and correctly use the USAID Identity on communications products and other program materials. Lower-tier recipients of funding under the PAPA need to be better informed about the USAID branding requirements. USFS should ensure that the USAID guidelines on branding are communicated to all personnel implementing USAID-funded activities, including lower tier recipients, who may be directly producing or managing the production of communications products under the PAPA. This could be accomplished through a 2-3 page guidance document that includes the relevant information from the USAID Graphic Standards Manual and the PAPA Standard Provisions, examples of correct uses of the USAID Identity, and a contact name for a USFS staff member who can answer any USAID branding questions related to co-branding of communications products produced under the PAPA. USAID should consider updating the PAPA Standard Provisions on Marking and Communications Products to refer to the relevant sections of the USAID Graphic Standards Manual. ANTI-TERRORISM SCREENING The anti-terrorism screening procedures used by USFS and METI are in compliance with the “Support to Terrorism” clause of the PAPA and provide reasonable assurance that USAID funds will not inadvertently be used to support terrorism. USAID should update the anti-terrorism clause in the PAPA template to the following March 2002 version currently in use for other types of instruments: “Executive Order on Terrorist Financing (March 2002) The Contractor/Recipient is reminded that U.S. Executive Orders and U.S. law prohibits transactions with, and the provision of resources and support to, individuals and organizations associated with terrorism. It is the responsibility of the contractor/recipient to ensure compliance with these Executive Orders and laws. This provision must be included in all subcontracts/sub-awards issued under this agreement.” FINANCIAL MANAGEMENT Financial Reports As a result of the preliminary findings of this report, USFS conducted a detailed analysis to identify the source of the variances between its internal financial reports and the quarterly financial report submitted to E3/FAB, and has largely eliminated these variances. Going forward, USFS should conduct this type of analysis on a quarterly basis to ensure the accuracy of financial reports submitted to E3/FAB. Field Finance Operations With regard to field operations, the relative risks of different operating modes in the field should be assessed and balanced against the effectiveness of each mode. Because operating through field offices presents a higher risk of fraud than managing funds from the USFS-IP office in Washington DC, USFS should consider minimizing the use of field offices unless the effectiveness of the technical program dictates this type of operating mode. 43 MEASURING IMPACT – PAPA EVALUATION REPORT CONCLUSIONS AND RECOMMENDATIONS FOR BEST PRACTICES In those cases where the field office mode of operations is deemed to be the most effective, implementation of the USFS’ overseas office operations manual, currently in the final stages of development, is recommended. The field operations manual best practices should include current procedures and practices designed to ensure sound financial management such as:      Review of monthly bank reconciliations by the field office and USFS-IP staff; Monthly desk review of expenses and supporting documentation by USFS-IP staff; Frequent site visits for the purpose of reviewing financial and internal control procedures; USFS-IP and field office joint quarterly funds reviews of expenditures, projections, and projected procurements; and Hiring only certified accounting professionals for positions that are responsible for carrying out field accounting procedures. In addition, it is recommended that USFS-IP institutionalize conducting an annual financial review, internal audit, or audit by an external audit firm of all project expenditures and internal controls for field offices. Unexpended Advances The use of advance funding for the PAPA was discontinued in 2012; therefore, once USFS either expends all FY2012 obligations, there will no longer be a compliance issue in relation to ADS 602.3.2, which states that “the upper limit of pipelines normally is not expected to exceed 18 months of anticipated expenditures for on-going activities.” In November 2013, USFS presented to USAID a plan for expending by the second quarter of FY2015 the remaining unexpended funds from FY2012 and prior years’ obligations. USAID should monitor closely the implementation of this plan and work together with USFS to ensure that the plan stays on track and that advanced funds are fully expended by the end of the second quarter of FY2015. Although ADS 306.3.12 provides deobligation procedures for IAA closeout upon either reaching the Completion Date or one year after the completion of all activities, the ADS does not provide procedures for deobligating excess program funds prior to the completion of an IAA. Thus it is unclear how the deobligation process would be carried out if unexpended funds remain at the end of the second quarter of FY2015. Going forward, USAID should ensure that it continues to monitor and manage obligated balances in accordance with the ADS Chapter 621, Obligations. IMPROVING THE TECHNICAL AND MANAGERIAL PERFORMANCE OF THE PAPA USAID staff are overall quite satisfied with USFS-IP technical performance under the PAPA. These recommendations are designed to address current challenges and improve performance over the long term by addressing both technical and managerial challenges. IMPROVING THE SELECTION OF DETAILERS Detailers from within the ranks of the USFS are the heart of the technical assistance provided by the USFS-IP under the PAPA. Detailers, even those that are very technically sharp, that are ill equipped to deal with the realities of international assignments in developing country contexts are not effective. At the same time the pool of USFS staff that have tested overseas experience is too small to cover all the PAPA needs. The USFS-IP has a need to ensure opportunities for a broad range of USFS staff while providing T/A that is effective in the context required by USAID. The USFS-IP has been working to improve detailer selection for some time and developed in 2012 a set of best practices covering the entire process of managing a detailer assignment, from selection to in￾country management. The only recommendation we have for this process is to explore the potential for adding a more structured training or orientation step that would confront potential detailers with the realities of assignments in developing country contexts and may encourage self-reflection among potential detailers. The Peace Corps may be a good source of information for this kind of training with its emphasis on cultural sensitivity and adaptability, as may be USAID training for new DLI staff. MEASURING IMPACT – PAPA EVALUATION REPORT 44 CONCLUSIONS AND RECOMMENDATIONS FOR BEST PRACTICES IMPROVE MONITORING AND EVALUATION (M&E) USAID and other donors are putting a renewed emphasis on evidence-based programing and understanding the real impacts of the work done by international development practitioners. USAID is expecting better integration of M&E into development practice that leads to learning and adaptation of approaches to achieve desired outcomes and impacts. The USFS-IP could greatly benefit from improved M&E that leads to real learning about the impacts of technical assistance. There have already been discussions within the USFS-IP to develop an internal M&E team that can lead on this topic. The key will be finding a formula for working with USAID to jointly define the outcomes and impacts desired from T/A assignments and then evaluating the actual results. This goes beyond simple pre- and post-testing of training recipients to understanding the longer term impacts of T/A. It would not be necessary, or even possible, to track each and every buy-in in such a way but a system for randomly selecting buy-ins for assessment (before work actually begins) would allow an M&E team to get involved early in the definition of outcomes and impacts sought and baseline conditions. Randomly stratifying buy-ins based on location, technical area, and delivery modes would allow the USFS-IP to begin to produce actionable evidence of the real impacts of its work. This M&E work would not necessarily need to be done by the USFS; USAID could contract an outside firm to carry out these activities as they are doing in many programs. IMPROVE JOINT PLANNING AND COORDINATION AT THE MISSION LEVEL It is clear that PAPA activities are more effective in Missions where there is a good integration of USAID, USFS and implementing partners in the planning and execution of buy-ins (e.g. supporting RDMA work in climate change). The USFS-IP has found over the years that one off buy-ins and buy-ins where they are distanced from and do not have any control of the context in which the buy-in takes place is a recipe for uncertain results. The USFS-IP has found that in situations where they have more control over the context (such as Brazil FEC), or are able to more collaboratively plan activities with USAID (such as RDMA), they feel their activities are more effective. In some cases the USFS desire to play a broader role in controlling the context of their T/A may be taking it beyond its mandate and putting it more and more into direct competition with the private sector (including non-profit NGOs). These and other communication issues highlighted in this report illustrate the importance of consistent and clear points of contact as well as clear roles and responsibilities by both USFS, USAID and other partners. We recommend that a best practice model be developed detailing a more collaborative form of planning between USAID Missions, USFS and implementing partners at the Mission level. The model should be flexible allowing for multiple scenarios (one off buy-ins, urgent needs, and multiple USG funding sources) and should take into account improved M&E (above) activities. This should also include the detailing of a combined work plan for a given country or Mission that shows the range of USFS activities taking place there no matter who is funding the work (USFS-IP, DOS, etc). This would help USFS partners to better understand the full extent of activities and allow them to coordinate and plan accordingly. IMPROVE THE INTEGRATION OF CONCERNS FOR WOMEN AND DISADVANTAGED GROUPS INTO THE DESIGN OF BUY￾INS As mentioned above, and has been found in other development endeavors by NGOs and contractors, the USFS main actions to integrate women and disadvantaged groups into their T/A is to try to ensure access to PAPA training by women. While this is important and the PMPs show a significant number of female participants receive training, there is much more to ensuring positive impacts for women and disadvantaged groups. Missions and USAID partners also struggle with this important requirement for effective development. If the Mission and its partners have done a good job of assessing potential impacts to women and disadvantaged groups then improved collaborative planning (see above) should include helping the USFS-IP incorporate gender aspects into their Scopes of Work. In situations where this is not the case the USFS will need to be able to deliver gender audits (such as have been done under STEWARD and RDMA support) to ensure positive impacts on women and disadvantaged groups. The USFS-IP should invest in the development of a gender strategy that ensures it is taking advantage of opportunities to improve outcomes for women and disadvantaged groups whenever possible. INCORPORATE A STANDARD FOR LIMITING THE POTENTIAL FOR COMPETITION WITH THE PRIVATE SECTOR As mentioned above most T/A provided under the PAPA is justified under the current rationale for IAA’s as defined by the FAA Section 621 (a) and the ADS Section 103.3.8.4. Issues of competition with the private sector over the implementation of some buy-ins could be argued and should be addressed by 45 MEASURING IMPACT – PAPA EVALUATION REPORT CONCLUSIONS AND RECOMMENDATIONS FOR BEST PRACTICES USAID in a proactive fashion. We recognize that the implementation of development programs is a dynamic process and that a one-size-fits-all standard can lead to reduced effectiveness. For instance the use of the PAPA to cover management of the STAR project in Uganda while a new procurement process could be put into place was an excellent use of the mechanism that ensured USAID did not lose time nor significant financial investment. However it seems clear that in some cases the PAPA is being used in situations where a more open procurement process is needed. We recommend that USAID develop a standard by which USAID Missions and bureaus can judge whether or not a potential buy-in should be implemented by the private sector or the USFS. To do this it would be advantageous for USAID to include representatives from Missions, NGOs and contractors as well as the USFS-IP in a discussion of what these standards should look like and how they might be implemented. Once a standard is established it can be used to assess current buy-ins and recommend transitioning those that may be competing with private sector to open procurements. DEVELOP A FEEDBACK MECHANISM TO TRACK SATISFACTION WITH T/A As mentioned the PAPA is a large and complex mechanism. There are concerns at the E3/FAB level that there is not adequate awareness within USAID of the full range of PAPA activities and no real feedback on the quality of T/A being provided that is shared within USAID. Improvements in reporting (outlined above) will help to alleviate some of these concerns. Another recommendation is to design and require use of an online survey that would provide standardized input on Mission staff satisfaction with buy-in implementation. The survey would be required at the end of each buy-in (and/or before seeking subsequent buy-ins) and would allow Mission staff to rate their satisfaction on a range of variables important to the success of T/A. E3/FAB staff would have access to the raw survey results and could work periodically with the USFS-IP to review and discuss feedback provided. IMPROVE SHARING ON THE SCOPE OF PAPA ACTIVITIES Several USAID staff voiced concerns over a lack of information about the scope of PAPA activities. There was a concern that opportunities for synergy were being missed because of a lack of information on the full range of activities being implemented in a given region or within a given technical theme (GCC, Biodiversity, etc.). A first step in addressing this would be to bring back the practice of producing more detailed annual reports that include bullet statements of the major activities implemented under the PAPA by region and/or technical theme. A further step could include periodic (every 6 months) thematic and/or region focused meetings (i.e. Africa, LAC or Asia) between USFS-IP and interested USAID technical staff to share information about the range of activities being implemented under the PAPA in that region or under that technical area. ENCOURAGE MORE DIRECT (MISSION AND BUREAU LEVEL) PAPA AGREEMENTS There is concern among USAID staff that the PAPA has grown too large to be effectively managed from E3/FAB. We recommend the development and adoption of thresholds or standards that could be used to trigger discussions with Missions or bureaus to encourage them to shift towards a direct PAPA agreement with the USFS-IP. This shift to direct agreements is already happening in cases where investments are becoming significant in both time and finances. USAID would need to work with the USFS to identify the level of funding, number of buy-ins, length of time or some combination thereof, which could be used as thresholds or standards to trigger these discussions. MEASURING IMPACT – PAPA EVALUATION REPORT 46 ANNEXES ANNEXES ANNEX I: EVALUATION SCOPE OF WORK ANNEX II: EXAMPLES OF SURVEY INSTRUMENTS USED ANNEX III: LIST OF PERSONS INTERVIEWED ANNEX IV: LIST OF DOCUMENTS REVIEWED ANNEX V: COMPLIANCE DESK REVIEW ANNEX VI: BRANDING COMPLIANCE TABLE ANNEX VII: DATABASE FIELDS 47 MEASURING IMPACT – PAPA EVALUATION REPORT ANNEXES ANNEX I: EVALUATION SCOPE OF WORK I. Background USAID has re-committed, through the Evaluation Policy, to measure and document project achievements and shortcoming so that the Agency’s multiple stakeholders gain an understanding of the return on investment in development activities. The policy states that evaluation at USAID has two primary purposes: accountability to stakeholders and learning to improve effectiveness. The Measuring Impact project (MI) is an effort to enhance the impact of biodiversity and forestry programs throughout USAID through improved knowledge, evidence-based programming and adaptive management. An important component of MI is to enhance the capacity of USAID’s Bureau for Economic Growth, Education, and the Environment, Forestry and Biodiversity Office (E3/FAB) and selected USAID Missions for adaptive management by operationalizing new policies and by undertaking monitoring and evaluation (M&E) and capturing and disseminating lessons learned of select USAID forestry and biodiversity programs. The USFS PAPA has been identified by MI as a USAID-funded project under E3/FAB that will be useful for demonstrating a structured methodology for performance evaluation. The PAPA enables USAID and the USFS to collaborate in the development and implementation of projects addressing natural resource management issues worldwide, and enlists USFS expertise to support USAID forestry programs. Illustrative areas of technical assistance provided by the USFS to USAID include:             Protected area management and biodiversity conservation Fire ecology, prevention, response and impact mitigation Forest monitoring, remote sensing and geographic information systems Greenhouse gas (GHG) emissions reduction and enhanced carbon sequestration from land use management and forestry, including REDD+ assessments, carbon measurement methodologies, and low emission development strategies (LEDS) Tree-based biofuels production Community forestry Agro-forestry Smallholder wood production systems Regional forest planning Invasive species and forest pest/disease management Disaster response and preparedness, including Incident Command Systems Governance of natural resources. Assistance provided by the USFS includes a variety of forms, including field-based technical assistance, formal and informal training programs, policy and economic analysis, study tours, and applied research to promote individual and institutional capacity building. The PAPA has been the major mechanism from which E3/FAB provides technical support to missions and bureaus in the areas of forest management and conservation. Through the services of the USFS, USAID missions and bureaus are able to acquire expertise to assist in the design and implementation of projects and programs related to forest conservation and forest management. In addition to coordinating USFS assistance to USAID, the PAPA provides an opportunity for both agencies to leverage external resources and form mutually-beneficial partnerships with outside organizations. Such organizations include host-country government agencies, multilateral organizations, non-governmental organizations, and private sector entities that share USAID and USFS goals of improving natural resources management and forest conservation. This partnership is intended to help USAID address regional and cross-sectoral issues that cannot appropriately be addressed bilaterally. It also provides a platform from which USAID can support presidential initiatives and participate in multilateral forums. MEASURING IMPACT – PAPA EVALUATION REPORT 48 ANNEXES II. Purpose Consultant will provide senior technical expertise and support to USAID Forestry and Biodiversity (E3/FAB) Office’s Measuring Impact (MI) project to determine the extent to which the USFS PAPA is being used and performing towards its intended objectives, and to provide recommendations on how its use and performance may be improved to increase the overall effectiveness of this Inter-Agency Agreement (IAA). III. Scope of Work, Deliverables & Timing Consultant will provide senior technical expertise in the area of data analysis, survey design and implementation, informant interviewing and qualitative analysis for the PAPA evaluation under the MI project. The Scope of Work detailed below was agreed to by E3/FAB and constitutes the work plan for the Consultant, pending results of the database analysis (Deliverable A). Following a review of the database summary, changes in the number of buy-ins to be evaluated as well as any adjustments to the evaluation questions will be incorporated into the scope of work. The Consultant will engage with USAID and USFS staff in Washington and in the field, USFS partners, and USFS detailers to respond to the evaluation needs. Consultant will work closely with the MI team to develop a final report and presentation on the PAPA evaluation’s findings. These tasks will be carried out between 5 August 2013 and 31 January 2014, according to the approximate timeframes described with the deliverables below. Although the intermediate deliverables listed below (e.g., database analysis, desk study summary, etc.) will be reviewed and approved by MI to inform the overall evaluation, they may be made available to USAID for internal use and as such should be delivered in a well-written and useful format (i.e., good draft form), whether or not they are included as appendices to the final report. Task 1: Improve our understanding of how the PAPA is being used by USAID Operating Units to meet their development objectives The Consultant will analyze the existing USFS PAPA buy-in Access database and respond to the evaluation questions described in the appendix to the scope of work (see below). The database contains all the USFS buy-ins and the necessary fields to conduct the requested analysis. Should the Consultant require additional information related to the buy-ins, s/he will consult with the MI Technical Supervisor to determine the feasibility of gathering this information. Deliverable A: Summary of results of database. The summary will include tables and/or graphics for each of the questions highlighted in the Appendix to the Scope of Work along with any additional analyses considered relevant by the Consultant. The summary report will also include narrative explaining any noteworthy findings associated with each of the analyses performed. Task 2: Assess the technical, program management, and financial performance of the current PAPA. The Consultant will review program documentation provided to him by MI, USAID and USFS staff to understand the context of each selected buy-in and to document the objectives, implementation experience and expected and actual results for each buy-in. The final list of buy-ins to be evaluated will be determined in consultation with USAID following the completion of the database analysis. Using a survey tool designed by the Consultant and approved by the MI Technical Supervisor, the Consultant will conduct key informant interviews with up to 10 program staff for each buy-in to assess whether performance of the buy-in was aligned with technical, management and financial expectations. The Consultant will receive support from the team in selecting and providing contact information for stakeholders. The Consultant is expected to schedule interviews or other modes of data collection with stakeholders. The Consultant will also develop and conduct an e-survey with USFS detailers to gather their perspectives on assignments undertaken by them. Deliverable B: Summary of desk study in the form of a memo highlighting buy-in objectives, management considerations, and documented results. Deliverable C: Approved survey instruments. 49 MEASURING IMPACT – PAPA EVALUATION REPORT ANNEXES Task 3: Assess the contribution of the USFS PAPA to the purpose and objectives of both E3/FAB and USFS-IP. Is the PAPA meeting the purpose and scope of the IAA? The USFS PAPA was put in place to fill an identified need by both the USFS and USAID. These needs and opportunities are spelled out in the PAPA agreement. Following a review of selected buy-ins, the Consultant will assess to what degree the buy-ins, individually, and/or collectively, are meeting these stated objectives. Deliverable D: Key Informant Interview Summary Report. The Consultant will provide a narrative report highlighting the key findings from key informant interviews. The report will include the interview guide used and identify the individuals interviewed along with their relationship to the program. Deliverable E: Survey Monkey e-survey for PAPA detailers. Survey will validate information gathered from key informant interviews. Deliverable F: Tabulation report of all returned e-surveys. This report will tabulate frequency of responses and characteristics of respondents and will not include analysis of the data. A discussion of findings and implications will be included in Deliverable G. Task 4: Based on the findings from the assessment of use and performance from above, identify key recommendations/best practices for the continued use of the PAPA to maximize the value to USAID Operating Units and ensure compliance with applicable regulations. MI has engaged a separate Consultant to provide an assessment of USFS compliance with selected PAPA regulations. The compliance assessment will be completed in sufficient time for the Consultant for this set of tasks to incorporate the compliance findings into the narrative related to the overall performance of the PAPA. Following the completion of tasks 1 through 4 above, the Consultant will present a draft evaluation report to be reviewed and approved by the MI Technical Supervisor. This draft report will be shared with the PAPA AOR, the IR2 Activity Manager and members of the MI IR2 team for review and comment; USAID may choose to request comments from other individuals for incorporation into the final version. Comments will be provided to the Consultant for incorporation into the de-briefing PowerPoint presentation to be delivered in person to USAID, MI and USFS staff (Deliverable H) and the Final Evaluation Report (Deliverable I). Deliverable G: Draft Evaluation Report. The Consultant will deliver the Draft Evaluation Report to the MI Technical Supervisor in hardcopy (1 copy), in addition to an electronic version. The report will be a comprehensive report of all deliverables associated with this subcontract, and will:          represent a thoughtful, well-presented, well-researched, and well organized effort to objectively evaluate what worked in this project, what did not work, and why; be a high quality technical report, in a professional writing style; address all questions included in this scope of work; include all the key sections: cover sheet, table of contents and acronym list/glossary of terms, executive summary, background, objectives, questions, methods, findings, conclusions and lessons learned, and recommendations; include the scope of work as an appendix; include an introduction that adequately describes the project, explains where it is implemented, includes contextual information, and includes the “theory of change” or development hypotheses that underlie the project; describe the methodology in detail and all tools used such as questionnaires, checklists, and discussion guides will be included in an appendix in the final report; describe findings with gender disaggregate outcomes and impacts; describe any limitations to the methods (e.g. selection bias, recall bias, unobservable differences between groups, etc.);      MEASURING IMPACT – PAPA EVALUATION REPORT 50 ANNEXES present the findings as analyzed facts, evidence, and data and should not be based on anecdotes, hearsay, or a compilation of opinions. Findings should be specific, concise, and supported by strong quantitative or qualitative evidence; properly identify sources of information and list them in an appendix; clearly distinguish between conclusions, findings, and recommendations; support any recommendations by a specific set of findings; and provide recommendations that are action-oriented, practical, and specific with defined responsibility for the action. Deliverable H: Evaluation de-briefing presentation. An in-person PowerPoint presentation of the evaluation process and findings intended for USAID and USFS staff. Deliverable I: Final Evaluation Report. IV. Consultant Profile Paul Cowles has over 20 years of international and domestic experience, specific to conservation and USAID (Africa and Asia), US Forest Service (Idaho), Bureau of Land Management (Alaska), and 15 years of assessment and evaluation experience. Paul has technical expertise in results-based project design using conceptual modeling and theory-of-change approaches to develop practical results chains, as well as in qualitative and quantitative research and data analysis, including the ability to manage large databases. Paul has experience in detailed desk reviews of program documents, as well as Extensive experience developing and carrying out key informant interviews as part of assessment and evaluation data collection methodologies. V. Budget and LOE A summary budget is provided in Table 1. Budget values will be a primary performance management tool in this contract, however only the budget total is a not-to-exceed amount. An estimated cost and effort distribution is provided in Table 2. Expenditures deviating more than 10% from the budget distribution must be agreed to in writing by both parties before expenses are incurred. LOE is provided for convenience and is not intended to be used for performance management in the same way as budgets. Table 1. Summary Budget Category Budget (US Dollars) LOE (100 days @ $544.50/day) $54,500 Travel (lodging, meals, incidentals for 2 field visits and up to 2 domestic roundtrip tickets Seattle – DC for Consultant to conduct interviews and present evaluation findings) $11,751 Total $66,251 51 MEASURING IMPACT – PAPA EVALUATION REPORT ANNEXES Table 2. Estimated level of effort, cost and delivery dates for Consultant delivery of products. Description LOE (days) COST Date of Delivery Deliverable A: Summary of data analysis 4 $2,178.00 August 16, 2013 Deliverable B: Summary of desk study 16 $8,712.00 September 15, 2013 Deliverable C: Approved e-survey instruments 1 $544.50.00 September 15, 2013 Deliverable D: Key Informant Interviews and write-ups 40 $21,780.00 September 30, 2013 for highest priority and November 30, 2013 for others Site travel to specific buy-ins (if necessary) 12 $6,534.00 TBD Deliverable E: Written survey 5 $2,722.50 October 20, 2013 Deliverable F: Evaluator to collate survey data and analyze all data 5 $2,722.50 November 15, 2013 Deliverable G: Draft Report 10 $5,445.00 December 15, 2013 Deliverable H: Evaluation de-briefing presentation 2 $1,089.00 January 15, 2014 Deliverable I: Final Report 5 $2,722.50 January 31, 2014 Total 100 $54,450.00 VI. Logistical Support Consultant is based in Seattle, WA and will conduct most of the assignments under this subcontract remotely. Consultant will be responsible for arranging his own logistical support for this assignment, however the MI project will assist Consultant with in-country logistics and the USFS and/or USAID missions may provide additional support. Any international travel will require USAID approval and electronic Country Clearance (eCC). VII. Oversight and Management Consultant will report to and be under the supervision of the MI Technical Supervisor, Roberto Martin. Consultant should work with other MI and USAID staff as appropriate and necessary to fulfill work requirements. MEASURING IMPACT – PAPA EVALUATION REPORT 52 ANNEXES APPENDIX TO SCOPE OF WORK NOTE: Prior to the initiation of surveys or interviews, the questions in this appendix may be refined through agreement with MI, USAID and the Consultant. Key questions, methods and indicators for Task 1: Evaluation Question: What is the breakdown of “buy-ins” by funding amount, region, country, funding source (e.g. mission), technical area, and delivery mode being supported under the PAPA. Methods: Analysis of USFS PAPA buy-in database Indicators:           Average funding amount for a buy-in Frequency by region, country, and funding source, technical area, and delivery mode, direct beneficiary Total funding amount by region, country, funding source, technical area, and delivery mode, direct beneficiary Funding amount each year by region, country, funding source, technical area, delivery mode, buy-in size, and direct beneficiary Analysis of observed patterns over time in the use of the PAPA buy-in mechanism, including: value of buy-ins technical area focus of buy-in recipients of TA provided by buy-in geographic distribution of buy-in requests modes of technical assistance Evaluation Question: According to the breakdown above, what are the predominant (i.e. receiving most funding from “buy-ins”) technical themes and delivery modes being implemented under the IAA? Methods: Analysis of USFS PAPA buy-in database Indicators:     Technical areas receiving most funding Delivery modes receiving most funding Direct beneficiary receiving the most funding Combinations of the above 3 categories receiving most funding Key questions, methods and indicators for Task 2: Technical performance Evaluation Question: What is the quality of the detailers and the technical assistance/training provided by USFS-IP to USAID Operating Units for the most relevant technical areas and delivery modes? Methods: Interviews/survey with USAID and USFS PAPA activity managers, field technical staff, and in￾county recipients of USFS TA and/or training who have received technical assistance over the past 2 years. Prepare and test questions in advance. Qualitative Indicators (rating on a scale, provide examples)    Was the detailer able to target the specific technical needs of recipients? Did the detailer have the required level of experience/expertise to provide the needed technical assistance, including: knowledge of the languages and culture of the countries?       53 MEASURING IMPACT – PAPA EVALUATION REPORT ANNEXES knowledge of the natural resource issues? ability to make decisions about program the direction and sequence of activities? ability to disseminate information/findings? ability to facilitate cross-region learning? Did the detailer seem credible in their delivery of the training/technical assistance? For example, was s/he capable of persuading recipients that their approach to solving the issue at hand would be successful? Did the detailer demonstrate responsiveness and/or flexibility to the specific needs of the recipients? For example, was s/he willing to listen to recipients and/or able to fulfill an originally unanticipated need of the recipient? Evaluation Question: What was the effectiveness of the detailers and technical assistance/training provided by USFS-IP to USAID Operating Units for the most relevant technical areas and delivery modes? Methods: Review of any USFS best practices for relevant technical areas. Interviews/survey on the phone and/or in the field with USAID and USFS PAPA activity managers, field technical staff, and in￾county recipients of USFS TA and/or training of the most relevant technical areas, delivery modes, and direct beneficiaries. The respondent will be asked to indicate whether their assessment is based on direct observation, discussions with partners directly involved, or a general impression. Test questions will be prepare and tested in advance. Analysis of the USFS database will help the evaluation team to determine whether the size of the buy-in, number of beneficiaries, or complexity of the program warrant a field visit in addition to remote key informant interviews. Qualitative Indicators (rating on a scale, provide examples):       Was there a strategic link between the technical assistance provided and the USAID mission’s ongoing or planned activities? Did the technical assistance deliver specific, actionable recommendations which were acted on by the Mission or partners? If not, why not? Did the right people receive the technical assistance/training? Was the technical assistance/training appropriate (technology/cost) for the recipients? Did the technical assistance result in important outcomes for the recipients (e.g. improved management of forests, reduction in threats to forests, improved conservation status of forests)? M&E and data quality (DQAs)   DQAs are conducted at global level, but do any large buy-in missions with multi-year programs also conduct DQAs? Should they conduct them? Program Management Evaluation Questions: 1. What is the USFS International Program’s performance in managing the PAPA for the most relevant technical areas and delivery modes? 2. Are the tools that USFS International Program uses to manage the PAPA adequate (e.g. mechanisms to allow USFS, spend funds, and track budgets and results), or do they need improvement? 3. Does the relatively recent trend of having USFS-IP place or hire staff overseas to help manage USAID-funded projects significantly improve project implementation, coordination and communication? What are the tradeoffs regarding cost, efficiency, and coordination/communication? MEASURING IMPACT – PAPA EVALUATION REPORT 54 ANNEXES Methods: Interviews/survey with USAID and USFS PAPA activity managers, field technical staff, and in￾county recipients of USFS TA and/or training. Prepare and test questions in advance. Qualitative Indicators related to Program Management:                    ability to respond to requests, flexibility to modify programs as requested level of contact and communication with Operating Units regarding program planning and implementation knowledge of the relative strengths and weaknesses of the array financial and administrative mechanisms to obligate USAID funds and implement programs knowledge of the languages and culture of the countries knowledge of the natural resource issues ability to make decisions about program the direction and sequence of activities ability to disseminate information/findings ability to facilitate cross-region learning coordination of multiple-donor or multiple-agency programs (an increasingly common role for USFS-IP) ability to develop and maintain partnerships ability to select and manage technical detailers ability to communicate with Missions and Bureaus ability to manage overlapping buy-ins from several sources ability to coordinate interagency relations and communication timeliness and content of reporting timeliness and effectiveness of follow up activities, if needed quality of the guidance to detailers/providers of TA on communication and reporting protocols quality of guidance to field based USAID liaison on communication and reporting protocols clarity with respect to accountability for oversight of detailer performance Evaluation Question: Is the management of the PAPA in compliance with rules and regulations applicable to USAID? Methods: Review rules and regulations, identify potentially high risk activities within the buy-in portfolio, randomly select from high risk activities and assess level of compliance through interviews and related paperwork. Provide examples. Evaluation Question: How were funds used relative to budgets, and what is the perceived value by Missions relative to the actual value? Methods: Review of the planned budgets (including USFS cost-share) and actual financial reports for select buy-ins? Interviews with Mission staff responsible for buy-ins. Evaluation Question: What are the requirements (or expectations) for cost-share under the IAA or specific buy-ins? What is the cost-share (auditable) provided by USFS for management of the IAA and direct implementation? If cost-share is not a reporting requirement under the buy-in, but is under the program being supported, how is cost-share being calculated? Methods: USFS financial data reporting. Data will determine how we assess cost-share provided. Quantitative Indicators:   Cost-share $ provided by USFS for management/buy-in  Small vs. large projects Cost-share $ provided by USFS for direct implementation/buy-in  Cost-share may be in-kind services – e.g. server space. Evaluation Question: What are the funds leveraged from non-USAID sources? Are leveraged amounts linked to buy-ins or to the program being supported by the buy-in? 55 MEASURING IMPACT – PAPA EVALUATION REPORT ANNEXES Methods: The Consultant will determine if, and how, USFS collects and reports leveraged funds data. Quantitative Indicators:  $ leveraged/buy-in Key questions, methods and indicators for Task 3: Evaluation Questions: 1. Has the established rationale for implementing an IAA mechanism between USAID and the USFS been adequately met? Methodology: review of buy-in requests. Qualitative Indicators:   Unique skills provided by USFS Providing government functions 2. What is the evidence that USFS technical assistance activities have achieved outcomes for intended recipients (including women and other disadvantaged groups) in support of the stated purpose and objectives of E3/FAB? Methods: USFS-IP data Quantitative Indicators:   # of people trained, disaggregated by gender (if available) # of TA days During the life of the PAPA, some data collected for results may not have been disaggregated by gender, which will limit the analysis. The evaluator(s) will review program monitoring plan to determine if attention to gender and/or disadvantaged groups was reflected in the data collection plan. If so, evaluator will assess whether program gender data was collected and used for adaptive management. 3. What is the evidence that USFS technical assistance activities/delivery modes have achieved outcomes towards the stated purpose and objectives of USFS-IP? 4. Have some technical assistance activities/delivery modes achieved greater outcomes than others? Methods: Evidence from interviews/survey of Detailers, USFS-IP staff USFS-IP data and reports. Discussions, if not structured interviews, with U.S. Embassy personnel or others not directly involved in the buy-in. Quantitative Indicators:   # of USFS personnel given international experience # of activities with host country governments, etc. Qualitative Indicators:        Exposure to innovations by foreign agencies/organizations Contribution to issues of international significance (e.g. combating illegal logging, invasive species) Contribution to US foreign policy and security issues Morale boost for Detailers Efficient use of time of seasonal/part-time employees, retirees Knowledge exchange when Detailers return (e.g. brown bags) Cultivation of collaborative relationships (e.g. Malawi)  MEASURING IMPACT – PAPA EVALUATION REPORT 56 ANNEXES Building of long-term relationships (e.g. Mexico) Key questions, methods and indicators for Task 4: Evaluation Questions: 1. Based on the evaluation findings, what recommendations can be made to increase the financial, program management and technical performance of the PAPA? 2. What categories (e.g. technical areas, delivery modes) of USFS-IP-managed activities most benefit USAID and its partners? For activities that are less beneficial, how can they be improved or better aligned with the objectives of the PAPA and USAID? 3. Based on the evaluation findings, what best practices already developed by USFS or additional best practices should be incorporated into the Scope of Work for future buy-ins to the PAPA from USAID Operating Units for those technical areas and delivery modes assessed? 4. Based on the evaluation findings, what are the parameters for using the PAPA versus alternative mechanisms? Are there particular technical areas which are more or less appropriate for an IAA mechanism, or for USFS assistance? 5. USFS-IP has developed a number of mechanisms to allow it to manage USAID-funded programs, spend funds, and track budgets and results. How can these be leveraged or improved for better management of the overall agreement by both USAID and USFS? 6. In what situations does it make sense for Missions to develop direct PAPA or PASA agreements with the USFS, rather than going through the E3/FAB PAPA? What are the tradeoffs regarding coordination of the overall USFS/USAID relationship, the administrative burdens placed on E3/FAB and USFS-IP, and the role of E3/FAB in monitoring and promoting Mission programs? How are the different mechanisms for engaging the USFS promoted or publicized and are their relative advantages accurately described? Methodology: Key informant interviews with staff from both USFS and USAID. Possible follow up items after the evaluation:   Develop a template for a SoW with best practices for predominate technical areas and delivery modes. Provide guidance on roles and responsibilities for compliance with rules and regulations (e.g. IEE standards – Missions’ responsibility to comply, E3/FAB’s responsibility to ensure they comply.) AUDIENCE, INTENDED USES AND DISSEMINATION OF EVALUATION FINDINGS The audiences for the PAPA evaluation findings include USAID (both E3/FAB and field missions), the USFS International Program, and the broader biodiversity conservation community including donors, practitioners, and governments. Per the USAID Evaluation Policy, the findings from the evaluation will be shared as widely as possible, with a commitment to full and active disclosure. Furthermore, a summary including a description of methods, key findings and recommendations will be made available by USAID to the public on-line in a fully searchable form. EVALUATION METHODOLOGY A) Data Collection and Analysis Methods The Consultant will use a combination of qualitative and quantitative methods to collect and analyze the data as described in the Tasks section (see methods and indicator for each Task and Question). The Consultant will use secondary and primary data sources. 57 MEASURING IMPACT – PAPA EVALUATION REPORT ANNEXES Collection and Analysis of Secondary Data: In terms of secondary data, the Consultant will conduct the analysis of the PAPA database of buy-ins to answer the questions in Task 1. The Consultant will then summarize the findings from the data analysis and recommend priorities to PAPA AORs and USFS-IP staff for buy-ins to be evaluated under Task 2. PAPA AORs and USFS-IP staff will then confirm the list of priority buy-ins to be evaluated in detail under Task 2, and any secondary data related to the priority buy-ins to be evaluated in more detail. The Consultant will then conduct a desk study of available secondary data for each priority buy-in in order to document and synthesize project outcomes, and provide PAPA AORs and USFS-IP with a summary. Data will be disaggregated by gender and geographic region, to the extent possible from secondary sources. Existing monitoring data/reports that USAID staff will provide to the Consultant include the following:         Action Memo Amendments Annual USFS work plans Quarterly reports Annual PMP reports Final reports USFS Financial data for priority buy-ins USFS information on best practices for relevant technical areas Other documents or sources of information that will be useful to the Consultant include the following:   government or international data local partner organization data Collection and Analysis of Primary Data: In terms of primary data, the Consultant will use survey instruments and structured interviews to solicit information from key informants for the priority buy-ins. Key informant interviews will be informal, guided conversations with individuals who are particularly knowledgeable about a specific topic. Key informants will include USAID (mission and Washington) and USFS (field and Washington), partners and beneficiaries, and other individuals familiar with the assistance provided under the PAPA. To develop a comprehensive list of key informants to be interviewed, the Consultant will work with the PAPA AORs and USFS-IP staff. For interviews and survey, the evaluator(s) will aim to get solid representation on the following aspects:     Types of technical assistance and deliver modes of USFS support under the PAPA Type of organization or agency (e.g., USAID, USFS, NGO and government partners and beneficiaries) Geography Knowledge about the strategies implemented through the PAPA The Consultant will receive support from the Evaluation Team in selecting and providing contact information for key informant. The evaluator(s) is expected to schedule interviews or other modes of data collection with key informants. The Consultant will work with the Evaluation Team to develop key informant topic guides that are comprised of a series of focused questions centered on a specific topic to be addressed by a set of relevant key informants. After most of the key informant interviews are completed, the Consultant will design a survey that will be sent to individuals. The purpose of this survey is to build on and deepen the preliminary analysis of the key informant interviews. A secondary purpose of the survey will be to reach a wider set of individuals – and thus get an even broader perspective than that provided by key informants. In addition, the more MEASURING IMPACT – PAPA EVALUATION REPORT 58 ANNEXES quantifiable nature of the online survey will complement the more qualitative key informant interviews. Finally, the results of the survey will help to identify important issues that require follow-up with additional key informant interviews or site visits. Once key informant interviews are completed, the Consultant will collate and analyze the data, and summarize the findings and recommendations to answer the questions under Task 4. 59 MEASURING IMPACT – PAPA EVALUATION REPORT ANNEXES ANNEX II: EXAMPLES OF SURVEY INSTRUMENTS USED USAID Mission Personnel (Activity Managers) Discussion Guide 1. Please describe the PAPA activity you are managing. 2. How did this activity come about? How was the USFS chosen to implement it? Why was the FS a better choice than some other procurement mechanism? 3. How does this activity fit into the overall Mission/bureau’s ongoing activities? Does it respond to specific intermediate results or objectives from the Mission/bureau’s strategic plan? 4. Is the Activity meeting your needs in terms of contributing to the Mission/bureau’s strategic plan/logic model/results framework? Why or why not? 5. How does your bureau/Mission monitor progress and performance of USFS activities under the PAPA? 6. Besides the Initial Environmental Examination (IEE) were any environmental impact assessments done for activities that could result in direct environmental impact? 7. What are (some of) the results delivered by the PAPA for your Mission/bureau? 8. What evidence do you have (quantitative or qualitative) that these results are having or may lead to significant outcomes or impacts? Prompts: Are the behaviors of recipients of USFS support changing in any appreciable way? Are they using the knowledge they gain from USFS T/A? 9. What types of T/A by the forest service are most effective? Why? Least Effective? Why? 10. What constituents / targets is the most appropriate target of interventions? Who is FS best suited to supporting with T/A? Least suited? 11. What has been done to ensure issues of gender and disadvantaged groups are addressed in the activity? 12. Were the funds expended as agreed, to implement the activity? Explain. 13. Was there an expectation of cost share or leveraging of non-USAID funds under this activity? Were those expectations met? Explain. 14. What aspects of the activity are not going well? 15. How would you improve the activity? (in terms of implementation or management). 16. How would you rate your satisfaction with the following aspects of the USFS PAPA? a. (Using a scale of 1-10 where 1 is not satisfied at all and 10 is completely satisfied) b. Responsiveness to the Mission/bureau’s requests _____ Why? c. Management of the activity ______ Why? d. Ability to develop and maintain partnerships _____ Why? e. Communication with Mission/bureau ____ Why? f. Achievement of results ______ Why? g. Cost effectiveness _______ Why? h. Ease of use of the mechanism ______Why? MEASURING IMPACT – PAPA EVALUATION REPORT 60 ANNEXES i. Technical expertise of detailers ______ Why? j. Ability of detailers to deliver needed results _____ Why? k. Level of contact and communication ______ Why? l. Follow up after T/A (reporting, communication) ______ Why? m. Timeliness of reporting on results _____ Why? n. Content of reporting ______ Why? o. Monitoring and evaluation of impacts of the activity _____ Why? USFS PAPA Manager Discussion Guide 1. Please describe the activity you are managing. 2. How did this activity come about? How was the USFS chosen to implement it? 3. How does this activity fit into the overall Mission/bureau’s ongoing activities? Does it respond to specific intermediate results or objectives from the Mission/bureau’s strategic plan? 4. If grants have been awarded under this activity, describe how these grants are managed and monitored and what you would do to improve these processes. 5. How is USAID support of activities communicated to program beneficiaries and other stakeholders? 6. What are (some of) the significant results delivered by the T/A? 7. What evidence do you have that these results are having or may lead to significant outcomes or impacts? 8. What have been the most important and/or significant outcomes or impacts of the activity? 9. What has been done to ensure issues of gender and disadvantaged groups are addressed in the activity? 10. Were the funds expended as agreed to implement the activity? Explain. 11. Was there an expectation of cost share or leveraging of non-USAID funds under this activity? Were those expectations met? Explain. 12. What aspects of the activity are not going well? 13. How would you improve the activity? (in terms of implementation or management). 61 MEASURING IMPACT – PAPA EVALUATION REPORT ANNEXES COMPLIANCE KEY INFORMANT INTERVIEW GUIDE FOR USFS-IP MANAGEMENT (DC) General and Reporting: 1. The PAPA completion date is 4/1/2012, does this mean that USAID support under the PAPA ended in April 2012 or has it continued beyond that date? 2. Was there an amendment to extend the PAPA? For example, the SWAMP Project Proposal is dated August 2012 and covers the period 2012-2015. 3. Under some buy-ins semi-annual reports are submitted, under others reporting is quarterly . Why is this? Grants: 4. How are grants awarded? Directly by USAID or by USFS? 5. Is there a standard grant agreement template? If so, please provide a copy of the template. 6. How is grantee performance monitored and by whom? 7. Have there been any grants that require an audit in accordance with USG regulations >$300,000/year in expenditures of federal funds. If grantees receive USG funds from more than one source how is this tracked for audit compliance purposes? Environmental Compliance (IEE): 8. How and by whom is IEE compliance (Regulation 216) monitored in the field and at the HQ level? 9. Does USFS maintain a central repository of IEE documents related to the PAPA? Anti-terrorism screening – METI contractor and USFS-IP: 10. How is anti-terrorism screening carried out? Besides the SDN list are any other lists checked? 11. Is a central repository of SDN checks maintained and if so, who has access to the records? 12. How does USFS (or party responsible for screening) ensure that all contractors, subcontractors, grantees, subgrantees, and individual consultants are screened? 13. Are just the organization names checked or are the principals in charge of the organization (Pres., V.P., etc.) also checked against the SDN list? Financial tracking systems: 14. Describe the financial tracking systems and tools used by USFS in managing the buy-ins under the PAPA. 15. Are expenses recorded when the obligation is incurred (for example, a two-year contract is signed) or when the obligation is liquidated and the expenditure actually occurs? 16. What causes unspent balances to carry over multiple fiscal years and how are these unspent balances being resolved? 17. What happened in Sierra Leone with STEWARD and the fraud investigation? MEASURING IMPACT – PAPA EVALUATION REPORT 62 ANNEXES COMPLIANCE DISCUSSION GUIDE FOR AORS 1. What reports do you receive and what is the frequency of reporting? 2. Besides the Initial Environmental Examination (IEE) were any environmental impact assessments done for activities that could result in direct environmental impact? 3. What can you tell me about branding of project sites, program activities, and communications products produced under the PAPA? 4. What is your level of involvement with the grantees? How are grantee activities monitored? 5. Is there anything else you would like to tell me about the PAPA? COMPLIANCE DISCUSSION GUIDE FOR USFS PROGRAM MANAGERS (DC & FIELD) 1. What reports are submitted to the Mission and what is the frequency of reporting? 2. If grants have been awarded under this activity, describe how these grants are managed and monitored and what you would do to improve these processes. 3. What can you tell me about branding of project sites, program activities, and communications products produced under the PAPA? 4. Besides the Initial Environmental Examination (IEE) were any environmental impact assessments done for activities that could result in direct environmental impact? 5. Is there anything else you would like to tell me about the PAPA? 63 MEASURING IMPACT – PAPA EVALUATION REPORT ANNEXES ANNEX III: LIST OF PERSONS INTERVIEWED U.S. FOREST SERVICE                       Alex Moad, Assistant Director for Technical Cooperation Vanessa Felder-Pinkney, Grants and Agreements Specialist Ronald Ingram, Financial Manager Sarah Banks, Africa Program Specialist Michael Rizo, Latin America, Caribbean & Canada Urban and Community Program Specialist Shelley Gardner, Illegal Logging Program Coordinator Christa Anderson, Climate Change Specialist Lauren Chitty, Africa Program Specialist Brehan Doud, Africa Program Specialist Annie Nagy, Africa Program Specialist Jim Beck, Africa Program Specialist Cynthia Mackie, Asia-Pacific Program Coordinator Elizabeth Lebow, Asia-Pacific Program Specialist Lara Peterson, Russia, Europe and Near Asia Program Coordinator Liz Mayhew, Latin America, Caribbean & Canada Program Coordinator Michele Zweede, Latin America, Caribbean & Canada Program Specialist Camille McCarthy, Latin America, Caribbean & Canada Program Specialist Rob Clausen, USFS/USAID Environment Officer, Haiti Geoff Blate, USFS Field technical coordinator at RDMA Jan Broekhuis, Director STEWARD Program, Sierra Leone Joe Krueger, USFS Flathead National Forest, Montana Mathew Edwardson, Former USFS-IP staff member USAID                Rebecca Butterfield, E3/FAB Olaf Zerbock, E3/FAB Alexandre Mancuso, USAID/Brazil Rebecca Carter, USAID/Uganda Robert Senkungu, USAID/Uganda Sudi Bamulesewa, USAID/Uganda Nicodeme Tchamou, USAID/West Africa Erik Streed, USAID/Indonesia Orestes Anastasia, USAID/RDMA Suphasuk Pradubsuk, USAID/RDMA Barry Flaming, Former USAID/RDMA Alicia Grimes, E3/FAB Tegan Blaine, USAID/AFR/SD Tim Resch, USAID/AFR/SD Evan Notman, USAID/E3/GCC METI, INC.  Alvin Johnson, Program Manager OTHER INTERVIEWEES        Joberto Veloso de Freitas, Director, Brazilian Forest Service Daniel Meireles Tristão, Brazilian Forest Service David Ganz, COP LEAF Program John Wells, COP LEAD Program Del McClusky, Director, DAI John Nitler, Director, Tetra Tech ARD An additional 28 USAID Mission staff and T/A recipients completed online surveys. MEASURING IMPACT – PAPA EVALUATION REPORT 64 ANNEXES ANNEX IV: LIST OF DOCUMENTS REVIEWED In addition to the documents and websites listed in the Environmental Compliance section of Annex V and the Branding Compliance Tables in Annex VI, the evaluators reviewed the documents identified in the tables below. Table 3: PAPA Documents - General DOCUMENT TITLE/TYPE OF DOCUMENT DOCUMENT FILE NAME(S)/ WEB SITE AUTHOR/ CONTRIBUTORS Action Memo PAPA with USFS-IP 2007 PAPA_Action_Memo.doc USAID PAPA Annex A, Program Description 2007 PAPA Program Description.doc USAID w/USFS PAPA Schedule and Annex C, Standard Provisions 2007 PAPA_Schedule.doc USAID PAPA Modification #5 PAPA_Mod 5.pdf USAID PAPA Modification #9 PAPA_Mod 9.pdf USAID PAPA Modification #14 cover sheet PAPA amendment 14 signed face sheet.pdf USAID PAPA Modification #14 program description and budget Amend 14 program description final.docx USAID w/USFS Final Evaluation 1997-2006 USAID and USDA/US Forest Service Interagency Agreement Forest Resources Management Project (AAG-P-00-97-00003-00) http://pdf.usaid.gov/pdf_docs/Pdach055.pdf Diamond, Nancy K., for METI, Inc. 65 MEASURING IMPACT – PAPA EVALUATION REPORT ANNEXES Table 4: USFS Documents DOCUMENT TITLE/TYPE OF DOCUMENT DOCUMENT FILE NAME(S)/ WEB SITE AUTHOR/ CONTRIBUTORS Scopes of work for PAPA buy-ins: Focus Programs for evaluation SOWS for evaluation.docx USFS Federal Financial Assistance Award of International Grant International_Grant_template.docx USFS Q3_2013 PAPA Quarterly Financial Report Q3_2013_PAPA Quarterly Report.xlsx USFS Q3_2012 PAPA Quarterly Financial Report Q3_2012 PAPA Quarterly Financial Report.xls USFS PAPA advance balances Nov 2013 PAPA advance balances Nov 2013.xlsx USFS Annual Report Summary FY2013 USFS 2013 PAPA Performance Indicators updated Dec 23.xlsx USFS Annual Report Summary FY2012 USFS 2012 PAPA performance indicators.xlsx USFS Annual Report Summary FY2011 2011 PAPA Summary Report2.xlsx USFS Annual Report Summary FY2010 2010 PAPA Summary Report.xlsx USFS Annual Report Summary FY2009 2011 PAPA Summary Report2.xls Table 5: Liberia Documents (Reviewed for IEE Compliance Only) DOCUMENT TITLE/TYPE OF DOCUMENT DOCUMENT FILE NAME(S)/ WEB SITE AUTHOR/ CONTRIBUTORS FINAL REPORT October 1, 2011 – May 31, 2012 Liberia Forestry Support Program (LFSP) USFS Grant 11-DG-11132762-363 LFSP_Final_Report_revised.pdf Peter De Waard Chief of Party, ACDI/VOCA Liberia Environmental Education U.S. Forest Service Technical Assistance Trip 16-28 October, 2011 Liberia_USAID_USFS_Env Ed report_Nov_2011.pdf Tamberly Conway, USFS SOWs for Liberia buy-ins Liberia buy-ins.docx USFS MEASURING IMPACT – PAPA EVALUATION REPORT 66 ANNEXES Table 6: Peru documents (reviewed for IEE compliance only) DOCUMENT TITLE/TYPE OF DOCUMENT DOCUMENT FILE NAME(S)/ WEB SITE AUTHOR/ CONTRIBUTORS Work Plan 2010-2011 Peru Forest Sector Initiative Work Plan Narrative 2010.docx USFS USAID/USFS Peru Forest Sector Initiative Annual Report FY 2010 USFS PFSI Annual Report 2010 FINAL.pdf USFS USAID / USFS Peru Forest Sector Initiative 4TH Quarter and Annual Report – FY 11 USFS PFSI Annual Report FY11.pdf USFS USAID / USFS Amazon Forest Sector Initiative Quarter Four and Annual Report – FY 12 AFSI FY12 Q4_FINAL.pdf USFS SOWs for Peru buy-ins Peru buy-ins.docx USFS 67 MEASURING IMPACT – PAPA EVALUATION REPORT ANNEXES ANNEX V: COMPLIANCE DESK REVIEW PAPA EVALUATION COMPLIANCE COMPONENT - DESK REVIEW REPORT INTRODUCTION AND SUMMARY OF WORK COMPLETED This document is the desk review report for the compliance component of the evaluation of the Participating Agency Program Agreement (PAPA) between the U.S. Agency for International Development (USAID) and the International Programs Office of the U.S. Forest Service (USFS) for the period 2007-2013. This report responds specifically to Task 1 of the scope of work for the compliance specialist. In this phase the USFS has provided the compliance specialist with documents as source information. In addition, the compliance specialist has obtained source documents online through the USAID Environmental Compliance Database, and through internet searches. Key informant interviews with USAID, USFS, METI and local partner organizations, took place concurrently during this phase of the work and in some instances may have informed the results of the desk review. This report should be read as a statement of findings at a point in time, rather than in conclusion of any particular line of enquiry. A key role of this report is to highlight the issues that appear to be important and will be developed in the later stages of the evaluation. The main findings at this stage concern reporting, grants management, environmental compliance, branding, anti-terrorism screening, and financial tracking tools. Although recommendations cannot be finalized on the basis of the desk review alone, issues the compliance specialist considers should be included in the final recommendations are identified at this point. The main focus of work during the desk review was to analyze compliance with the PAPA agreement for a sample of up to eight “buy-ins”. The majority of the time spent by the compliance specialist was devoted to review of documents for these projects, the PAPA Schedule and Standard Provisions, as well as online sources of information such as PAPA-funded websites and the USAID Environmental Compliance Database. OBJECTIVES OF THE EVALUATION The objectives of the evaluation are set out in Section II of the Compliance Statement of Work, and state that the compliance specialist will “determine the extent to which the USFS PAPA is being used and performing towards its intended objectives, specifically in the area of compliance with applicable USAID policies and directives. The compliance specialist will also provide recommendations on how the USFS PAPA’s use and performance in compliance may be improved to increase the overall effectiveness of this Inter-Agency Agreement (IAA).” In addition, the compliance specialist has been tasked with assessing compliance with the subsection of the USAID Automated Directive Systems (ADS) regulation for IAAs that states IAAs should be used for “inherently government functions”. DESK REVIEW SCOPE AND METHODOLOGY Program related documents such as performance reports, activity reports, trip reports, communications products, financial reports and grant agreements requested and received from USFS were reviewed in order to assess USFS compliance with the terms of the PAPA in the following five areas:      Progress, performance and task completion reporting Grant/Sub-grant management 22 CFR 216, Agency Environmental Procedures and IEE standards Acknowledgement of USAID support (branding) Anti-terrorism screening In addition to these five areas, the efficacy of financial tracking tools and compliance with ADS 306.3.2.10 were also assessed. MEASURING IMPACT – PAPA EVALUATION REPORT 68 ANNEXES Financial tracking tools were requested and received from USFS in order to determine the adequacy of the tools that USFS uses to manage the PAPA and whether they provide mechanisms that enable USFS to spend funds and track budgets effectively. Financial tracking is accorded a separate section as it is a centralized function that is handled for the most part by USFS-IP in Washington. Program documents were also used to assess compliance with ADS 306.3.2.10, PAPA – A Participating Agency Program Agreement (PAPA) and When USAID Should Use it to “Buy” Services from Another Federal Agency. ADS 306.3.2.10 compliance is also accorded a separate section as this is a matter of USAID compliance with its internal regulations, which is a separate issue from USFS compliance with the terms of the PAPA. This desk review was limited in scope to only those buy-ins specifically identified by USAID for review. Sources of information were mainly limited to documents provided by USFS and documents publicly accessible on the USAID website. Information was also obtained through discussions with USFS International Programs Assistant Director for Technical Cooperation, Mr. Alex Moad, and Sarah Banks, Africa Program Specialist. In accordance with the Statement of Work for the compliance component of this evaluation, a summary of the desk review follows. COMPLIANCE WITH PROGRESS, PERFORMANCE AND TASK COMPLETION REPORTING REQUIREMENTS Evaluation Questions - Reporting   Do buy-in bureaus/Missions receive a task completion report as outlined in the IAA provisions? What task completion reports do bureaus/Missions receive? PAPA Reporting Requirements Sections D.9, D.10 and D.12 of the PAPA Schedule and section F.2 of Annex C, Standard Provisions to the agreement establish the program reporting requirements for quarterly, annual and task completion reports. According to section D.9, annual reports of technical information related to each task must be submitted within 60 days of the end of each fiscal year. However, section F.2 requires that periodic progress reports must be submitted within 30 days of the close of the period they cover. Contrary to the phrasing of the evaluation question, the PAPA Schedule does not actually require that Task Completion Reports be submitted to the bureaus or Missions but requires only that the annual report submitted to USAID include a “Task Completion Report”, i.e., a final report, summarizing the activities and results from each bureau or Mission buy-in that came to a close during the reporting period. Section D.9 further stipulates “Annual reports may consist of copies of reports prepared for and submitted to the Mission partners if they meet the requirements of Section E (2)”. However, Section E(2) makes no mention of reports at all, although Section F(2)does, therefore it is assumed that this inconsistency is merely a typographical error and F(2) was intended. Section D.10 states that quarterly progress reports must include information on grants awarded to non￾US nongovernmental organizations in the amount of $300,000 or more per year. However, there is no mention in D.10, or in any other subsection under Section D, what other content quarterly progress reports should include. A discussion of reports actually submitted to the Missions for each project/region follows. Brazil FEC Semi-annual progress reports were submitted by FEC to the Mission for each six-month period from October 1, 2008 through September 30, 2011 STEWARD On May 6, 2011, a final report was submitted for STEWARD II. STEWARD III annual reports were submitted to the Mission for the periods April 1 – September 30, 2011 and October 1, 2011 – March 30, 2012. 69 MEASURING IMPACT – PAPA EVALUATION REPORT ANNEXES SWAMP The TWINCAM final report for October 1, 2011 – September 30, 2012 was submitted to the Mission. The first two quarterly reports were for the periods Oct-Dec. 2012 and Jan-Apr 2013. RDMA Quarterly program reports have been submitted to the Mission for every quarter beginning with the 3rd quarter of FY08 through the 4th quarter of FY13, ending September 30, 2013. Uganda For the STAR II program, quarterly reports for Oct-Dec 2011 and Jan-Mar 2012 were submitted to the Mission. The final report for STAR II for the period August 2011 – August 2012 was also submitted to the Mission. Note: It is not clear if the Aug-Sep 2011 and Apr-June 2012 reports were done but were not forwarded for the desk review or they were not done and therefore were not included in the package of documents for the desk review. Initial observations based on desk review In general, reports are submitted to Bureaus/Missions at regular quarterly or semi-annual intervals as well as annually. Final reports (task completion reports) are submitted to the Mission at the completion of each program. One program may consist of one or more buy-ins, (i.e., tasks) funded over one or more fiscal years. Reporting requirements in the PAPA Schedule are not concise and comprehensive, thus they may result in reports not submitted on time, not submitted quarterly, and/or reports may lack the necessary content for tracking progress towards program objectives. Task completion reports are not included in the annual reports submitted to E3/FAB. Follow-up for interviews Confirm with bureaus and Missions that both quarterly and final reports were expected and were submitted consistently and on-time by USFS for each buy-in. Initial conclusions and recommendations The PAPA Schedule and Standard Provisions should be revised to clearly define reporting requirements in terms of both timing and content, for quarterly, annual, and final reports. Section F(3) of the Standard Provisions should be revised to specifically state that reports may be required by the Missions in addition to centralized reporting to EGAT. GRANT/SUB-GRANT MANAGEMENT Evaluation Questions – Grants     Are applicable grant/sub-grant management rules and procedures being followed by USFS and their contractors? Do the grants/sub-grants contain the proper flow down clauses? What can be improved in grant/sub-grant management? Are Missions/activity managers providing proper oversight of grants/sub-grants? Compliance Standards - Grants 2 CFR 215, Uniform Administrative Requirements for Grants and Agreements with Institutions of Higher Education, Hospitals, and other Non-Profit Organizations (OMB Circular A-110), establishes the administrative requirements for Federal grants and agreements to non-profit organizations. These requirements also apply to subrecipients of Federal funding. MEASURING IMPACT – PAPA EVALUATION REPORT 70 ANNEXES The PAPA Standard Provisions section G.2, sets forth the requirements for audits of non-U.S. nongovernmental organizations receiving grant funds under the PAPA. For organizations receiving $300,000 or more in grant funds per year, an audit clause requiring an independent financial audit of the funds provided under such grants must be included in the grant agreement. Non-U.S. nongovernmental organizations receiving less than $300,000 per year are exempt from the financial audit requirements, but are subject to the requirement to make records available upon request for review by authorized U.S. Government officials. Other than the audit clause and the books and records clause, there are no flow-down clauses in the PAPA Schedule or Standard Provisions. Initial observations based on desk review USFS uses a grant agreement template that is approved by OMB and includes the necessary provisions to ensure compliance with 2 CFR 215, Uniform Administrative Requirements for Grants and Agreements with Institutions of Higher Education, Hospitals, and other Non-Profit Organizations (OMB Circular A-110). The USFS grant agreement template includes an audit provision with a threshold of $500,000 in annual expenditures of U.S. Forest Service awards rather than the $300,000 in “grant awards” threshold prescribed by the PAPA. Grantees receiving more than $10,000 but less than $500,000 annually in awards are subject to the requirement to make records available upon request for review by U.S. Forest Service officials or their designees. Initial conclusions and recommendations The USFS grant agreement template should be revised to include the USAID audit requirements. Follow-up for interviews Determine what procedures USFS is using to manage grants and monitor grantee performance. REGULATION 216, AGENCY ENVIRONMENTAL PROCEDURES Evaluation Questions - IEE   Where activities could result in direct environmental impact, has USFS advised the buy-in bureau/Mission that an additional impact assessment could be required? Has the buy-in bureau/Mission completed a separate IEE for those activities, and were the required work adjustments or mitigation measures stipulated in the separate IEE carried out? Compliance Standards - IEE Section F.2 of the PAPA Schedule sets forth the requirement for compliance with USAID environmental regulations codified at 22 CFR 216, Agency Environmental Procedures (“Regulation 216”). Per the Initial Environmental Examination (IEE), activities under the PAPA are subject to a negative determination with conditions. Section F.2 further stipulates that no further action is required on the part of USFS unless activities other than those described in the IEE categorical exclusion are undertaken under the PAPA. The PAPA IEE describes three categories of activities and how each should be handled by USFS: 7. For activities that will not result in any environmental impact, such as conducting training courses and workshops, participating in forums and one-on-one negotiations, information gathering Missions, etc., that are of the type that will not result in actions that could have a direct environmental impact (e.g. cutting trees, applying pesticides) no environmental review will be required. 8. For activities that include recommendations for activities that could result in some environmental impact (e.g.: providing TA on land-use strategies, forest management practices, silviculture recommendations, natural resource management options), recommendations made by USFS personnel will include an Initial Environmental Examination or Environmental Assessment under 71 MEASURING IMPACT – PAPA EVALUATION REPORT ANNEXES 22 CFR 216, or at least advise the Mission/bureau that is funding the TA of the need to undertake such analysis for 22 CFR 216 compliance. 9. Any activity that could degrade natural tropical, temperate or boreal forests, including but not limited to any kind of tree harvesting activities, changes in land use, removing vegetative cover, applying pesticides, etc., will include, as part of the scope of work for the technical assistance, a briefing/recommendation to the funding Mission/bureau, regarding the requirement to undertake a 22 CFR 216 Environmental Assessment. The USFS will provide expert advice on completing this USAID requirement, either through the use of USFS expertise, or with other resources. Although many of the activities carried out under the selected buy-ins fall into category 1, there were also activities that fall into category 2. In accordance with 22 CFR 216.2(c)(2)(xiv) an IEE is required for activities such as studies, projects or programs intended to develop the capability of recipient countries to engage in development planning if those studies, projects or programs are designed to result in activities directly affecting the environment. There were no category 3 activities noted in the documents reviewed. A discussion of compliance with the PAPA IEE/Regulation 216 for each bureau/Mission follows. Brazil FEC On September 16, 2009, LAC BEO issued an Environmental Threshold Decision (ETD), valid for the period 01 October 2009 – 30 September 2012, with a determination of Categorical Exclusion for USAID/Brazil Environment Programs, including FEC. On February 22, 2013 the ETD was extended for one year through 30 September 2013. The Brazil ETD Categorical Exclusion includes a reference to 22 CFR 216.2(c)(2)(xiv), which should have resulted in a separate IEE if any of the studies, projects or programs carried out were designed to result in activities directly affecting the environment. In fact, the FEC program included activities such as providing TA on land-use strategies, forest management practices, and natural resource management options, which were identified in category 2 of the PAPA IEE and required that at a minimum that USFS would have advised the Mission about the need for a separate IEE. However, the Brazil IEE supporting the ETD concluded that “all Forest Enterprise Cluster (USFS) activities include only components of training, education, technical assistance, studies, surveys, workshops, meetings, development of environmental monitoring and planning tools, therefore, there is no foreseen adverse impacts to the environment, and the activities qualify for a Categorical Exclusion.” Thus, the Brazil ETD precluded any further action on the part of the USFS with regard to additional IEEs for its activities under the Brazil FEC project. STEWARD The May 6, 2011 Final Report for STEWARD II, details the monitoring by STEWARD/USFS-IP of the mitigation measures stipulated in the EMMP for STEWARD II and describes how these measures, including measures that were incorporated into grant agreements with program partners, were carried out. Refer to pp. 36-43 of the STEWARD II Final Report for details. The West Africa Regional Mission initiated and carried out a separate IEE for STEWARD on June 9, 2012, therefore it was not necessary for the USFS to advise the Mission to do so. Based on the presentation of proposed activities and expected results in the STEWARD III Program Description of April 3, 2011, the IEE identified recommended actions for each activity and included: 1) Categorical Exclusion; Negative Determination; Negative Determination with Conditions; and Deferral. The specific and general monitoring and implementation requirements stipulated in the IEE are summarized in sections 3 and 4 of the IEE and include the development of an Environmental Mitigation and Monitoring Plan (EMMP). The STEWARD FY13 work plan also references implementation of the USAID-approved EMMP. SWAMP (formerly TWINCAM) The activities in SWAMP countries and regions can be grouped into several broad categories: studies, policy analysis through modeling, remote sensing and capacity building. More specifically, SWAMP has carried out activities such as providing technical assistance, training, analysis and organizational support MEASURING IMPACT – PAPA EVALUATION REPORT 72 ANNEXES to promote REDD+ readiness in targeted developing countries; collecting data; conducting research and studies; publishing papers; delivering field training in soil sampling and analysis; and conducting the Wetlands Global Carbon Survey. All the activities carried out under SWAMP (and TWINCAM) fall within category 1 of the PAPA IEE and/or were within one of the categorical exclusions found at 22 CFR 216 subparts (c)(1)(i), (iii), or (c)(2)(i), (ii), (iii) and (xiv), therefore no additional IEE was necessary for this program. RDMA PAPA activities under RDMA are primarily technical assistance, training, workshops and conferences, and information sharing and dissemination, thus there are no apparent activities that would be subject to an IEE under 22 CFR 216. Nevertheless, RDMA initiated an IEE that resulted in a Categorical Exclusion for USFS activities carried out under the Interagency Agreement with USAID. The IEE stated “No activities proposed under this program are expected to have a significant impact on the environment and all fall under one or more categorical exclusions outlined under 22 CFR 216(c)(2).” Uganda The STAR II program implemented the following types of activities: developed models to demonstrate ways in which tourism supports biodiversity conservation; facilitated cross-sector meetings and activities for tourism stakeholders; provided training and technical assistance in sustainable trail techniques; developed a tourism for conservation assessment tool; carried out a tourism and conservation policy study; and provided small grants for tourism infrastructure development. An IEE initiated by the Uganda Mission in August 2008 included the STAR project and recommended a Categorical Exclusion for activities involving technical assistance, education, training workshops, meetings, information analysis, research and studies, advocacy, capacity building, and resource provision. Thus, many of the activities implemented by the STAR II program were included under the Categorical Exclusion. A Negative Determination with Conditions was recommended for community natural resource conservation activities (including biodiversity conservation), with the condition that the communities are well sensitized and trained on the application of recommended practices for natural resources management. Based on the training and technical assistance activities conducted under STAR II, it can be concluded that this condition was met. A Negative Determination with Conditions for infrastructure development, such as protected area visitor centers, with conditions of identifying and mitigating any direct adverse impacts on the existing or surrounding physical environment arising from structure construction, and that “all construction activities will be conducted following principles for environmentally sound construction, as provided in Chapter 3: Small Scale Construction of the USAID Environmental Guidelines for Small-scale Activities in Africa (EGSSAA), which can be found at http://www.encapafrica.org/SmallScaleGuidelines.htm.” It was also recommended that all activities within the sub grants components should be subjected to the environmental review process in accordance with the USAID Africa Bureau Environmental Guidelines for Small-Scale Activities in Africa (EGSSAA). Liberia The main activities carried out under the Liberia Forestry Support Program were training and capacity building. There were no IEE implications related to these activities. The LFSP represents only about 20% of the funds expended during the course of the PAPA, additional information on Liberia activities is needed in order to fully assess compliance with 22 CFR 216. Peru The main activities carried out under the Peru program were training, workshops and capacity building; providing technical assistance in the planning, design and implementation of a National Forest Information System and Control Module; and facilitating communication among key actors in forest management in Peru. There were no IEE implications related to these activities. 73 MEASURING IMPACT – PAPA EVALUATION REPORT ANNEXES Initial conclusions and recommendations In cases where a project-specific IEE was warranted the IEE was initiated by the Mission of its own accord, not based on the advice of the USFS. The extent of compliance with required work adjustments or mitigation measures stipulated in the separate IEE varied by project, from complete compliance and thorough documentation by the STEWARD project, to a lack of documentation or evidence that the mitigation measures were carried out for the Uganda STAR II project’s tourism infrastructure and small grants activities. The inconsistencies between the PAPA IEE and the project-specific and Mission specific IEEs/ETDs, such as the Brazil IEE and ETD, the STEWARD IEE, and the Uganda IEE raises the question of which IEE should have taken precedence and how such inconsistencies should be handled in the future. Follow-up for Interviews Determine whether DC-and field-based personnel are aware of the Regulation 216 requirements and how would they handle any conflict between the existing IEEs. MEASURING IMPACT – PAPA EVALUATION REPORT 74 ANNEXES Table 7: USAID Environmental Compliance Database Documents DOCUMENT TITLE/ TYPE OF DOCUMENT DETERMINATION LINKS TO USAID ENVIRONMENTAL COMPLIANCE DATABASE AUTHOR / CONTRIBUTORS PAPA IEE 2007, Feb. 2007, covering the period 2007-2012 Negative with Conditions http://gemini.info.usaid.gov/egat/ envcomp/repository/pdf/136.pdf USAID PAPA IEE Amendment 1, Sep. 2007 Negative with Conditions http://gemini.info.usaid.gov/egat/ envcomp/repository/pdf/287.pdf USAID PAPA IEE Amendment 2, Oct. 2007 Negative with Conditions http://gemini.info.usaid.gov/egat/ envcomp/repository/pdf/288.pdf USAID PAPA IEE Amendment 3, June 2010 Negative with Conditions http://gemini.info.usaid.gov/egat/ envcomp/repository/pdf/2238.pdf USAID PAPA IEE Amendment 4, Feb. 2011, extends the IEE through April 1, 2017 Negative with Conditions http://gemini.info.usaid.gov/egat/ envcomp/repository/pdf/37376.p df USAID Brazil ETD 2009, Sep. 2009, for the period October 2009 – September 2012 Categorical Exclusion and Negative with Conditions http://gemini.info.usaid.gov/egat/ envcomp/repository/pdf/1581.pdf USAID Brazil ETD 2013, Feb. 2013, extends original ETD and IEE through Sep. 2013 Categorical Exclusion and Negative with Conditions http://gemini.info.usaid.gov/egat/ envcomp/repository/pdf/39630.p df USAID STEWARD III IEE, June 2012, for the period 2011- 2015 Categorical Exclusion, Negative with Conditions, and Deferral http://gemini.info.usaid.gov/egat/ envcomp/repository/pdf/38547.p df USAID /Cadmus RDMA IEE, Sep. 2005, for the period 2005-2008 Categorical Exclusion http://gemini.info.usaid.gov/egat/ envcomp/repository/pdf/2796.pdf USAID RDMA IEE Amendment 1, May 2008, extended original IEE through 2011 Categorical Exclusion and Negative with Conditions http://gemini.info.usaid.gov/egat/ envcomp/repository/pdf/2828.pdf USAID RDMA IEE Amendment 2, April 2012; extended original IEE through 2014 Categorical Exclusion http://gemini.info.usaid.gov/egat/ envcomp/repository/pdf/37311.p df USAID RDMA IEE Amendment 3, May 2012; extended original IEE through 2017 Categorical Exclusion http://gemini.info.usaid.gov/egat/ envcomp/repository/pdf/38491.p df USAID Uganda IEE 2009-2014 Categorical Exclusion and Negative with Conditions http://gemini.info.usaid.gov/egat/ envcomp/search.php [search “Uganda”] USAID COMPLIANCE WITH BRANDING REQUIREMENTS - ACKNOWLEDGEMENT OF USAID SUPPORT 75 MEASURING IMPACT – PAPA EVALUATION REPORT ANNEXES Evaluation Questions – Branding   If USAID funding is used to finance equipment, materials, or communications products such as reports, pamphlets, or videos, are these items marked with the USAID red, white, and blue logo and brandmark? How is USAID support of activities communicated to program beneficiaries? Compliance Standards - Branding According to ADS 320.3.4.1 entitled “Programs, Projects, or Activities Funded through Agreements between Participating U.S. Government Agencies or Other Donors and USAID”, USAID’s policy is that programs, projects, activities, public communications, or commodities implemented or delivered in cooperation with other U.S. Government agencies or other donors must be co-branded and marked appropriately, in accordance with the terms of the applicable interagency agreement. Section N. Communications Products of the PAPA Standard Provisions defines “Communications Products” as “any printed material (other than noncolor photocopy material), photographic services or video production services”, and establishes the requirement for compliance with USAID standards for Communications Products financed by USAID. Although not explicitly stated, this is generally understood to mean compliance with the USAID Graphic Standards Manual found at: http://www.usaid.gov/sites/default/files/documents/1869/USAID_Graphic_Standards_Manual.pdf, which requires co-branding with the USAID Identity (i.e., the red, white, and blue USAID brandmark and the USAID tagline “From the American People”) for projects that are co-funded. In addition, this manual requires that all USAID-funded websites meet certain technical and accessibility standards that are defined on the XWeb section of the USAID website. Unfortunately the referenced XWeb page is currently inaccessible on the USAID website. Section R. Marking of the PAPA Standard Provisions requires that “all USAID-financed equipment and materials must be marked with the USAID red, white, and blue emblem; and all construction sites and other locations receiving USAID financing must display signs marked with the USAID red, white, and blue emblem and indicating participation by the United States of America. These signs should be erected at an early date in the construction or implementation phase and be replaced by permanent signs, plates or plaques, marked with the USAID red, white, and blue emblem, at the end of this phase.” A discussion of compliance with branding guidelines by each bureau/Mission follows. Brazil FEC The Brazil FEC draft proposal stated, “It is important to note, that USAID – Brazil Mission will receive full credit for all program activities and products.” Clearly, this indicates an awareness of the requirement to acknowledge including the USAID Identity on its primary communications product, the “Boletim” in accordance with the terms of the PAPA and current USAID branding policy for co-funded projects. No other communications products were included in the documents for review. Although not a requirement of the USAID branding policy, program reports were also branded. STEWARD STEWARD communications products that were required to be branded with the USAID Identity were not, whereas program reports that were not required to be branded were. The STEWARD project website is branded with the STEWARD logo at the top left of the homepage rather than the USAID Standard Graphic Identity as required by the USAID Graphic Standards Manual. Based on the documents and the website reviewed, “STEWARD” has been established as a brand that supersedes the USAID brand on program reports, communications products, and the USAID-funded STEWARD website. This is contrary to current USAID branding policy. MEASURING IMPACT – PAPA EVALUATION REPORT 76 ANNEXES SWAMP The SWAMP flyer shows the USAID Identity, which is the same size and prominence as the USFS, USFS-IP and CIFOR logos at the bottom of the second page, whereas the SWAMP identity is at the top of the first page. The SWAMP website is branded with the USAID Identity of the same size and prominence as the USFS and CIFOR logos as the bottom of the page. However, only the CIFOR logo is at the top of the page and it is the only logo visible upon loading the page. The user has to scroll to the bottom of the page to see the USAID logo. RDMA Nearly all the communications products and reports reviewed for branding compliance included the USAID logo with correct placement on the document. In several cases where the logo was missing, USAID support was acknowledged in the text. Uganda The STAR project has established its own brandmark that is used on communications products and program materials alongside the USAID Identity and USFS logo. The USAID Graphic Standards Manual discourages branding of individual projects. Photos embedded within communications products show the USAID Identity on buildings constructed with partial funding from USAID. In a training report a photo shows the USAID Identity on a banner at the training site. The STAR final report (pp. 11 and 44) shows use of a USAID-branded banners at a project sponsored events. The STAR final report (p. 37) also shows use of a USAID branded plaque at the entrance to a building constructed with USAID funds. Two out of three active project sponsored websites, Pearls of Uganda and Birding Uganda, are correctly branded with the USAID Identity. On the Uganda Wildlife Authority (UWA) website USAID support is acknowledged in the footer of the page. Since UWA is a semi-autonomous government agency it may have deemed inclusion of the USAID Identity as inappropriate. Such exceptions to USAID branding policy are allowed per the Graphic Standards Manual. The Friend-A-Gorilla (http://www.friendagorilla.org) and EcoTrust Uganda websites could not be accessed. Initial Conclusions and Recommendations Although the level and extent of compliance with USAID branding requirements varies from project to project, all focus areas showed at least an awareness of the branding requirements and attempted to comply, albeit to a greater or lesser extent, depending on the project and, in some cases, depending on the level of involvement of third parties in the development and dissemination of communications products. The USAID Graphics Standards Manual advises against giving precedence to a project brand rather than the USAID Identity. However, it was seen that in at least two of the focus areas, STEWARD and SWAMP, the project brand was given precedence over the USAID brand. The PAPA Standard Provisions for Communications Products and Marking should be updated to refer to the relevant sections of the USAID Graphic Standards Manual. The definition of “Communications Products” should also be updated to reflect the fact that although quarterly and annual program reports may include color photos and graphics, i.e., they are color photocopy materials, per current USAID policy they are not required to be branded. USFS should ensure that the USAID guidelines on branding are communicated to all personnel, including lower tier recipients, who may be directly producing or managing the production of communications products under the PAPA. This could be accomplished through a two or three-page guidance document that includes the relevant information from the USAID Graphic Standards Manual and the PAPA Standard 77 MEASURING IMPACT – PAPA EVALUATION REPORT ANNEXES Provisions, examples of correct uses of the USAID Identity, and a contact name for a USFS staff member in Washington who can answer any USAID branding questions related to co-branding of communications products produced under the PAPA. Follow-Up for Interviews Find out more about what is actually being done on the ground as far as marking project sites, activities, and/or equipment (if any). ANTI-TERRORISM SCREENING Evaluation Questions – Anti-terrorism   Are the policies regarding for screening all recipients of USG funding being followed? Is the appropriate documentation being kept in the files? Compliance Standards – Anti-terrorism PAPA Schedule section F.7, Support to Terrorism, states: “The USFS is reminded that U.S. Executive Orders and U.S. laws prohibit transactions with, and the provision of resources and support to, individuals and organizations associated with terrorism. It is the legal responsibility of the USFS to ensure that all sub agreements, contracts, and grants issued under this Agreement comply with these Executive Orders and laws.” Anti-Terrorism Screening Policies and Procedures: Anti-terrorism screening procedures carried out by METI for all employees, consultants, and contractors providing services to, and on behalf of, USFS under the PAPA are in accordance with METI’s written policies and procedures, which also includes sample documentation of the test wire and the issue resolution. The procedures include an initial SDN check of each individual and legal entity, followed by a test wire through the banking system, which triggers an SDN and OFAC check by the Federal Reserve, to further confirm that no potential SDN or OFAC issues exist. In the event that a test wire is initially rejected due to a potential SDN or OFAC issue, METI’s practice is to investigate the issue and ensure that no actual SDN or OFAC issue exists. Anti-terrorism screening procedures for grants are conducted by USFS in accordance with its internal grant procedures, which include checking the names of grantees against the SDN list. Grantee responsibility for compliance with Executive Order 13224, Executive Order on Terrorist Financing, is established by a standard provision in the grant agreement. This provision includes a flow-down clause that requires grantees to apply E.O. 13224 compliance procedures to lower-tier recipients. Initial Conclusions and Recommendations for Anti-Terrorism Screening: METI follows anti-terrorism screening policies and procedures that are in compliance with the anti￾terrorism clause of the PAPA, and maintains adequate documentation to show compliance with its policies and procedures. The anti-terrorism clause in the PAPA template is not the most recent version currently in use for other USAID funding instruments such as contracts, grants and cooperative agreements. The PAPA clause lacks the flow-down requirement for the anti-terrorism clause to be included in all subawards. USAID should consider updating the anti-terrorism clause in the PAPA template to the following March 2002 version currently in use for other types of instruments: “Executive Order on Terrorist Financing (March 2002) The Contractor/Recipient is reminded that U.S. Executive Orders and U.S. law prohibits transactions with, and the provision of resources and support to, individuals and organizations associated with terrorism. It is the responsibility of the contractor/recipient to ensure compliance with these Executive Orders and laws. This provision must be included in all subcontracts/sub-awards issued under this agreement.” MEASURING IMPACT – PAPA EVALUATION REPORT 78 ANNEXES FINANCIAL TRACKING TOOLS Evaluation Questions – Financial Tracking Tools   Are the tools that USFS International Program uses to manage the PAPA adequate (e.g. mechanisms to allow USFS to spend funds, and track budgets and results), or do they need improvement? Were funds used in accordance with budgets? Initial Observations for Financial Tracking Tools Analysis of PMFAS report and comparison to quarterly reports submitted revealed some small variances between the two reports that are most likely attributable to reporting lag time. Variances between financial information submitted to Missions (for activities under RDMA and the STEWARD program) are more significant in value and cannot necessarily be attributed to lag time. Follow-Up for Interviews Seek explanation of variances between various financial reports. ADS 306.3.2.10 Use of PAPA for Inherently Government Functions Discussion: The Action Memo section entitled “Rationale for Using a PAPA” states that “USFS will provide services that are particularly suitable and unique to their facilities and resources, are not competitive with private enterprise, and can be made available without unduly interfering with its domestic programs.” This comes from a different section of ADS 306. “306.3.13.5 The Facilities of a Federal Agency “Particularly Suitable” to Provide Technical Assistance To conclude that USAID may use the facilities and resources of another Federal agency because the agency is “particularly suitable” to provide technical assistance (see 306.3.1.1) requires a showing that the agency has demonstrable expertise of an exceptional nature. The standard of “particular suitability” requires more than a finding that the proposed Participating Agency would win a contract for the technical assistance if it were put up for a competitive award. The proposed Participating Agency must have a clear and substantial superiority over other sources, both private and public. The preponderance of resources available for application to the target program must be much broader in scope and be demonstrably more readily available from the Participating Agency than from other sources. Alternatively, the type of service requested could be such that it involves the development of a governmental regulatory function that can best be performed by the Federal agency charged with the performance of the same regulatory function within the United States Government. Certain Federal agencies might be considered particularly suitable to provide services: Department of Agriculture:      Soil conservation Tropical forestry Photogrammetric data analysis in support of crops, inventories, crop yields and crop substitution programs Agriculture extension Agriculture marketing [Examples from other agencies omitted.] The Participating Agency must be able to carry out the assistance without unduly interfering with its domestic programs. This would demonstrate that the Participating Agency has excess capacity that can be provided to USAID for purposes of foreign assistance.” 79 MEASURING IMPACT – PAPA EVALUATION REPORT ANNEXES Initial Findings on Use of the PAPA for “Inherently Governmental Functions” There is nothing in the Action Memo justification for the PAPA to indicate that the ‘inherently governmental functions” justification applies. Instead, the rationale for this PAPA is that the USFS will provide services that are particularly suitable and unique to their facilities. MEASURING IMPACT – PAPA EVALUATION REPORT 80 ANNEXES ANNEX VI: BRANDING COMPLIANCE TABLE Table 8: Brazil FEC DOCUMENT TITLE/TYPE OF DOCUMENT DOCUMENT FILE NAME(S) AUTHOR /CONTRIBUTORS BRANDING REQUIRED? USAID IDENTITY Y/N SIZE AND PLACE-MENT? REMARKS FEC Bulletins 01/2009, 03/2010-17/2012, and 18- 21 (Aug 2012-Jul 2013) FEC Boletim.pdf USFS and 4 local partners Y Y Y 2001 version of the USAID Identity was used for the first 6 issues thru July/August 2010; the correct 2004 version was used for Sept /Oct 2010 and thereafter Forest Enterprise Cluster Program, Annual Performance Plan Report Performance Plan and Report Final.docx USFS and 4 local partners N Y Y FEC Semi-annual Report October 1, 2010 – March 31, 2011 Final FEC Semi￾Annual Report_2011 USFS and 4 local partners N Y Y Forest Enterprise Cluster Annual Report, FY2011 USAID Annual Report FEC 2010- 2011.docx USFS and 4 local partners Y Y Y Forest Enterprise Cluster Annual Report, FY2009 Final USAID Annual Report 2009.docx USFS and 4 local partners Y Y Y 81 MEASURING IMPACT – PAPA EVALUATION REPORT ANNEXES Table 9: STEWARD DOCUMENT TITLE/TYPE OF DOCUMENT DOCUMENT FILE NAME(S) AUTHOR /CONTRIBUTORS BRANDING REQUIRED? USAID IDENTITY Y/N SIZE AND PLACE-MENT? REMARKS Programmatic Overview of Sustainable and Thriving Environments for West Africa Regional Development (STEWARD): Design Phase to Implementation Phase Jan 2008 - Sept 2012 STEWARD Overview.2007- 2012.DRAFT 4.June26.WITHO UT LINKS Stephanie Otis, USFS-IP N Y Y STEWARD Performance Management Plan PMP STEWARD 27Aug12 BBHC Final2-1 USFS-IP N Y incorrect The STEWARD program logo is larger and positioned more prominently than the USAID Identity CLIMATE CHANGE VULNERABILITY & ADAPTATION DESKTOP STUDY: STEWARD PRIORITY ZONES STEWARD_VA_FI NAL_FEB2013 USFS-IP Y N incorrect The USAID Identity is not present on the cover page of this report; USAID support is acknowledged in the text on page 1 STEWARD Fiscal Year 2013 Work Plan STEWARD FY13 Work Plan 01 10 2012 Final USFS-IP N Y incorrect The STEWARD program logo is larger and positioned more prominently than the USAID Identity STEWARD III Semi￾Annual Report October 1, 2011 to March 30, 2012 2011_Semi￾Annual Report Phase III Oct1 2011 to March 30 2012 April 30 2012.FINAL USFS-IP N Y Y STEWARD III Semi￾Annual Report April 1, 2011 to September 30, 2011 2011_Semi￾Annual Report Phase III April 1 to Sept 30 USFS-IP N Y Y MEASURING IMPACT – PAPA EVALUATION REPORT 82 ANNEXES DOCUMENT TITLE/TYPE OF DOCUMENT DOCUMENT FILE NAME(S) AUTHOR /CONTRIBUTORS BRANDING REQUIRED? USAID IDENTITY Y/N SIZE AND PLACE-MENT? REMARKS 2011_USAID STEWARD II Phase II Final Report October 2010 to February 2011 2011May6_Final_ Progress_Report_ STEWARD_to USAIDWA_FINAL USFS-IP N Y Y STEWARD website http://stewardprogr am.org/ STEWARD Y Y Incorrect Branded with the STEWARD logo at the top left of the homepage rather than the USAID Identity, which appears instead at the bottom of the page Programmatic Overview of Sustainable and Thriving Environments for West Africa Regional Development (STEWARD): Design Phase to Implementation Phase Jan 2008 - Sept 2012 STEWARD Overview.2007- 2012.DRAFT 4.June26.WITHO UT LINKS Stephanie Otis, USFS-IP N Y Y STEWARD Performance Management Plan PMP STEWARD 27Aug12 BBHC Final2-1 USFS-IP N Y incorrect The STEWARD program logo is larger and positioned more prominently than the USAID Identity CLIMATE CHANGE VULNERABILITY & ADAPTATION DESKTOP STUDY: STEWARD PRIORITY ZONES STEWARD_VA_FI NAL_FEB2013 USFS-IP Y N incorrect The USAID Identity is not present on the cover page of this report; USAID support is acknowledged in the text on page 1 STEWARD Fiscal Year 2013 Work Plan STEWARD FY13 Work Plan 01 10 2012 Final USFS-IP N Y incorrect The STEWARD program logo is larger and positioned more prominently than the 83 MEASURING IMPACT – PAPA EVALUATION REPORT ANNEXES DOCUMENT TITLE/TYPE OF DOCUMENT DOCUMENT FILE NAME(S) AUTHOR /CONTRIBUTORS BRANDING REQUIRED? USAID IDENTITY Y/N SIZE AND PLACE-MENT? REMARKS USAID Identity STEWARD III Semi￾Annual Report October 1, 2011 to March 30, 2012 2011_Semi￾Annual Report Phase III Oct1 2011 to March 30 2012 April 30 2012.FINAL USFS-IP N Y Y MEASURING IMPACT – PAPA EVALUATION REPORT 84 ANNEXES Table 10: RDMA DOCUMENT TITLE/TYPE OF DOCUMENT DOCUMENT FILE NAME(S) AUTHOR /CONTRIBUTORS BRANDING REQUIRED? USAID IDENTITY Y/N SIZE AND PLACE-MENT? REMARKS LEAF workshop agenda and reports for degradation workshops held in Laos, Cambodia, Vietnam and Bangkok (5 documents in total) LEAF_Degradatio n workshop agenda FINAL.pdf; Laos degradation final.docx; degradation report final Cambodia.docx; vietnam degradation report final v15 25jun2013.pdf; Bangkok 2013 Forest Degradation Workshop Proceedings GTR prepub proof2.pdf USFS Y Y OK Mekong Climate Change Curriculum Development Scoping Trip Report, October 2012 LEAF_USFS_Curr iculum_Developm ent_Scoping_Rep ort_20121111.pdf USFS/University of VA, Utah State University, University of San Francisco Y Y OK LEAF’s Regional Climate Change Curriculum Development Training Workshop Report, August 2013 USFS LEAF Currriculum Development Training Workshop Report – FINAL.pdf USFS/University of VA, Utah State University, University of San Francisco, and LEAF Y Y OK Enhancing Sustainable Forest Management: Collaborative Activities Between the Halperin_April09_ FINAL REPORT.doc James Halperin, USFS Y Y OK 85 MEASURING IMPACT – PAPA EVALUATION REPORT ANNEXES DOCUMENT TITLE/TYPE OF DOCUMENT DOCUMENT FILE NAME(S) AUTHOR /CONTRIBUTORS BRANDING REQUIRED? USAID IDENTITY Y/N SIZE AND PLACE-MENT? REMARKS United States Forest Service and the Responsible Asia Forestry and Trade Program/Mission report, 4 November 2008 – 31 March 2009 Linking Tourism and Payment for Ecosystem Services in Bidoup Nui Ba National Park Lam Dong Province, Vietnam, report PES_ecotourism report_Final English.pdf Dr. Trista M. Patterson & Jennifer Burns, MLA, USFS Y Y OK Payments for forest environmental services in Vietnam From policy to practice/ study report PES review publication 2013.pdf Pham Thu Thuy, CIFOR; Karen Bennett, USFS; Vu Tan Phuong, VAFS; Jake Brunner, IUCN; Le Ngoc Dung, CIFOR; Nguyen Dinh Tien, Hanoi Agriculture University Y N N/A USAID support is acknowledged in the text Report of technical assistance visit to Vietnam: Lâm Đồng Province, Vietnam, PES Pilot Project, September 10–26, 2009 TechAssistance_R eport fr USFS scientist.pdf Michael J. Furniss, USFS Y N N/A MEASURING IMPACT – PAPA EVALUATION REPORT 86 ANNEXES DOCUMENT TITLE/TYPE OF DOCUMENT DOCUMENT FILE NAME(S) AUTHOR /CONTRIBUTORS BRANDING REQUIRED? USAID IDENTITY Y/N SIZE AND PLACE-MENT? REMARKS PAYMENT FOR ECOSYSTEM SERVICES In the Mekong Region A REPORT ON TRAINING, December 2007 Report on Mekong PES Training Activities.pdf USFS Y N N/A USAID support is acknowledged in the text Guide to identifying ramin wood/program material Raminsmallfilesize .pdf USFS Y Y Y Regional Training Workshop on Ramin Wood Identification/description of planned activity, SoW RegionalRaminWo rkshop_Logos.doc USFS N Y Y This type of document is not required to be marked with the USAID Identity 87 MEASURING IMPACT – PAPA EVALUATION REPORT ANNEXES Table 11: Uganda STAR II DOCUMENT TITLE/TYPE OF DOCUMENT DOCUMENT FILE NAME(S) AUTHOR /CONTRIBUTORS BRANDING REQUIRED? USAID IDENTITY Y/N SIZE AND PLACE-MENT? REMARKS Uganda Sustainable Tourism in the Albertine Rift (USAID-STAR II)/ Final Report August 2011-August 2012 USAID-STAR II Final Report.pdf Solimar International N Y OK Uganda Sustainable Tourism in the Albertine Rift (USAID-STAR II), Quarterly Report, October – December 2011 USAID-STAR II Qtrly Report Oct￾Dec 2011_FINAL.pdf Solimar International N Y OK Uganda Sustainable Tourism in the Albertine Rift (USAID-STAR II), Quarterly Report, January – March 2012 USAID-STAR II Qtrly Report Jan￾March 2012_FINAL.pdf Solimar International N Y OK Press Release RE: New Trail and Visitor Information Center in the Rwenzori Mountains August 30th 2012 Rwenzori Launch Announcement.pd f USFS / Ecotrust, Uganda Wildlife Authority, and Geolodges Y Y OK USFS shores-up USAID STAR technical assistance to UWA in three national Parks/”In the News” item USAID STAR _ In News Feb 15th 2012.pdf USFS & STAR partners(?) Y Y OK Final Report on the management of selected Uganda trails for STAR, January 2012 STAR trail report FINAL.pdf John Neary, USFS Y N N/A USAID support is acknowledged in the text Mountain Rescue Training Course Report, Rwenzori Mountains National Park, Uganda February 29 to March 2, Mountain Rescue Course Report Rwenzori 2012.pdf Steve Brigham, USFS Y N N/A 4-page training report w/color photos MEASURING IMPACT – PAPA EVALUATION REPORT 88 ANNEXES DOCUMENT TITLE/TYPE OF DOCUMENT DOCUMENT FILE NAME(S) AUTHOR /CONTRIBUTORS BRANDING REQUIRED? USAID IDENTITY Y/N SIZE AND PLACE-MENT? REMARKS 2012 Conservation Tourism: Investment Management & Evaluation Framework, December 2012/Final report to Uganda STAR II Uganda STAR II - Conservation Tourism - Investment Management & Evaluation Framework - Introduction and Annexes 1-3 Final Rept.docx African Wildlife Foundation Y N N/A Only the AWF logo appears on the cover page Buhoma Visitor Center Bwindi Impenetrable National Park Conceptual Design - Nov. 2010 report UWA Buhoma Report sm 110520.pdf USFS Y Partial OK The USAID tagline “USAID From the American People” is on the document; the USAID logo is missing USAID STAR Launch Rwenzori Mountain National Park Visitor Center, and Muhoma Trail/ “In the News” item in the news _ USAID STAR Rwenzori Visitor Center and trail launch.pdf USFS and STAR partners(?) Y Y OK A photo on page 2 shows a plaque with the USAID Identity on a building constructed with partial funding from USAID. Pearls of Uganda website http://www.pearlso fuganda.org/ Uganda Community Tourism Association Y Y OK Birding Uganda website http://birding￾uganda.com/ Uganda Tourism Board Y Y OK Uganda Wildlife Authority website http://ugandawildlif e.org/ Uganda Wildlife Authority N N N/A UWA is a semi-autonomous government agency – USAID branding was not required 89 MEASURING IMPACT – PAPA EVALUATION REPORT ANNEXES ANNEX VII: DATABASE FIELDS Fields in the database include: 1. General Information by Buy-In:        Title Region Country or sub-region USFS-IP contact person Buy-in Source PAPA amendment number Buy-in Amount ($) 2 . Funding Themes:     Biodiversity Climate Change Disaster Other 3. Technical Areas: 1. Biodiversity/protected area management (biodiversity conservation, habitat management, wildlife assessments, illegal wildlife trafficking ) 2. Biodiversity/protected area use (tourism, trails, visitor centers, interpretation, conservation education) 3. Climate change/carbon monitoring (REDD, carbon-focused inventory, remote sensing and GIS) 4. Climate change/other (LEDS, adaptation, GCC analyses) 5. Disaster management (non-fire ICS) 6. Fire management (fire ICS, ecology, prevention, response) 7. Illegal logging/Lacey Act (wood ID, forensics, genetics, log tracking) 8. Natural resources management practices (RIL, invasive species and pests, non-carbon forest inventories, forest monitoring) 9. Natural resources management policies (land-use planning; environmental, economic assessments; institutional capacity building) 10. Production forest use (community forestry, agroforestry, tree-based biofuels, smallholder wood production systems) 11. USAID programming support (Reg. 118/119 assessments, ETOAs, IEEs, institutional assessments) 12. Watershed management (ecosystem services) 4. Project/Activity Title(s) if Any (Unique for each Technical Area) 5. Allocation of Funds (Reported as $ portion of total buy-in)    Direct implementation (including METI staff and consultants) Grants or transfers to other USG agencies (FWS, USGS) Grants or METI wire transfers to NGOs, universities, multilaterals MEASURING IMPACT – PAPA EVALUATION REPORT 90 ANNEXES 6. Delivery Mode: What Is the Primary Means of Delivering Services? 1. Field-based technical assistance within target country/region 2. Formal and informal training programs within target country/region 3. Policy and economic analysis 4. Environmental and program assessments, program design, program management, and monitoring and evaluations 5. Study tours, workshop and seminars in US or third country 6. Applied research 7. Direct Beneficiary: who are the primary recipients of the assistance? 1. host country government agencies 2. international NGOs 3. multilateral organizations (CIFOR, FAO, WB, EU) 4. local NGOs 5. communities 6. international contractors 7. USAID Mission/Bureau 8. private sector 9. USG agencies 10. academic institution