APEC TATF MID-TERM EVALUATION January 2013 This publication was produced for review by the United States Agency for International Development. It was prepared by Michael Farbman and Abdul Wahab under purchase order numbers AID-486-0-12-00147 and AID-486-0-12-00160 i APEC U.S. TATF Mid-Term Contractor Evaluation Volume 1 Prepared for: The United States Agency for International Development Regional Development Mission Asia January 2013 Prepared by: Michael Farbman and Hyatt Abdul Wahab Under Purchase Orders AID-486-O-12-00147 AID-486-O-12-00160 The views expressed in this document are those of the authors, and do not necessarily represent the view-point of the U.S. Agency for International Development or of any other body or person. ii APEC U.S. TATF Mid-Term Contractor Evaluation Volume 1 – The Report: Findings and Recommendations ACRONYM LIST ............................................................................................................... iv EXECUTIVE SUMMARY .................................................................................................... vi 1. BACKGROUND/INTRODUCTION: WHAT IS APEC? ............................................... 1 A. FOUNDING OF APEC ............................................................................................. 1 B. GOALS AND OBJECTIVES OF APEC ..................................................................... 1 C. PROGRESS TOWARDS BOGOR GOALS .............................................................. 2 D. HOW APEC IS ORGANIZED TO CARRY OUT ITS OBJECTIVES ...................... 4 E. U.S. GOVERNMENT SUPPORT TO ASIA-PACIFIC REGIONAL INSTITUTIONS 5 II. FILLING AN APEC NEED WITH THE TATF ............................................................... 7 A. RATIONALE: THE U.S. GOVERNMENT’S OBJECTIVE IN SUPPORTING APEC WITH A TATF ................................................................................................................ 7 B. ORIGINS OF THE TECHNICAL ASSISTANCE AND TRAINING FACILITY ........ 7 C. APPROVAL AND AUTHORIZATION BY USAID FOR ESTABLISHING THE APEC TATF .................................................................................................................. 10 D. CONTRACT AWARD: RDMA AS APEC’s TATF IMPLEMENTATION OVERSIGHT AGENCY ................................................................................................ 10 E. HOW TATF OPERATES ......................................................................................... 11 F. PROGRAM CONTENT .......................................................................................... 13 G. HOW THE U.S. GOVERNMENT POLICY ROLES IS CARRIED OUT AND COORDINATED ......................................................................................................... 15 III. METHODOLOGY, EVALUATION QUESTIONS, AND SURVEY FINDINGS ......... 16 A. EVALUATION METHODOLOGY SUMMARY ..................................................... 16 B. MID-TERM EVALUATION PURPOSE, OBJECTIVES, AND HYPOTHESIS .......... 16 C. SURVEY RESULTS .................................................................................................. 18 IV. CONCLUSIONS AND LESSONS LEARNED ............................................................ 20 QUESTION 1(A): HOW HAS EACH APEC TATF PROGRAM CHANGED APEC AS AN INSTITUTION? ...................................................................................................... 21 QUESTION 1 (B): HOW HAS THE IMPLEMENTING PARTNER CHANGED APEC AS AN INSTUTUTION? .............................................................................................. 25 QUESTION 2: TO WHAT EXTENT DID THE SERVICES PROVIDED BY THE TATF FURTHER INTERNATIONAL TRADE AND INVESTMENT AND ASSIST REGIONAL INTEGRATION? ...................................................................................... 28 iii QUESTION 3: WHAT ARE THE MOST IMPORTANT AREAS OF FOCUS FOR U.S. ASSISTANCE IN THE FUTURE TO ACHIEVE THE GREATEST IMPACT GIVEN LIMITED BUDGETS AND APEC CAPACITY? ............................................................ 34 V. RECOMMENDATIONS .............................................................................................. 36 A. RECOMMENDATIONS FOR USAID OVERSIGHT .............................................. 36 B. RECOMMENDATIONS FOR POSSIBLE FUTURE U.S. FOREIGN ASSISTANCE TO APEC ...................................................................................................................... 40 Appendices: Refer to Separate Volume 2 iv ACRONYM LIST ABAC APEC Business Advisory Council ADB Asian Development Bank ADS Automated Directive System APEC Asia-Pacific Economic Cooperation ASEAN Association of Southeast Asian Nations AusAID Australian Agency for International Development BMC Budget and Management Committee CIEM Central Institute for Economic Management (Vietnam) COP Chief of Party COR Contracting Officer's Representative CTI APEC Committee on Trade and Investment DAI Development Alternatives, Inc. DNG DAI/Nathan Group (Development Alternatives, Inc. and Nathan Associates) DOE Department of Energy EAP/EP Bureau of East Asian and Pacific Affairs, Office of Economic Policy ECOTECH Economic and Technical Cooperation ED Executive Director of the APEC Secretariat EEC European Economic Community EGS Environmental Goods and Services EODB Ease of Doing Business EOP End-of-project ESF Economic Support Funds FAS Foreign Agricultural Service (USDA) FTA Free Trade Agreement FDI Foreign Direct Investment GOVN Government of Vietnam GNI Gross National Income GDO General Development Office HDI Human Development Index ICT Information and Communications Technology IMF International Monetary Fund IAP Individual Action Plans for meeting Bogor Goals IPR Intellectual Property Rights IQC Indefinite Quantity Contract IT Information Technology LOP Life of Project M&E Monitoring and Evaluation MFN Most Favored Nation NGO Non-Governmental Organization PD Program Director PE Program Executive PIO Public International Organization PMP Project Monitoring Plan PMU Project Management Unit PNG Papua New Guinea POC Point of Contact PPL Bureau for Policy, Planning and Learning v PPP Purchasing Power Parity PSU Policy Support Unit PVO Private Voluntary Organization RDMA USAID’s Regional Development Mission for Asia REI Regional Economic Integration RTA Regional Trade Agreement SCE SOM Steering Committee on ECOTECH STAR Support for Trade Acceleration in Vietnam TATF Technical Assistance and Training Facility of APEC Secretariat SCE Steering Committee on Economic and Technical Cooperation SEADI Support for Economic Analysis Development in Indonesia SOM Senior Officials Meetings SOW Scope of Work STAR Support for Trade Acceleration in Vietnam TA Technical Assistance TATF Technical Assistance and Training Facility TILF Trade and Investment Liberalization Fund TO Task Order TPP Trans-Pacific Partnership USAID United States Agency for International Development USD United States Dollar USDA U.S. Department of Agriculture USG United States Government USTR The Office of the U.S. Trade Representative VNCI Vietnam National Competitiveness Index WB World Bank WTO World Trade Organization vi EXECUTIVE SUMMARY This summary presents the findings, conclusions and recommendations of a Mid-Term Evaluation of the U.S. Technical Assistance and Training Facility (TATF) efforts to strengthen the capacity of the Asia-Pacific Economic Cooperation (APEC), including the APEC Secretariat and policy makers in APEC member economies, for accelerating progress toward Regional Economic Integration and meeting APEC’s“Bogor Goals.” The Asia-Pacific region is critical to America's economic growth and is a top priority of the Obama Administration.1 APEC’s 21 member economies of nearly three billion citizens represent a mix of some of the strongest and the weakest economies in the world. APEC aims to create by 2020 the world’s largest free and open trade and investment area. To this end, APEC’s developed and developing member economies pursue a broad agenda of regulatory, structural, and economic reforms that are recognized as essential for advancing regional economic integration (REI). Unlike other economic trading blocs, APEC members’ commitments to carrying out the reforms are voluntary and are legally non-binding. The objective is to provide outcomes with high aspirations while, at the same time, allowing flexibility in the pace of implementation that recognizes the diversity in members’ levels of development. A comprehensive report on progress of 13 APEC developed and developing economies, based on publicly available data concluded in 2010, suggested that the participating members have indeed been reducing barriers to trade and investment, and in aggregate, APEC is making material progress in the direction of regional economic integration.2 To spur progress in reaching the Bogor Goals, in 2008 the U.S. Department of State requested that USAID manage a technical assistance initiative that would be embedded within the APEC Secretariat and structured as a platform to provide a wide array of services in response to the internal needs of the Secretariat as well as the 21 member economies. In September 2008, USAID/RDMA created a project to establish the TATF “in furtherance of U.S. foreign policy goals of greater Regional Economic Integration and to strengthen APEC as a regional institution.” The APEC TATF would work in three technical areas: 1) trade and investment liberalization; 2) business facilitation; and 3) economic and technical cooperation. The project is scheduled to be completed in December 2013. In order to evaluate the contractor’s performance and effectiveness of the TATF, an external participatory Mid-Term Evaluation of the project was completed in September – November 2012. The evaluation aimed to address the following three questions: 1) How has (a) each APEC TATF program and (b) the implementing partner changed APEC as an institution? 2) To what extent did the services provided by the TATF further international trade and investment and assist regional integration? 3) What are the most important areas of focus for U.S. assistance in the future to achieve the greatest impact given limited budgets and APEC capacity? FINDINGS AND CONCLUSION OF THE EVALUATION The evaluation findings and conclusions were based on data gathered from document reviews, observations, stakeholder feedback, surveys, rapid appraisals, site visits with TATF beneficiaries, 1 Opening Remarks by President Obama at APEC Session One, November 12, 2011. http://www.whitehouse.gov/the-press￾office/2011/11/13/opening-remarks-president-obama-apec-session-one 2 APEC’s Bogor Goals Progress Report – Highlights of Achievements and Areas for Improvement, APEC Policy Support Unit, August 2012. vii partners in the APEC region, and APEC U.S. Government representatives. In brief, the evidence is overwhelmingly in support of TATF’s overall quality performance and prodigious contributions to helping APEC make progress on reaching its Bogor Goals. Such performance is linked to strong professional interaction and shared interests between a well-conceived and intellectually nimble TATF team and a highly effective contractor oversight supported by well￾qualified, enthusiastic and energetic human resources. TATF has been instrumental in helping APEC to accelerate the pace of REI as a strategic approach to reducing the development gap among the member economies. While other options exist for enhancing APEC’s institutional capabilities and for satisfying the cooperation needs of U.S. Government stakeholders, the U.S. Government’s support for APEC through a TATF-like structure has not only resulted in many concrete contributions to APEC operations, but has earned the U.S. Government respect and credibility as a partner and counterpart. Some examples of performance with respect to the three overarching questions follow: Question 1 (a). How has each APEC TATF program changed APEC as an institution? The TATF services, projects and programs have enhanced the stature of APEC as a premier institution for promoting regional economic integration. In order to progress toward the Bogor Goals, APEC member economies must act individually as well as in concert to influence policy, regulatory, and economic reforms that relate to accelerating the pace of trade liberalization, facilitating business investments and capital flows, and economic cooperation. Diverse TATF facilitation efforts are resulting in:  Upgrading of professional, administrative and operational capabilities of the APEC Secretariat.  Human capital formation at the level of APEC member economies that generates commitment and skillfulness for positive and appropriately paced and scaled development outcomes.  Effective outreach and communications throughout the Secretariat and 21APEC member economies.  The integrated in-house consultancy structure of the APEC TATF has evolved into an integral part of the Secretariat. It is widely respected within APEC and is contributing to the achievement of United States economic and national interests for spurring regulatory and trade reforms that enable accelerated economic integration and broad￾based growth in the Asia-Pacific region.  Overall improvements in success rates in APEC’s project approval process – especially for developing member economies – are attributed to TATF training of project proponents and of TATF-designed proposal approval process improvements. Question 1 (b). How has the implementing partner changed APEC as an institution? Because of the implementing partner’s demonstrated technical skills and experience in the Asia￾Pacific region, and its ability to strategically plan and smoothly implement international trade and business development, and economic and technical cooperation activities, APEC as an institution is now better equipped with a cadre of trained and motivated technical and administrative staff, and state-of-the-art ICT equipment. As evidence: viii  APEC more effectively and efficiently carries out its day-to-day functions and operations within the Secretariat, and in coordination withits 21 member economies.  The quality of the contractor’s personnel and the contractor’s worldwide network of experts and consultants, have combined to showcase APEC as a strong change agent for REI, while distinguishing the United States as a champion supporter for economic growth and stability in the Asia-Pacific region. For example, during the project period, the contractor deployed hundreds of recognized technical experts and trainers to present state-of-the-art knowledge and carefully tailored data to approximately 2,000 participants at more than 40 U.S. APEC TATF-led events covering all aspects of REI.  Representatives of APEC economies applied their newly acquired knowledge and skills in developing medium-term strategic plans and quality projects to inform national-level decisions for enabling complex reform measures and accrual of economic and social benefits. These outcomes are contributing to strengthening APEC’s institutional capacity and relationships both within the Secretariat and beyond. Question 2: To what extent did the services provided by the TATF further international trade and investment and assist regional integration? The TATF held dozens of workshops and training events with a primary focus on improving the enabling conditions for international trade and investment, and accelerating the rate of Asia￾Pacific regional integration. Some of these efforts were ongoing prior to the start-up of TATF assistance. APEC’s 2010 report on measuring progress of five developed and eight developing economies toward the Bogor Goals concluded that the economies “have indeed been further reducing barriers to trade and investment since 1994. So too has APEC as a whole.” The data also indicate that progress has been made in achieving higher level goals of APEC in terms of sustainable growth and development. Progress is significant across an array of widely adopted economic, trade, investment and social indicators, and APEC economies have outperformed the rest of the world in many aspects. Also, the development gap is narrowing among APEC members, as part of APEC’s ultimate objective of broad-based growth and strengthening the Asia-Pacific community.  While the APEC 2010 Report makes no claim that the Bogor Goals were fully responsible for the outcomes achieved, it argues that the Bogor Goals inspired and contributed to the achievements and outcomes.  Similarly, the TATF evaluation survey data showed that TATF’s services have led to improving APEC’s institutional performance, both at sectoral and at whole-of￾government levels. Also, development outcomes influenced by the TATF platform have been noticeable. Notably, other APEC donor partners who commented on the timeliness, appropriateness, and utility of TATF assistance further support this claim. Hence, it is clear that the TATF too is contributing on many fronts to advancing regional economic integration and progress in reaching the Bogor Goals. Question 3: What are the most important areas of focus for U.S. assistance in the future to achieve the greatest impact given limited budgets and APEC capacity? The clearest picture of this evaluation emerges from the uniform comments and feedback by stakeholders that the TATF platform and structure, strategically embedded in the Secretariat, has helped to accelerate the pace of reforms in APEC for reaching Bogor Goals, and has enhanced the efficiency and effectiveness of the APEC institution. Also clear was that the TATF ix has earned high marks for the United States’ demonstrated commitment to REI. Given the critical importance of the Asia-Pacific region to United States economic growth, and the remaining challenges faced by the developing economies in improving their institutional ability to implement their voluntary pledges to carry out select reforms for reaching the Bogor Goals, a follow-on assistance project merits serious consideration. The possible new support to APEC should have a similar structure to the TATF and aim to build on past successes and lessons learned. The main objectives would be to continue to build APEC institutional capability for developing implementation protocols, and to promote policy reforms and regulatory frameworks leading to further acceleration of free and open trade and investments in the region. In all, 14 recommendations are presented in the final chapter of the Report. Summaries are provided below: 1. USAID and the Department of State should streamline and formalize the work planning process. 2. USAID needs better representation at the Washington-based inter-agency table. 3. RDMA should take immediate steps to ensure quality control in TATF through more regular and more intensive oversight. 4. TATF should inform Embassies and USAID Missions of activities relevant to their project, reporting or diplomatic efforts. 5. TATF, RDMA and State EAP/EP should install a real-time calendar and dedicated messaging system to ensure up-to-date information sharing. 6. The TATF concept as an embedded technical assistance and training platform should be a starting point for consideration of a follow-on project. 7. The profile of monitoring and evaluation should be given higher priority in any follow-on project. 8. A higher priority should be given in any follow-on project to down-stream implementation and institutionalization of reforms in member economies. 9. Focus should be on incorporating ways to initiate and support innovative high-pay￾out policies and practices that enhance REI in a developmentally sound way. 10. U.S. Government resources in APEC should be better leveraged by establishing more conscious and systematic APEC donor coordination. 11. Consider shifting project balance from capacity-building activities toward consolidating APEC efforts that directly support REI and reduce the development gap. 12. Improve technical cooperation between APEC and ASEAN. 13. Balance sustainability considerations of APEC support while still considering the benefits to United States national interests. x 14. More flexible procedures should be devised in order to expeditiously accommodate short lead-time contractor work-load changes and skill-mix requirement. 1 1. BACKGROUND/INTRODUCTION: WHAT IS APEC? A. FOUNDING OF APEC In 1989, the growing interdependence of Asia-Pacific economies, the advent of regional trade blocs in other parts of the world, and the slow pace of progress on structural and regulatory reforms within their own countries provided the impetus for the leadership of 12 Pacific Rim countries including the United States to launch the Asia-Pacific Economic Cooperation (APEC), in Canberra, Australia.3 The founding APEC member economies affirmed that in order to maintain a trajectory of steady outward-oriented economic growth and the imperative to establish new markets, inefficient and protected sectors of their respective economies had to undergo major economic and structural reforms. B. GOALS AND OBJECTIVES OF APEC The first APEC meeting, which began as an informal dialogue between senior officials and ministers, was held in Australia in 1989. The United States hosted the first annual APEC Economic Leaders’ Meeting in 1993. Their vision was for an Asia-Pacific region that harnesses the energy of its diverse economies, strengthens cooperation, and promotes prosperity, in which the spirit of openness, voluntarism, and partnership deepens, and dynamic growth continues, contributing to an expanding world economy and supporting an open international trading system. They envisioned continued reduction of trade and investment barriers so trade could expand within the region and with the world; and goods, services, capital, and investment flow freely among APEC economies. At the APEC Leaders’ Meeting in 1994 in Bogor, Indonesia, APEC leaders enunciated a vision of creating the world’s largest area of free and open trade and investment by 2020. Under this declaration, developed member economies would achieve free trade by 2010 – recently revised to 2015 – and developing economies would follow in 2020.4 By then, APEC hoped to have achieved its stated goal of creating the world’s largest area of free trade and investment. These targets became known as the “Bogor Goals,” a manifestation of APEC’s shared belief that free and open trade and investment is essential to realize the growth potential of the region and enhance economic and social outcomes for all APEC 21 member economies. APEC’s goal of promoting economic growth, fostering and strengthening trade, and improving the living standards of the less developed economies in the Asia-Pacific region is progressing through the voluntary commitment of its member economies to work towards raising living standards and education levels through sustainable economic growth, and to foster a sense of 3 The original 12 Asia-Pacific members were: Australia, Brunei Darussalam, Canada, Indonesia, Japan, Malaysia, New Zealand, Philippines, Singapore, South Korea, Thailand and the United States. Currently, there are 21 APEC members. The nine additional economies are: Chile, China, Hong Kong China, Chinese Taipei, Mexico, Peru, Papua New Guinea, Russia and Vietnam. 4 APEC’s current and historic use of words such as “developed” and “developing” are organization-specific, and do not conform to current standard classifications of economies such as those used by the World Bank, UNDP or IMF. Originating in 1994 in the context of the “Bogor Goals,” APEC initially used the word “industrialized” to differentiate its then most developed member economies: Australia, Canada, Japan, New Zealand and the United States. The remaining economies were and are referred to as “developing members.” Clearly, in 2013, as measured by per capita gross national income (GNI) or other standard accepted development measures, only member economies Papua New Guinea (PNG), Indonesia, Philippines and Vietnam are classified as “low income” or “lower middle income” – read “developing economies” – with per capita GNI less than US$3975, the World Bank’s upper limit for “developing”. Further, in compliance with the U.S. Government’s determination related to “need” and to other policy considerations, operationally for the TATF, only nine member economies currently are eligible to receive certain forms of assistance under the project. Those economies are: Chile, Indonesia, Malaysia, Mexico, Philippines, Peru, PNG, Thailand, and Vietnam. Current U.S. policy does not allow participants from China and Russia to receive support from TATF. By most standards and measures, among the APEC economies, the above-mentioned nine economies may be referred to credibly as APEC’s developing member economies. For further reference, see Asia-Pacific Economic Cooperation, "Progress towards the APEC Bogor Goals - Perspectives of the APEC Policy Support Unit,” APEC Policy Support Unit, November 2010. 2 community and an appreciation of shared interests among Asia-Pacific countries. From its formation in 1989 as a loose consultative forum with no large bureaucracy to support it, through 1998, APEC’s membership grew from 12 to 21 economies. The nine additional economies that gained entry into APEC are Chile, the People’s Republic of China, Hong Kong, Chinese Taipei, Mexico, Papua New Guinea, Peru, Russia, and Vietnam. C. PROGRESS TOWARDS BOGOR GOALS According to estimates of GNI per capita and their respective levels of human development, APEC members represent a mix of some of the strongest and the weakest economies in the world. The three APEC economies with the highest estimated GNI per capita in 2010 were Singapore, United States, and Canada at $55,790, $47,310, and $38,370 respectively. And, the three APEC economies characterized by the lowest GNI per capita were Papua New Guinea, Vietnam, and Philippines with incomes of $2,420, $3,070, and $3,980.5 Correspondingly, the levels of human development as estimated by the human development index (HDI) in 2011 were very high for the three top income members – averaging 0.90; medium for Philippines and Vietnam – averaging 0.62; and low for Papua New Guinea at 0.47.6As growth continues in the Asia-Pacific region, income disparities and the gap in levels of human development between and among APEC’s developed and developing economies are decreasing, albeit at varying rates. APEC seeks to advance REI efforts by means of implementation of a regime of free and open trade and investment liberalization policies; structural, regulatory, and economic reforms; greater economic and technical cooperation; and promoting policies that facilitate foreign direct investment (FDI). Driven by this imperative, and with support from its Secretariat, which manages APEC’s day-to￾day administration, APEC pursues a program aimed at: human capital formation; the utilization of state-of-the-art information, communications and technology systems; economic and trade policy analysis and diagnostics for informing structural and regulatory reform policies; the removal of impediments to free and open trade of goods and services and movement of capital; expanding economic opportunities for women entrepreneurship; small and medium enterprise development; regional business competitiveness; mitigating global climate change; improving corporate governance; and promoting greater transparency and wider participation in the trading system and rule-making processes. APEC commitments are made in good faith but are legally non-binding. Of APEC’s 21 member economies, eleven are also members of the Trans-Pacific Partnership (TPP), a regional free trade agreement currently under negotiation, and seven are members of the Association of South East Asian Nations (ASEAN).7 ASEAN is a geo-political and economic institution of ten countries and its charter is broader than that of APEC. Agreements within ASEAN under its three pillars – economic, political security, and socio-cultural – are legally binding. The demonstrated appreciation and demand by APEC and ASEAN member nations for U.S. Government-sponsored technical assistance and training, coupled with a renewed U.S. Government emphasis on deepening trade and diplomatic relationships between the United States and the Asia-Pacific countries have created new opportunities for furthering an enabling environment conducive to sound economic policies and for bolstering regional economic 5 World Development Indicators 2012, available at: http://data.worldbank.org/data-catalog/world-development-indicators. 6 Human Development Report 2011 “Human development statistical annex,” United Nations Development Program, pp 127-130. 7 Currently, the eleven TPP negotiating parties are: Australia, Brunei Darussalam, Canada, Chile, Malaysia, Mexico, New Zealand, Peru, Singapore, the United States, and Vietnam. 3 integration, free and open trade, competitiveness and greater investment flows.8These opportunities have led to measurable outcomes reflecting encouraging trends in progress towards reaching Bogor Goals, including in the specific areas identified by member economies summarized as follows:9  Tariffs continue their downward trend and have been eliminated for many tariff lines in some economies. In 2010, APEC’s most favored nation (MFN) average tariff was equal to 5.8 percent. However, average tariffs in agriculture remain higher, beyond 50 percent in some economies, in comparison with other sectors (11.9 percent to 4.9 percent).  Economies have reported progress concerning the elimination or reduction of certain non-tariff measures. However, some restrictions still remain and new measures restricting or potentially restricting trade continue to be implemented.  Member economies have continued to make progress in services liberalization and facilitation in many sectors. Nevertheless, there is still room for improvement as some restrictions in terms of market access, national treatment, and local presence are maintained.  Economies are improving investment conditions conducive to FDI. However, sectoral restrictions to foreign investment are common in areas considered of domestic strategic interest.  Efforts to align to international standards are hindered due to concerns and weaknesses regarding systems to be established for meeting certain sanitary and phytosanitary requirements and technical regulations.  Economies have made substantial progress to facilitate trade by streamlining Customs procedures such as the initiation of a single window.  With respect to increased transparency and market access in government procurement, while there has been progress, concerns remain especially in terms of preferences accorded local suppliers and restrictions on the origin of the goods and services.  Progress continues with enforcing and strengthening intellectual property rights (IPR) and competition policy systems through modifications to existing laws, and the adherence to multilateral treaties.  Positive trends continue across APEC economies for regulatory reforms to reduce market distortions, increase efficiency, and reduce the cost of doing business.  Implementations of new measures concerning government procurement are enhancing transparency, competition and efficiency as more relevant information concerning laws, regulations, guidelines, and administrative procedures are becoming easily available via internet. One example, the electronic submission of documents, is reducing time and cost to access government services.  The network of regional and free trade agreements (RTAs and FTAs) is expanding for all 21APEC economies. A review10 of individual member’s self-assessment reports and Action Plans indicated that all four of APEC’s developing economies in the lowest income tier – Papua New Guinea, Vietnam, Philippines, and Indonesia – are making steady progress, albeit at varying rates, towards voluntarily reaching Bogor Goals. Similarly, those APEC economies with GNI per capita significantly higher than the World Bank’s threshold for developing economies, such as Chile, China, Malaysia, Mexico, Peru and Thailand are seriously pursuing REI-based reforms. And, the 8 http://www.whitehouse.gov/the-press-office/2011/11/13/opening-remarks-president-obama-apec-session-one 9 APEC’s Bogor Goals Progress Report – Highlights of Achievements and Areas for Improvement, APEC Policy Support Unit, August 2012. 10APEC’s Bogor Goals Progress Report – Highlights of Achievements and Areas for Improvement, APEC Policy Support Unit, August 2012. 4 highest GNI per capita APEC economies continue to implement new measures that ensure close alignment with APEC as well as adherence to multilateral treaties. In some instances accelerated progress on REI-concerned issues converged with individual economies’ strategic economic development priorities. And, in other instances, less than robust progress was influenced by the particular economy’s institutional weaknesses to implement needed reforms, or to effectively enforce policies that would further their attainment of Bogor goals. D. HOW APEC IS ORGANIZED TO CARRY OUT ITS OBJECTIVES Structure of the Secretariat The APEC Secretariat is the core support mechanism for all APEC operations. It provides coordination, technical and advisory support, as well as information management, communications and public outreach services. It performs a central project management role, assisting APEC Member Economies and APEC fora with overseeing more than 250 APEC￾funded projects. APEC's annual budget is administered by the APEC Secretariat. Leadership of the Secretariat is provided by the Executive Director (ED). The ED is always a citizen of an APEC member economy, and is appointed by consensus of the members for periods of 3-4 years. The incumbent is ultimately responsible and accountable for all facets of the Secretariat’s operations and core staff of approximately 62 individuals. He/she is an experienced professional who, previously to his/her appointment as ED, has served nationally at senior-most levels across several functional areas. He/she is well versed and proficient in key aspects of regionalism, transnational, and global issues including trade, finance, governance, security, conflict prevention, and political stability. As ED, the incumbent guides development of the APEC Secretariat’s medium-term program and management strategies for ensuring improved efficiencies and effectiveness of the Secretariat’s functions and the delivery of myriad tailored services to APEC’s diverse stakeholders. The ED has played an active role in TATF work plan development and downstream implementation of activities that assist member economies in reaching Bogor Goals. Such TATF engagement includes assisting in staff professionalization, IT upgrades, data systems management, staff performance appraisal and review policy, strategic planning of the technical working groups’ programs, and counseling on facing new challenges concerning regional and global trade competitiveness, economic resilience, environment and green growth, and other key human capital formation efforts for achieving APEC institutional excellence. The ED represents the Secretariat as appropriate, at Senior Officials Meetings (SOMs), meetings of APEC’s four committees, and at the annual Leaders Meeting. The Program Directors (PDs), as a community, are responsible for supporting the APEC Committees, technical working groups and other technical fora that take the lead in organizing the myriad workshops, seminars, case studies, diagnostics, and training initiatives of APEC, which TATF is called upon to support. The PDs are all secondees of their respective member economy governments. There is expected to be one PD for each member economy, though at any given moment there may be as many as two or three (in the case of a small number of developed member economies such as Australia, Japan and Korea), or possibly none, which may occur due to the high perceived cost to developing member economies of detailing an officer, or simply due to turnover and transition. In addition, there are other professional and administrative staff members that fulfill specialist and support functions. Many of these individuals also interact with TATF staff and activities. 5 Compared to ASEAN’s, APEC’s 62-person Secretariat staff is relatively small, and as noted above, 21 are secondees whose salaries are paid by their member economies. APEC’s core costs are sustained by a relatively small annual operating budget that has not increased in real terms in a decade. Also, the institution has significant institutional weaknesses including a lack of mandate to technically advocate strongly for reforms that would increase its effectiveness and accelerate the development of its developing member economies. Thus, the APEC Secretariat requires specialized technical assistance to support targeted APEC member economies, as well as Secretariat technical functions such as the Project Management Unit’s (PMU) and Information Technology (IT) Unit’s ability to prepare and support developmentally sound and high quality projects that will help APEC achieve its aspirations as outlined in the Bogor Goals, and that underpin APEC’s three pillars (trade and investment liberalization, business facilitation, and economic and technical cooperation). APEC Committees, Fora, and Related Functions Engaging Member Economies The 21 member economies are committed to conducting their activities and work programs on the basis of open dialogue, transparency, and consensus, with equal respect for the views of all participants. The APEC Chair, which rotates annually among member economies, is responsible for hosting the series of working level meetings and Senior Officials Meetings (SOMs) throughout the year, culminating in the annual APEC Ministers’ Meeting and APEC Economic Leaders’ Meeting. SOMs – usually three or four – are held throughout the year, prior to every annual Leaders’ Meeting. At each year's Leaders’ Meeting, representatives of member economies define work programs for APEC's four committees (Committee on Trade and Investment, Budget and Management Committee, Economic Committee, and the SOM Steering Committee on ECOTECH), numerous technical sub-committees and working groups, and other APEC fora and sub-fora. Committees work on issues concerned with APEC’s “three pillars,” providing information and analysis on economic trends, economic and technical cooperation, and APEC administration and budget issues. Working groups promote practical economic and technical cooperation. The major working groups include Agriculture Technical Cooperation, Anti-Corruption and Transparency, Emergency Preparedness, Energy, Experts Group on Illegal Logging and Associated Trade, Health, Human Resources Development, Oceans and Fisheries, Small/Medium Enterprise, Telecommunications and Information, Tourism and Transportation. See the appendix for an organogram showing the complete structure of APEC Committees, Sub￾Committees, and other fora. E. U.S. GOVERNMENT SUPPORT TO ASIA-PACIFIC REGIONAL INSTITUTIONS U.S. Government-financed programs have directly supported ASEAN since 2004 and APEC since 2008. USAID/RDMA's involvement began with the joint recognition by the U.S. Department of State and USAID that USAID assistance provided added value to the U.S. foreign policy relationship with these regional institutions. USAID/RDMA has forged strong partnerships with the relevant State Department offices to provide management and technical oversight, and co-funding in the case of ASEAN, for multi-year technical assistance programs to support the priorities of United States policy engagement with these institutions, including increasing the strategic and tactical management capacity of their respective Secretariats. 6 U.S. Government APEC-focused support was designed to enhance REI and cooperation within the Asia-Pacific region; contribute to APEC’s effort to become a stronger, more strategically managed regional institution; and assist APEC and its Secretariat to address the issues of the three pillars, including the possible creation of an Asia-Pacific Free Trade Area. U.S. Government assistance consisted of long- and short-term technical assistance to develop and implement priority technical capacity building initiatives, including policy studies, assessments, training, and advisory services, taking into account U.S. Government legal and policy restrictions. Activities would directly support and achieve measurable progress in furthering APEC priorities, as well as complement ongoing work by APEC and other donors.11 In both APEC and ASEAN – and notwithstanding the notable distinction that the U.S. Government is a full member economy in APEC, while only a recognized external partner to ASEAN – support services to both organizations are provided from platforms structured as Technical Assistance and Training Facilities, embedded in the organizations’ respective Secretariats. Importantly also, RDMA's programs operate alongside other USAlD and broader U.S. Government support to these institutions and seek to add value to the full spectrum of U.S. assistance in accordance with USAID’s comparative strengths, while extending the benefits of such assistance across trans-national boundaries in consonance with the concept of regionalism. U.S. foreign policy engagement through these key regional institutions has been greatly enhanced over the last several years. Due to strong relationships built within ASEAN and APEC and effective programs that tangibly contribute to their regional integration agendas, USAID is now well-positioned at the center of the U.S. Government’s partnerships and policies in the region to address the trans-boundary and global challenges of balanced economic growth, disaster and healthcare management, climate change, food and energy security, human rights, transnational crime, and improved education. The demonstrated appreciation and demand by APEC and ASEAN member nations for U.S. Government-sponsored technical assistance and training, coupled with a renewed U.S. Government emphasis on deepening trade and diplomatic relationships between the United States and the Asia-Pacific countries have created new opportunities for furthering an enabling environment conducive to sound economic policies and for bolstering regional economic integration, free and open trade, competitiveness and greater investment flows. As broad-based growth continues, the development divide between the more developed and less developed member economies of the Asia-Pacific region is narrowing, and the human development index of the region as a whole is also improving. 11 Activity Approval Document, Regional Development Mission-Asia, Strategic Objective 486-002: Improved Regional Governance and Economic Reform, Amendment # 1, August 29, 2008. 7 II. FILLING AN APEC NEED WITH THE TATF A. RATIONALE: THE U.S. GOVERNMENT’S OBJECTIVE IN SUPPORTING APEC WITH A TATF The United States has always maintained an influential leadership role within APEC, through a combination of its high-level engagement in SOMs, Ministerial and Leaders’ Meetings, its active participation in the full array of technical fora, and through the additional voluntary financial and in-kind contributions it has routinely offered since it first affiliated with the organization. The Clinton, Bush and Obama Administrations all have reaffirmed during their respective terms that: 1) APEC remains America's primary vehicle for advancing both economic cooperation and trade and investment liberalization in the Asia-Pacific region; and 2)the United States is a Pacific nation and APEC is an essential element of U.S. engagement in the Asia-Pacific region.12 The rationale for current and potential future U.S. Government support to APEC is based on three key principles. First, the Asia-Pacific region is critical to current United States foreign policy and national interests. Also, the regional institutions of ASEAN and APEC are essential to addressing global challenges. At the U.S.-hosted annual meeting of APEC Leaders in 2011, President Obama stressed his Administration’s belief that “the Asia-Pacific region is absolutely critical to America's economic growth. And we consider it a top priority because we're not going to be able to put our folks back to work and grow our economy and expand opportunity unless the Asia-Pacific region is also successful.”13 Second, the provision of technical assistance aimed at narrowing the development gap between the developed and developing economies in the Asia-Pacific region constitutes sound and sustainable development policy and lies at the heart of USAID’s values and core objectives. It is being delivered largely in the form of capacity building of policy makers from APEC economies and through the APEC Secretariat. Third, APEC has created policies for regional economic integration and has achieved significant results that are benefitting its member economies.14 APEC also has ambitious forward-looking goals for integration and policy alignment that address critical trans-boundary issues facing the Asia-Pacific region. Achieving coherence between the U.S. Government’s regional and bilateral support in the APEC economies is critical for ensuring the effective achievement of United States foreign policy goals and maximizing development outcomes in individual countries by making them more productive members of their respective regional bodies. B. ORIGINS OF THE TECHNICAL ASSISTANCE AND TRAINING FACILITY As originally conceived, the goals to provide additional support are implemented by the U.S. Government by offering APEC through technical assistance and training. The goals are to enhance regional economic integration and cooperation within the APEC region; contribute to APEC's effort to become a stronger, more strategically managed regional institution; and assist APEC and the Secretariat in addressing the components of the institution’s three pillars, including the possible creation of an Asia-Pacific Free Trade Area. 12 http://2001-2009.state.gov/p/eap/rls/fs2007/91885.htm 13 http://www.whitehouse.gov/the-press-office/2011/11/13/opening-remarks-president-obama-apec-session-one 14“APEC’s Bogor Goals Progress Report – Highlights of Achievements and Areas for Improvement,” APEC Policy Support Unit, August 2012. 8 While the U.S. Government historically has provided “support” to many public international organizations(PIOs) in all regions of the world, including those nominally promoting trade development, customs reform, and regional economic integration, the objectives of such support and the modalities through which it has been provided vary largely over time and space. Searches in USAID’s Development Experience Clearinghouse15 with respect to such project assistance reveal an array of diverse approaches that can be, and have been, deployed. For example:  The Southern Africa Global Competitiveness Hub has received U.S. Government support to perform diagnostics of trade frameworks, to assist in implementing recommendations, and to mobilize stakeholders in trade policy, infrastructure, and finance. They are regarded as key facilitators of World Trade Organization and World Customs Organization trade assessments in the Southern Africa region.  With U.S. Government support, the Caribbean Competition and Consumer Law and Policy Technical Assistance program has assisted the Caribbean Community Secretariat and Caribbean nations to build institutional capacity to apply competition and consumer law and policies, along with other legal and regulatory frameworks that facilitate the successful operation of a competitive market economy characterized by conditions that encourage entry into markets and undistorted consumer choice among competing offers in the marketplace.  U.S. Government support for the Southern African Customs Union Free Trade Agreement has provided technical assistance to member states of the regional customs union to assist them in the negotiation and implementation of an FTA with the United States. The key take-away in examining this sample of assistance arrangements for regional economic integration PIOs is that there have been historically many patterns and approaches for organizing the technical assistance modalities of such support. Some assistance is short-term episodic; other assistance has been more long-term. Some calls for the technical support mechanism to be in-house, while some is organized external to the recipient organization. Some assistance is resourced predominantly with local consultants, while still other times relies on international experts. And some focus on working at the institutional center, while others are decentralized to work throughout the region. But in their structures and operating modalities, what they have in common is that all have been designed to respond to their patrons’ and stakeholders’ specific requirements in their respective areas of operation. Notably, no discernible one-size-fits-all models or approaches are found that can be deployed directly off the shelf. Each arrangement responds to particular circumstances and goals. The same principle applies to APEC, a unique organization with several stakeholder constituencies, idiosyncratic established procedures, internal and external structures, and operational requirements. Perhaps most notable among the drivers of APEC assistance is the community of deeply engaged diverse U.S. Government stakeholders who compete for, and are potentially able to profit enormously from, the broad array of support that financial and human resources provided through a TATF-like platform could address. This unique and compelling U.S. Government interest in servicing its agencies’ requirements for support in APEC processes (which will be addressed in section G of this chapter, below) is arguably the most compelling aspect that drove the technical assistance design for APEC, resulting in TATF. 15 See https://dec.usaid.gov/dec/home/Default.aspx. 9 Additionally, with the sole exception of ASEAN, organizational capacity building, as opposed to supporting sectoral activities, e.g., in agriculture, educational partnerships, health, or environment specifically, has seldom been the principal objective of the U.S. Government in its assistance to most PIOs. For APEC, as will be discussed below, an agenda of demand-driven and flexible administrative and multi-sectoral internal and external capacity building was clearly identified as a need at the outset. ASEAN’s experience with a dedicated TATF platform operating inside its headquarters was quickly perceived as the closest thing to a model available, and was embraced as the prototype around which modifications as described below could be applied to create an equally responsive facility for APEC. Assistance to strengthen the APEC administration and to support its program activities would be two-pronged – those contributing to supporting the category of policies that underpin the regulatory and structural reform processes leading to a free and open trade and investment environment, and those that would be of an institutional operations-strengthening nature. In the case of assisting through a TATF-like entity, its policy-support objectives would be determined through a consultative process that would serve to align APEC’s objectives with U.S. Government priorities, especially with respect to APEC’s structural and regulatory reform agenda. Specific activities would focus on filling the needs of developing member economies – especially Papua New Guinea, Vietnam, Indonesia and Philippines – to help these economies progress along the continuum of APEC’s policy framework for promoting economic growth and to foster a regulatory environment in which trade and investment can flourish. To this end, U.S. Government-supported long and short-term technical assistance and capacity￾building activities, including policy studies, assessments, training events, conferences and seminars to share best practices, and advisory services, would be intended to reinforce APEC’s efforts as they are organized under APEC’s Committees and sub-fora. With respect to technical assistance that focuses on improving the internal operational efficiencies of the APEC Secretariat, U.S. Government support would be offered to upgrade such areas as:  project design, implementation, management and oversight;  monitoring and evaluation of cooperatively conceived development interventions, intended to address development issues found in one or more of the APEC economies;  IT and data-base management;  staff training and team building;  long-term strategic planning;  the use of logical frameworks and identification of causal relationships in project conceptualization and medium-term strategic planning; and  project proposal development for managing technical issues related to the Bogor Goals. Such assistance would be intended to leverage and build upon APEC’s own project funding through the Trade and Investment Liberalization Fund (TILF), the APEC Support Fund, and the APEC Operational Account, as well as in partnership with other unilateral funding support such as Australia’s Effectiveness Grant. It would be coordinated closely with Australia, Brunei, Canada, Japan, New Zealand, South Korea, Singapore and other donors that provide extra￾budgetary support to these accounts. Programs developed by other member countries and submitted to one of APEC's funds for funding consideration could also be co-funded by U.S. Government resources, to the extent that the co-funded portion is eligible under the laws, rules, and policies governing United States foreign assistance. 10 C. APPROVAL AND AUTHORIZATION BY USAID FOR ESTABLISHING THE APEC TATF On August 29, 2008 USAID/RDMA authorized the use of funds for establishing a TATF platform that would be embedded in the APEC Secretariat to provide technical assistance “in furtherance of United States foreign policy goals of greater regional economic integration and to strengthen APEC as a regional institution.” The purpose of the APEC TATF would be to assist in regional economic integration, increasing the efficiency of the APEC operations, strengthen the APEC Secretariat, and work in three broad technical areas – trade and investment liberalization, business facilitation, and economic and technical cooperation to meet the Bogor Goals.16 The USAID APEC TATF authorization also stated “strategically facilitated progress in these three areas by the APEC TATF will enable poorer APEC member economies in the region to have stronger voice and participation, help focus APEC programs to respond to the needs of poorer members, reduce income disparity, further strengthen their economies, pool resources and achieve greater efficiencies.” D. CONTRACT AWARD: RDMA AS APEC’s TATF IMPLEMENTATION OVERSIGHT AGENCY State and USAID determined that based on four years of experience with U.S. Government￾supported technical assistance and training for the ASEAN Secretariat through an ASEAN TATF co-located at its headquarters in Indonesia, the modality for providing TA and training to APEC’s Secretariat in Singapore would be similarly configured. That is, a strategically positioned platform, an “APEC TATF,” would be co-located within the APEC Secretariat, from which diverse and highly specialized technical assistance services and training events concerned with regional economic integration and capacity building of the Secretariat are deployed. A more detailed discussion of the operating principles for the TATF follows: As originally contracted, the APEC TATF was to be a four-and-a-half-year USD17,657,720 project managed by the USAID Regional Development MissionAsia (USAID/RDMA) with Economic Support Funds (ESF) from the U.S. Department of State’s Bureau of East Asian and Pacific Affairs, Office of Economic Policy (State EAP/EP). It was awarded in late September 2008, and originally had an end-of-project (EOP) date of 1 March 2013. The EOP was recently extended to December 2013. Procurement of the APEC TATF implementing contractor was accomplished through a Task Order (TO) under the Support for Economic Growth and Institutional Reform-General Business, Trade and Investment (SEGIR GBTI II) Indefinite Quantity Contract (IQC). The IQC was selected through a full and open competitive procurement process. Under the IQC, the consortium of Development Alternatives Inc. and Nathan Associates was selected to implement the U.S. Government-supported activities in APEC. The DAI/Nathan Group (DNG) structure for implementation of the APEC TATF Task Order has played out as was articulated in their technical proposal to USAID in 2008. Under each task order released under the GBTI 2 IQC, DNG determined which firm would be the “lead” firm for technical and contractor implementation, and that firm would be responsible for the implementation of that task order. For the APEC TATF, Nathan was the lead firm and point of contact (POC) and would draw on DAI for short term technical assistance, which in fact was done in a number of 16 Action Memorandum for Mission Director, Activity Approval Document 486-002, "Improved Regional Governance and Economic Reform" – Amendment Number 1for Asia-Pacific Economic Cooperation Technical Assistance and Training Facility (APEC TATF), 8/29/2008. 11 areas including green buildings and project quality training. In practice, the characterization of the various roles that would be carried out under the DNG technical assistance consortium was actually implemented in the execution of the 2009 - 2012 annual work plans. Such assistance was intended to respond to the challenges faced by APEC members in reaching the Bogor Goals, especially as they relate to the gamut of policies and practices that underpin every facet of regional economic integration. In practice, Nathan Associates responded to institutional and human capacity strengthening needs of the APEC Secretariat to more effectively and efficiently serve its 21 members. Such assistance included IT training and communications equipment upgrade, staff professionalization, data management, and medium-term strategic program planning and implementation aimed primarily at the program leaders of the technical working groups, and representatives of the Project Management Unit. During the four-year implementation period, TATF held more than 40 Bogor Goals events related to APEC’s three pillars. The events were attended by approximately 2,000 APEC economies’ participants who benefited from technical presentations by hundreds of experts deployed from developed and developing economies alike. The U.S. Government support-eligible countries that could have drawn on TATF funding resources for TA support and travel of participants to TATF-organized events were clearly defined in the IQC Task Order. While the U.S. Department of State exercises high-level policy engagement with APEC and policy oversight on the APEC TATF project, USAID/RDMA supervises the TATF’s contractor and is the accountable administrator of State Department ESF funds to implement APEC assistance through the TATF. As the U.S. Government’s primary development assistance agency, USAID has a special stake in the developmental outcomes of TATF capacity-building and policy-development initiatives. These initiatives incorporate the principles of development to the extent that planned activities are also geared to address the issue of narrowing the development gap between the developed and developing APEC member economies. E. HOW TATF OPERATES The TATF platform is embedded in the APEC Secretariat and consists of five long-term Nathan employees, three of whom are expatriates serving in the capacity of COP, Deputy COP, and Program Coordinator and Trade Specialist respectively. The two other staff members are Training and Program Support Specialists, and citizens of Indonesia and Singapore respectively. The Singapore-based long-term team receives administrative and technical support from Nathan headquarters in Arlington, Virginia, where a full-time program officer and executive management oversight are provided. TATF also draws on the “DNG Consortium” of technical experts (a joint venture of Development Alternatives, Inc., and Nathan Associates) to fill short-term consultancy needs as they arise. In practice, the overwhelming majority of short-term experts were fielded by Nathan Associates. Now approaching the end of its fourth year of implementation, the technical assistance contractor has established a robustly functional assistance platform. They have collaborated with the APEC Secretariat’s leadership and other principal stakeholders – including secondees from the APEC member economies, core Secretariat staff, member economies, other APEC donors, regional and international organizations, the U.S. Government inter-agency working group led by the State Department, the private sector, and NGOs. They have produced a prodigious amount of deliverables centered on REI, and have earned the approbation of the APEC Secretariat leadership. 12 TATF and the Current-Year Host Economy The project responsibilities that are carried out in cooperation with APEC Committees, working groups and other fora, include TATF implementing or otherwise facilitating dozens of conferences, workshops, seminars, assessments and diagnostic studies each year. These activities take place throughout APEC’s member economies, focus on specific sectors of activity or sub-regional groups of economies, and are widely attended and/or contributed to by member economy government officials, private participants, academics, and experts coming mostly but not exclusively from member economies. Leadership in the process of setting the agenda for TATF’s diverse array of engagements with member economies generally comes from SOMs and from the U.S. Government inter-agency oversight committee, however it also is influenced perforce by the preferences of the economy that is “hosting” APEC for that year. Since TATF’s inception, Indonesia has been a power user and intensive participant in APEC activities. In this respect, Indonesian officials have had a wide range of contacts with TATF – i.e., through being a lead subject in many assessments and diagnostics, being a pathfinder economy in new areas, engaging in multi-event sequenced initiatives – and in a wide variety of technical areas, including ease of doing business (EoDB), structural and regulatory reform, public participation in the rule-making process, public sector governance, corporate governance, food security, and other areas under the SOM Steering Committee on Economic and Technical Cooperation (SCE), which Indonesia chaired in 2012. As Indonesia is slated to accede as APEC host in 2013, they were able to get a head-start influencing directions for their year through such Committee chairmanships as the SCE. Indonesia’s APEC interlocutors cited a number of ways in which TATF had assisted them, and in which they intended to call upon TATF for more support during the 2013 Indonesia year. One of the most promising areas mentioned was Indonesia’s intention to broaden the focus of smaller APEC working groups to integrate more directly and at a more strategic level with other fora responsible for complementary issues and thereby to more comprehensively address complex issues less narrowly. Examples included hoped-for TATF support for supply chain connectivity, food security, and disaster resilience and response, all areas where Indonesia has signaled its intention to focus during 2013. TATF and the Secretariat and Other Headquarters Functions TATF’s contributions to building capacity across the operational units of the Secretariat are diverse and deep. At the highest level of the organization, for example, at the inception of his tenure, the APEC Secretariat ED called upon TATF to support his initiatives on staff evaluation, strategic planning, team-building, developing a comprehensive Secretariat training program, implementing quality project management training for attendees at SOMs, and to upgrade the quality of the project development process. He invited TATF to develop a training-of-trainers program to help the developing as well as developed member economies put together higher quality project proposals, and he demonstrated his confidence in the resident TATF team by inviting them to participate in his weekly meetings with Secretariat senior staff. For the PMU, TATF supported the initial formulation of the APEC Project Guidebook and currently is working to digitalize the Guidebook and project concept note and proposal development template as an interactive on-line tool. TATF has helped in the development of training for members of APEC Committees and working groups to improve the quality of project design and management. TATF cooperated with the Project Management Unit in resourcing a PMU-designed SOW for a searchable project data-base, which, when completed in 13 early 2013, will be available to all APEC stakeholders, and will integrate operational, financial and project output information, facilitating implementation of multi-year programs and more effective monitoring and evaluation. And for the remaining period of the project, TATF will continue its cooperation with AusAID in designing a modern and effective post-project impact evaluation process. In the IT domain, TATF managed the procurement for consultants based in Singapore to upgrade the APEC website content management system, assisted with accounting system data￾sharing that contributes to support of multi-year projects, upgraded e-mail security and capabilities, and trained all Secretariat staff for transition to M/S Office 2007. TATF expeditiously upgraded the Secretariat’s networking capabilities with member economies by quickly and at low cost designing and installing a digital video communications system at APEC headquarters. All told, during TATF’s time at APEC, the Facility has contributed approximately USD 0.6 million to support IT equipment and software upgrades. The Policy Support Unit is an independent economic research and analysis entity affiliated with APEC that is supported through voluntary contributions largely from AusAID and a few other member economies, including the United States. It is governed by its own board of directors, and its main responsibility is to carry out evidence-based empirical research and analysis to support APEC project initiatives, and to advance and measure member economies’ progress in achieving critical milestones toward Bogor Goals. There is a symbiotic relationship between PSU and TATF. Where TATF has developed special expertise or is deeply involved in APEC initiatives such as Ease of Doing Business, regulatory and structural reform, and environmental goods and services, TATF is called upon to contribute from their comparative advantage – and harmoniously with the PSU – to such regular publications as the Economic Committee’s “APEC Annual Economic Policy Report.” Similarly, where TATF is involved, for example, in extensive training of member economy officials in implementation of structural reform, PSU offers support in conceptualizing the training programs and in vetting effectiveness of different training approaches. Another measure of the partnership between PSU and TATF has been their cooperation in leadership on APEC’s behalf of the sectoral “mapping exercise” and other dialog objectives established between APEC and ASEAN to identify areas of complementarities and redundancy in the two organizations’ programs. Although of only limited success thus far in affecting APEC￾ASEAN program harmonization, several candidate sectors including SME, health management with emphasis on pandemics and the harmonization of Customs procedures, have been targeted for possible action, as have more strategic questions such as linkage between the ASEAN Economic Community and the nascent Trans-Pacific Partnership, and the role of the “ASEAN Caucus” inside APEC. APEC TATF, in coordination with ASEAN TATF, have comprised an important circuit for such communications, albeit thus far informally. F. PROGRAM CONTENT TATF’s PMP as a Frame of Reference The following abstract from APEC TATF’s draft Performance Management Plan (PMP) of April 2009 (pp. 3-4) characterizes the project’s strategic objective and intermediate results as: 14 I. Expected Intermediate Results APEC TATF activities are implemented within the framework of the above-stated objectives that are pertinent to APEC’s current priorities. The TATF applies a two-pronged approach to supporting APEC, with its activities falling under either of the following two categories: Work aimed at the APEC Secretariat; and Work under APEC’s “Three Pillars” Category 1 activities cater to the needs of the APEC Secretariat in managing its day-to-day operations. These activities seek to improve the Secretariat’s internal operational efficiency and effectiveness by providing professional capacity building exercises, assisting with establishing IT systems for internal operations and communications with member economies, and supporting public outreach programs to promote regional principles for trade and investment in member economies. Category 2 activities support APEC’s policy objectives under its “Three Pillars” and enhance REI and cooperation within the APEC region. These activities are intended to complement APEC’s programmatic priority areas, as stipulated in APEC’s policy documents . . . Category 2 activities incorporate a wide range of program areas, including but not limited to customs, standards and conformance, electronic commerce, business mobility, competition policy, regulatory reform, public sector management, corporate governance, economic and legal infrastructure, environment, and gender. TATF’s focus for each work plan is determined by a consultative process that matches APEC’s policy objectives with the U.S. Government’s priorities. The contract between USAID and the DNG consortium called for the contractor to focus on the following areas:  Identification and provision of technical expertise to buttress the present and planned work of the APEC Secretariat and APEC economies in their efforts to achieve the requirements of the Bogor Goals and Regional Economic Integration (REI).  Identification and implementation of a strategic plan for the provision of technical assistance and training activities that build capacity of the APEC Secretariat as an organization and link the APEC Secretariat with policy centers of excellence around the world.  Expanded collaboration with the ASEAN Secretariat and other relevant regional and international organizations, as well as the donors, assistance providers and programs attached to them.  Collaboration with the private sector in APEC member economies.  Working in partnership with the APEC Business Advisory Council (ABAC), other business organizations, NGOs, and civil society organizations as appropriate, develop and implement activities that will improve regional cooperation on transnational issues, including but not limited to: trade and investment facilitation, customs facilitation, regional security and counter-terrorism, transparency and regulatory reform, anti￾corruption, human resource development, strengthening privacy and intellectual property regimes, biotechnology, energy security, environmental issues, health issues such as Avian Influenza and HIV/AIDS, disaster management and emergency response. 15 G. HOW THE U.S. GOVERNMENT POLICY ROLES IS CARRIED OUT AND COORDINATED The U.S. State Department (EAP/EP) chairs an inter-agency committee of technical departments and U.S. Government agencies who in most cases are the U.S. leads for APEC’s technical committees and working groups. These U.S. Government sectoral experts interface with their counterparts from other APEC member economies to identify programming gaps, define program areas and elements, and develop activities for U.S. Government support that would contribute to achieving economic integration and structural reforms in member economies as specified under APEC’s Bogor Goals. U.S. Government inter-agency participants are stakeholders in TATF, and in that capacity are partners in setting the annual program agenda for “projects” that TATF may contribute to executing or facilitating in conjunction with other member economies. The U.S. Government lead for each activity works closely with TATF staff to guide its implementation. Once activities are included in the TATF’s annual work plan and are in the planning stage, the relevant working group, committee, and eventually the APEC Budget and Management Committee is often offered the opportunity to clear on the technical appropriateness, timeliness, and cost-effectiveness of these proposed activities. Informal consultations on annual work plans or program agenda development begin around the end of the calendar year, based on outcomes from the annual Leaders’ Meeting. While the main drivers of this process are U.S. Government stakeholder agencies, including U.S. Department of State, USTR, USDA, and Department of Commerce, stakeholders in the development of this work plan are many, including leadership of the APEC Secretariat, and the host government that will chair the ensuing annual Leaders’ Meeting. In the case of activities funded by the U.S. Department of State through transfer of ESF funds to USAID, and for which the U.S. Department of State is responsible for policy engagement with APEC, RDMA is engaged in extensive consultations with the State Department’s East Asia and Pacific Bureau’s Office of Economic Policy – which comprises the U.S. APEC Senior Official and his staff – on the strategic programming focus of U.S. foreign assistance to this regional organization. Following establishment of general agreement among these stakeholders on the scope of effort to be expected of the APEC TATF, the TATF endeavors to submit the draft annual work plan and program elements to the RDMA COR by the second quarter of the fiscal year for approval. 16 III. METHODOLOGY, EVALUATION QUESTIONS, AND SURVEY FINDINGS A. EVALUATION METHODOLOGY SUMMARY A detailed discussion of the methodology employed to carry out the evaluation is presented in the “APEC TATF Mid-Term Evaluation Strategy and Action Plan,” which can be found in the appendix. An external two-person team, hired by RDMA, formed the core of the evaluation team and participated in all five phases of the exercise, which began formally on September 6, 2012. The first phase – the preparatory phase – was implemented in the United States, during which time the team acquired a comprehensive grounding on APEC and the APEC TATF U.S.- based stakeholders and on the implementation support structures of the contractor. During the second phase, the Strategy and Action Plan preparation, evaluation hypothesis, fieldwork and survey methodology were vetted and approved at USAID/RDMA/Bangkok over a two-week period beginning on September 24, 2012, and involved the participation of the TATF COR and TATF Contractor COP. In early October 2012, for three weeks, the evaluation’s focus moved to field visits and stakeholder interviews in Singapore, Indonesia, Vietnam and Thailand. At the same time, a large-scale survey was administered to former participants in TATF-facilitated events. Further details on these enquiries are found in the Appendices in Volume 2 of this report. B. MID-TERM EVALUATION PURPOSE, OBJECTIVES, AND HYPOTHESIS The Mid-Term Evaluation Statement of Work (see the appendix) calls for the evaluation team to evaluate ongoing programs and inform management decisions about the effectiveness of APEC TATF’s current and future activities and the broader context of United States assistance to APEC. There are two key objectives of this exercise: 1) evaluate contractor performance with respect to compliance with the award by meeting agreed-upon project objectives and intermediate results; and 2) develop a lessons-learned package of information around which the current program can be shaped and future projects can be designed. Additionally, the merits of the TATF platform are examined. Herein, the evaluation also addresses whether the TATF serves as an institutional catalyst for change within the regional body to encourage a movement from the best intentions of management to achieve intended outcomes through actionable work plans, and whether the TATF provides a platform to support regional and negotiated commitments to allow APEC to effectively achieve mutually beneficial goals of member economies. Toward this end, the evaluation identifies TATF’s approaches, programs, and interventions that have been successful and should be duplicated or continued in future program design. And along with such strengths, this evaluation also reports on weaknesses that USAID and State/EAP/EP should address in any future APEC project design. The lessons learned through this evaluation may also serve to inform additional projects with the APEC Secretariat and/or other U.S. Government engagement with regional bodies. The evaluation links TATF performance in providing training, technical assistance, and advisory services to the APEC Secretariat as well as through the Secretariat to launch projects and activities in member economies (primarily lesser developed), to the following measures: 1) measures of improved organizational and management performance by the Secretariat; 2) demonstrated evidence of education and consensus-building, and progress in meeting requirements around improved economic and trade practices and protocols for achieving the 17 Bogor Goals; and 3) inferentially derived measures of success in achieving United States foreign policy objectives in the Asia-Pacific region. Evaluation Questions: The Evaluation Statement of Work (pp. 19-20)17 identified three key evaluation questions to be answered: 1. How has each APEC TATF program, as well as the implementing partner, changed APEC as an institution? 2. To what extent did the services provided by the facility further international trade and investment and assist regional integration? 3. What are the most important areas of focus for U.S. assistance in the future to achieve the greatest impact given limited budgets and APEC capacity? As stated above, the Strategy and Action Plan preparation, evaluation hypothesis, fieldwork and survey methodology were vetted and approved at USAID/RDMA, and involved the participation of the TATF COR, TATF Alternate COR, and Contractor COP. During the vetting process the three evaluation questions were mapped into the following two-part evaluation hypothesis as discussed in the appendix of this Evaluation Report:18  Through demand-driven training for Secretariat staff, and the identification of priority management operations improvements, U.S. APEC TATF has strengthened the institutional ability of the APEC Secretariat to carry out its mandate; and  U.S. APEC TATF-supported assistance is furthering the aims of APEC member economies to achieve the Bogor Goals of regional economic integration and free/open trade, which materially support U.S. strategic interests in the Asia-Pacific region. As discussed earlier, TATF is constructed as a platform for delivering specialized institutional and capacity-strengthening services targeted at the APEC Secretariat, and is as well a source for supporting the Secretariat’s capacity to facilitate achievement of the APEC mandate. Therefore, to the extent possible, the evaluation team examined the appropriateness of the TATF’s structure for facilitating the capacity-building needs of the Secretariat and the effect of the structure on enabling the Secretariat to effectively contribute to achievement of the APEC mandate. The quality and effectiveness of the contractor and the contractor’s staff (addressed below in Chapter IV) were also considered. These two dimensions – structure and staff quality – were differentiated, as they could function independently to determine the quality of the overall performance and effectiveness of the Secretariat. Therefore, an objective of the evaluation exercise was to assess the engagement and outcomes of the current TATF staff and the contractor unit supporting APEC. The analysis is meant to provide USAID with insight into the efficiency and productivity of the contractor. The evaluation findings are aimed at allowing stakeholders to gain insights and reach conclusions about the effectiveness and efficiencyof specific activities conducted by the TATF for the Secretariat and participating APEC member economies in order to determine their value-added, both qualitatively and quantitatively. The evaluation findings also are intended to allow stakeholders to determine the validity of the evaluation hypothesis (described above), utility of performance monitoring efforts, factors in the 17 RDMA/GDO APEC TATF Mid-Term Evaluation Statement of Work, August 31, 2012 (see appendix). 18 Appendix - APEC TATF Mid-Term Evaluation Strategy and Action Plan, September 26, 2012 18 development context that may have an impact on achieving results, and the types of actions that need to be taken to improve performance in the designing and providing of assistance. C. SURVEY RESULTS In addition to the in-person interviews, through which the majority of information for this evaluation was gathered, a survey of participants in TATF-assisted events was designed and distributed in parallel with the field work and country visits. All told, the survey was sent to 1327 participants that attended 26 randomly selected seminars, workshops, and conferences – the 43 “major” events that TATF facilitated to some degree during the life-of-project through the period in October 2012 when the questionnaire was distributed. The response rate to the survey was sixteen percent. The survey instrument was designed to elicit opinions on the quality of the services and support provided by TATF, and to attempt to assess the utility of the workshop speakers, information, and discussions provided at these events in so far as they informed subsequent measures taken by participants in their home economies to implement or accelerate reforms and actions consistent with the achievement of Bogor Goals.19 A few summary findings from the survey may be instructive and are gathered here. To the question “to what extent did what you learned in this workshop facilitate your understanding and/or effectiveness in helping your economy to achieve the following APEC Bogor Goals,” responses were as follow: Helped A Lot Helped Somewhat Did Not Help at All N/A Trade and Investment Liberalization 60 96 11 28 Business Facilitation 79 89 10 19 Economic and Technical Cooperation 98 90 2 9 The responses above indicate that the vast majority of respondents believe they profited from attending these events, and demonstrate at the minimum that APEC’s intent in supporting these projects to achieve the Bogor Goals, as assisted in implementation by TATF, was largely achieved. With respect to the question “how helpful was the workshop in building your ability to advance the following reform objectives,” respondents clearly indicated that they perceived they had received benefits across the broad array of skills helpful to the reform process. 19 See the appendix for a copy of the survey questionnaire (distributed as a Survey Monkey link embedded in a personally addressed e-mail) and the appendix for the list of seminars/workshops/conferences that were canvassed. The appendix also contains summary quality and content data derived from the 212 partially and fully completed questionnaire returns. 19 Very helpful Somewhat helpful Not at all helpful N/A Planning a strategy for reform 90 97 5 10 Developing qualitative and quantitative indicators for measuring progress toward achieving reform objectives 83 96 13 10 Building consensus around reform objectives among other stakeholders in your economy 73 110 8 12 Defining technical assistance activities to help move forward the reform process 94 94 8 7 Implementing technical assistance activities to help move forward the reform process 71 108 13 11 Complementary to the above responses concerning skills acquired, on the question, “please rate how helpful this facilitated workshop was in helping you to work with colleagues and other organizations in your economy to achieve reform,” 96.9 percent of respondents indicated that they found their respective facilitated workshop very helpful or somewhat helpful. 20 IV. CONCLUSIONS AND LESSONS LEARNED This Chapter focuses on the three overarching questions that the evaluation team addressed: 1) how has (a) each APEC TATF program, as well as (b) the implementing partner, changed APEC as an institution; 2) to what extent did the services provided by the Facility further international trade and investment and assist regional integration; and 3) what are the most important areas of focus for United States assistance in the future to achieve the greatest impact given limited budgets and APEC capacity. These three questions have been divided into several parts, each falling within the overall rubric of institutional capacity development and human capital formation. These are essential components of a robustly able Secretariat as characterized by the efficient and effective discharge of its functions, thereby spurring growth of a vibrant and outward-looking economic trading bloc, as demonstrated by APEC’s historic performance and projected trends for the future. As a corollary, an operationally effective and technically well-endowed and administered APEC Secretariat would facilitate accelerated changes including structural, regulatory, and economic reforms across all member economies for deepening REI and broadening the base for sustainable growth. Successful trade integration calls for a modification of the economic structures of concerned countries. These changes necessarily happen slowly, as a gradual process, as has been seen in the case of the European Economic Community, which began its economic integration processes in the late 1950s, and took more than 40 years for fiscal and monetary reforms to be established, harmonized and implemented, thereby allowing for the adoption of a single currency, the Euro. The evidence that free trade spurs growth and raises income levels around the world is very strong, as evidenced by the proliferation of free-trading blocs, FTAs, and RTAs, APEC being no exception.20 In fact, every country that has managed to raise its standard of living has somehow integrated itself commercially with the rest of the world – a necessary condition. And, in the case of APEC, since its inception in 1989, the Asia-Pacific region has constantly been the most economically dynamic part of the world. For example, in its first decade APEC member economies generated nearly 70 percent of global economic growth, and the APEC region consistently outperformed the rest of the world. Moreover, despite the Asian Financial Crisis in 1997-1998, where poor corporate governance was identified as a contributing factor to the depth and breadth of the economic contraction that the APEC region experienced, by 2007 APEC member economies accounted for approximately 40 percent of the world's population, about 54 percent of the world's gross domestic product and about 44 percent of world trade. And, despite the external challenges confronting APEC, as evidenced by the recession in Europe and a lackluster economic recovery in the United States, the region as a whole is expected to 20“APEC’s Bogor Goals Progress Report – Highlights of Achievements and Areas for Improvement,” APEC Policy Support Unit, August 2012. 21 outperform global trends in the period ahead, with APEC growth forecast to accelerate to 4.3% in 2012 and 4.7 percent in 2013.21 The following section discusses the inextricable linkage between a responsive and responsible Secretariat and steady progress toward regional economic integration within the APEC member economies and reaching Bogor Goals as informed by the evaluation findings. QUESTION 1(A): HOW HAS EACH APEC TATF PROGRAM CHANGED APEC AS AN INSTITUTION? Strengths of the TATF Platform and Quality of its Personnel in Improving the Effectiveness and Efficiency of the APEC Secretariat This performance assessment does not have the benefit of the existence of a counterfactual. That is, nobody can answer the following question: if there had been no U.S. Government￾supported TATF platform co-located with and embedded in the Secretariat in the last four and a half years, would the pace of REI in member economies have accelerated, remained unchanged, or actually declined? Similarly, there is no comparative observational data on which to judge the advantages of an embedded TATF versus one similarly supported and staffed, but located outside of the Secretariat. Additionally, would there have been any significant differences in measurable achievements – outputs and outcomes – at APEC’s institutional and strategic programming levels had the U.S. Government support been channeled directly to the Secretariat? The following survey findings, assertions, inferences, and direct observations aim to address these bedrock issues as credibly as possible. First and foremost there was uniform appreciation for the TATF platform and the services it provided to strengthening the Secretariat and APEC as a whole. Citing a plethora of examples and illustrations in support of reaching Bogor Goals milestones, those interviewed expressed gratitude to the U.S. Government that the TATF platform is strategically located on-site at APEC headquarters, and has proven itself as an effective and efficiently managed extension of APEC’s in-house capabilities. Interviewees were unanimous in asserting that had the TATF structure been located off-site, this would have resulted in a protracted project start-up period and thus detrimental to outputs, as Secretariat staff would have had a less favorable environment in which to build a professional relationship with the TATF staff. As a result, the APEC Secretariat’s day-to-day administrative operations could not have been carried out as efficiently and effectively. Also, the shared sense of ownership and strong professional relationships between the ED, PDs, PMU, PSU, PEs and numerous technical working groups, and the U.S. Government-supported TATF platform, could not have been as readily cultivated. Discussions with U.S. Government stakeholders yielded a number of broadly shared opinions concerning the responsiveness and effectiveness of the TATF in directly supporting U.S. Government agencies’ projects and indirectly influencing APEC’s institutional effectiveness in reaching Bogor Goals:  All judged the TATF platform to be a critical adjunct in achieving their objectives, be it through administrative, operational, or highly specialized technical support services for reaching Bogor Goals and advancing regional economic growth and United States national interests in a prosperous and stable Asia-Pacific region. 21 “APEC Economic Trends Analysis,” APEC Policy Support Unit, May 31, 2012 22  All found utility in the fact that TATF is embedded at APEC headquarters and has good communications and outreach throughout the Secretariat and the 21 member economies. On the negative side, some U.S. Government agency participants admitted frustration with the TATF work planning and approval process, which often leaves decision-making on resource allocation to the eleventh hour. Second, most U.S. Government interagency representatives as well as representatives of all four APEC capitals (Singapore, Thailand, Indonesia and Vietnam) visited, emphasized that due to limited budgets and inadequately trained human resources to carry out their respective core international, technical committee, and working group continuing functions, without TATF expertise and support they could not have achieved their voluntary targets both at headquarters and among their committees and fora, dispersed throughout the APEC member economies. For many U.S. Government agencies, limitations in their international budgets mean that without TATF’s support they risk being unable to participate in important APEC technical fora or unable to drive initiatives in APEC with high value to the U.S. Government. Third, as a source and channel for eminent expertise in REI-related workshops, seminars, and conferences, and for conducting assessments, and country-level diagnostics, TATF was recognized by U.S. Government interagency stakeholders for facilitating the agencies’ known experts’ participation and for supplementing them with experts from their own (TATF’s) staff and rosters of expert consultants, a significant percentage of which were citizens of both APEC developing and developed member economies. And all were grateful for financial support that TATF contributed, both in direct support for case studies, diagnostics, assessments, organizing meetings, and indirectly through making available limited grants for international travel and related expenses of participants from developing member economies. Fourth, with few exceptions, core APEC Secretariat staff expressed profound respect and gratitude for TATF’s contributions to upgrading professional and administrative capabilities at APEC headquarters.  APEC’s Executive Director, currently completing his third/final year in that capacity, shared his observations and judgments concerning the many ways he turned to the TATF platform for assistance in achieving his vision of upgrading the professionalism of the Secretariat. The ED’s bottom line on the full-time presence and accessibility of TATF, on the quality and quantity of its contributions, and as a visible manifestation of support by the U.S. Government for the goals and objectives of APEC, was to declare the TATF platform, at once, a win-win for both improving APEC’s institutional capability and for the United States.  IT managers were quick to point out that the TATF platform’s real impact has been as much through the quality of consultants it has provided, the expedited decision-making that has enabled quick takeoffs on important institution building initiatives that likely would have languished while waiting for institutional funding (through the APEC Budget and Management Committee), and through TATF-provided state-of-the-art IT training and team-building. These outputs and achievements have contributed significantly to strengthening APEC’s institutional capacity and relationships both inside the IT establishment and between IT and other Secretariat users.  A typical example of the TATF platform’s responsiveness in filling critical day-to-day operational and administrative needs of APEC and the Secretariat was the lightning speed with which the TATF team responded to the Secretariat’s request for 23 modernizing its communications capability in a cost-effective manner. Whereas, feasibility studies by the Secretariat and efforts to secure adequate budgeting lasted for six years without a clear decision having been reached either to proceed or not to proceed with the upgrade, “the TATF was able to procure and install a functioning video-teleconferencing state-of-the-art facility within six months.”  As a result of TATF’s responsiveness, the APEC Secretariat now communicates much more frequently with all of its member economies and at a fraction of the previous cost. Moreover, this communications upgrade has obviated the need for APEC officials and member economies’ interlocutors to schedule regular visits to APEC headquarters, thereby resulting in substantial savings in travel time and cost, streamlined decision￾making, and vastly improved productivity. Implementation Challenges and Potential for Vulnerability to the U.S. Government A key question for U.S. Government management is, how can the TATF platform’s work plan development process be made more efficient? In this regard, as noted, inter-agency stakeholders expressed frustration with delays in reaching closure in work plan or project approval processes. At worst, such delays led to postponements in making commitments, and to uncertainties in scheduling events and booking speakers and consultants. However, most such delays and uncertainties were more likely to be the result not of TATF’s missteps but of the inherent complexity of the work planning process itself, of the occasional weaknesses of proposals they were charged with moving through the APEC project approval process, and the vagaries of the internal APEC process itself. Also, findings indicated U.S. Government stakeholders’ disappointment on a few occasions in the quality of experts that TATF supplied. This complaint was similarly expressed, albeit infrequently, by Secretariat and member stakeholders. But in context, it must be acknowledged that in assessing the quality of such consultants, the view was that they were predominantly highly competent, and that in such large numbers, not every one of them will have been expected to hit a home-run. As with U.S. Government stakeholders, Secretariat colleagues occasionally criticized the TATF platform for what was thought to be their inability to reach closure on pending initiatives. In very few cases, however, could such a deficiency be attributed solely to the TATF platform. Rather, the interactions with external players and the complexity of internal procedures generally accounted for most such delays. One senior official in the Secretariat voiced, albeit sympathetically, a concern that limited TATF staff size and infrequent oversight visits by USAID managers might lead to inadequate quality control over TATF products and services, which this interlocutor deemed a potential vulnerability. Though no examples of such quality failures were presented, the burden to produce a steady stream of substantive work outputs, and the high expectations placed on the TATF team were evident, and therefore the threat was credible. These criticisms are somewhat ironic, as the strategic planning process and medium-term plans, and standards for project acceptability and the average quality of project proposals was significantly improved during recent years thanks to TATF investments in improving the planning and proposal review process, and as a result of training in proposal writing provided to developing member economy stakeholders. What was unfortunately missing in this process was downstream validation for this improved quality in the form of more project financing resources, an aspect beyond the TATF platform’s ability to control. And last, while it is not a weakness as such, there were a number of occasions where, due to resource constraints (time or finance), TATF had to discontinue its association with a multi-stage project initiative after facilitating it through its beginning or into later stages, but prior to when the ultimate outcome 24 desired by the proponents had been attained. Notably, such occasions may present opportunities for USAID Missions to step in and provide the needed follow-up assistance. Program Sustainability – TATF Platform and Dependency in the Secretariat The consideration of sustainability is paramount, especially for USAID with respect to a possible follow-on assistance project to the APEC Secretariat. Although the U.S. Government through the TATF is not the only source of subsidy through voluntary contributions to the APEC Secretariat, during the TATF era beginning in 2009 it has been the largest “donor” providing supplementary resources for APEC operations (not counting the Government of Singapore’s contribution of the headquarters premises). The Secretariat’s regular budget based on annual member assessments has remained steady throughout the TATF era at just under USD 10 million. Assessments are differentiated in amount by member economies’ levels of development. The average annual additional contribution of the U.S. Government for supporting the TATF platform during its five-year life-of-project (LOP) is on the order of USD 3.7 million. The question arises as to whether U.S. Government supplementation for Secretariat operations as administered through TATF might be less – or more? – effective were it to be provided directly to the Secretariat as a cash transfer. A number of observations consistently support the notion that the TATF platform brings to the Secretariat a flexible, and relatively minimally bureaucratic means of quickly implementing technical assistance or training services in response to revealed need, complete with accountability and quality control that pure cash without dedicated oversight could not assure. As well, other donor members, including Australia, New Zealand, Singapore, Korea and Japan, make voluntary contributions, composed of varying mixes of financing for special funds or operations such as the Trade and Investment Liberalization Fund, the Effectiveness Grant, the Policy Support Unit, additional dedicated staff, and various other in-kind contributions. The more sizeable among these contributions are highly targeted as additionally to well established procedures (e.g., the funds to finance projects), usually are not recurrent, and often are accompanied by support for implementing institutional structures (e.g., PSU and PMU) sharing some characteristics with TATF, additional contract staff, or even additional Project Director secondees to oversee the proper channeling of these resources and to ensure high quality outcomes. Another aspect of the principle of voluntarism as it applies to the extra-budgetary financing of APEC operations is the nature of expectations of member economies regarding implementation of agreements in their home economies that are formally endorsed in APEC, but which, unlike in many other international organizations, are not considered binding and enforceable, with penalties for noncompliance. These characteristics are key manifestations of what makes APEC unique. They also influence the way subsidies from the better-off member economies are perceived – i.e., not as coercing by richer members of some actions on the part of poorer members. Similarly, it is argued, avoiding agreements that are binding and enforceable in favor of voluntary agreements is respectful of the diversity in members’ levels of development. The objective is to provide highly ambitious outcomes at which to aim, while allowing at the same time flexibility in the pace of implementation suitable to the members’ respective underlying development levels. The theory is that such voluntarism and flexibility in implementation lead eventually to stronger commitments – commitments that translate more effectively into actions and reforms in the 25 home economies and sustainability as viewed in the context of APEC’s demonstrated steady and incremental progress towards achieving Bogor Goals for REI.22 Regarding the overall volume of these subsidies and the implications they have for non￾sustainability in APEC Secretariat operations, if one were to total-up the off-regular-budget contributions to operations of, and investments in, many of the Secretariat systems and equipment upgrades, e.g., for training, in IT, for video-conferencing, data-base management, etc., provided by the U.S. Government’s TATF, the operations supported by the contributions of other member economies (such as PSU- and PMU-managed activities, and project fund budget), and the unreimbursed detailing-in of Project Directors and other professional staff amounting to nearly 35 percent of Secretariat staffing, the value would approximate the Secretariat’s regular annual assessed budget. The fact is, APEC would not have the capabilities it now has were it not for these voluntary contributions, and thus, it would be a very different organization, were it to work to its regular budget exclusively – which would lead to the downsizing or elimination of many core capabilities – or were it to attempt to raise its members’ regular budget assessments to a level sufficient to cover all these costs. Ordinarily, such “non-sustainability” would be deemed a development failure, at least at the level of the organization. However, many interlocutors, policy makers, and stakeholders insisted that this operating principle of voluntarism defines the essence of APEC, and, what is more, it would not be in the interest of the organization or its members to change it. For example, nearly all activities in APEC that receive external support (including through the TATF platform) also require some regular budget contributions, triggering a participatory and transparent project review and approval process requiring a consensus among the diverse member economies. This consensus guarantees both rigor in the activity review and consistency with APEC policies and Bogor Goals, and demonstrates acceptance that the project proponent donor economy is pursuing the common interests of the organization, and not necessarily its own national interests. Program Sustainability – the Projects Side The prospect of sustainability with respect to TATF-supported activities on the projects side would appear more promising than in the Secretariat. The attestations of many survey respondents in both developed and developing economies consistently demonstrated a sense that through their participation in training and other project events, participants had developed both capacity for, and commitment to, becoming reformers and change agents in their home economies. Such outcomes reinforce the notion that notwithstanding the technical, and financial unsustainability of TATF support for Secretariat operations, and the non-enforceability associated with APEC agreements, TATF training and facilitation is widely reported as generating genuine development commitment and skillfulness at the individual level, which is convertible into positive and appropriately paced and scaled developmental outcomes at the economy level. While no attempt was made to value such outcomes, the principle that they exist, and that they have positive worth as investment in human capital, is compelling. QUESTION 1 (B): HOW HAS THE IMPLEMENTING PARTNER CHANGED APEC AS AN INSTUTUTION? The quality of the contractor’s personnel – primarily the highly able and experienced Nathan Associates-fielded staff23 of the project and the contractor’s worldwide network of experts and 22“Assessments of Achievements of the Bogor Goals in 2012, Bogor Goals Progress Report of Twenty-one APEC Member Economies, www.apec.org. 26 consultants – have combined to showcase APEC as a strong change agent for REI, while distinguishing the U.S. as a champion supporter for economic growth and stability in the Asia￾Pacific region. In as much as this evaluation is a “contractor performance review,” it is germane to assess the relative contributions of the TATF platform to effectiveness as derived from the questions 1(a) and 1(b) above as a further validation of the project evaluation hypothesis. On the contractor’s management and human resource sides, e.g., the quality of oversight, the quality of the TATF permanent staff and the quality and performance of externally hired short￾term consultants and experts, responses gathered from the participants surveyed, as well as feedback from TATF clients and partners interviewed, were overwhelmingly positive. While feedback indicated there were rare disappointments with contracted experts, the majority of stakeholders recounted far more numerous instances of highly qualified and effective personnel mobilized by TATF to support, for example, IT in the Secretariat, expert trainers at workshops, renowned speakers at conferences, sectoral specialists engaged on assessments and diagnostics, presenters and trainers on the conceptualization and preparation of quality project proposals, and the development of medium-term strategic action plans for each of APEC’s dozen or so technical working groups in order to be ready to face future challenges to REI. Particularly in the developing member economies visited (notably Vietnam and Indonesia), responsible officials often applauded the contributions made by the TATF-provided technical assistance and training experts, and lamented the fact that these external consultants could not be retained or engaged to continue in order to enhance the rate of uptake of the new knowledge by their respective economies. This high praise ran the gamut of technical sectors of work that are inextricably linked to international trade and investment, and regional and global integration. Examples of those sectors that resonated with many APEC interlocutors of developing member economies included, EoDB, microfinance with an emphasis on women entrepreneurs, improved transparency in its effectiveness in the rulemaking process, training in the development of individual Action Plans for monitoring progress on structural reform and other measures for reaching Bogor Goals, low emission development strategies, advancing good corporate governance, assessing and improving agricultural data collection and dissemination for enhancing food security, training programs on improving project quality, public-private financing of infrastructure, green buildings and green growth. Similarly, in their responses to questions directed at the quality of work performed by the TATF permanent staff, stakeholders were uniformly positive. U.S. Government stakeholders were highly appreciative of the extraordinary surge services provided by Nathan headquarters and the Nathan field team in Singapore during the United States APEC year 2011 and thereafter, to rapidly launch implementation start-up actions on the decisions that emerged therefrom. Nathan headquarters has been highly supportive of its team in the field, providing energetic and engaged permanent staff at headquarters to liaise with U.S. Government stakeholders and for temporary staff supplementation as required in the field. And as for the permanent staff in the Secretariat, the comments from officials working most closely with TATF in the Secretariat referring to quality of the work done by the Chief of Party and TATF team were of the highest order. 24 While the “contractor” (DNG) was composed formally of a joint venture between Development Alternatives, Inc., and Nathan Associates, in practice all oversight from the center originated at Nathan headquarters in Arlington, Virginia, and all permanent staff assigned to TATF/Singapore were employees of Nathan. By agreement between DAI and Nathan Associates, DAI’s contributions to the project were limited to short-term technical expertise in the environment, green growth, and training in the development of quality project proposals – areas where DAI’s pool of technical expertise was superior to Nathan’s. 27  The TATF platform staff was commended for their technical expertise, skillfulness in working as problem-solvers, and their transparency, accessibility, and reliability.  TATF staff were highly valued for being accessible and responsive, and in accordance with the lean structure and financing of the APEC Secretariat, were correctly understood to be the only technical source and intellectual partner for responding to certain training and TA services of APEC’s diverse Secretariat staff. They were recognized as gap-fillers, both at their own and at others’ initiatives. Except when their role was to engage in institutional procedures and structures, TATF was highly credited for operating in a non-bureaucratic and results-oriented style, as distinct from some other Secretariat operations.  Leadership of APEC committees and working groups, and member economies’ participants and proponents of projects serviced by APEC, consistently expressed appreciation for the professional services of the TATF team and consultants. The vast majority cited the financial and technical resources that they received from TATF that made their projects feasible. And all were equally complimentary of TATF’s exceptional performance in arranging the venues, designing and supervising more than forty Bogor Goals case studies, diagnostics and assessments, organizing the agendas and continuity of workshops, and in trying to ensure that their training and conferencing was action￾oriented, and geared to stimulate pragmatic economic reform and change in consonance with the Bogor Goals. Other donor member economies that teamed with TATF (as the U.S. Government’s operational partner) including Australia, Canada, Korea, Japan, and New Zealand, reported a high degree of satisfaction in the cooperation they received, including joint-sponsorship activities based in the APEC Secretariat and in the member countries.  Their qualifications/expertise in areas of trade, as trainers, and as development change agents are widely respected by colleagues in the Secretariat and in ministries of member economies.  Interview responses were replete with remarks citing instances where TATF personnel were called upon to contribute to, or even in one case to oversee a major overhaul of official APEC reports. These actions went beyond the formal agenda of services normally provided or expected, were offered collegially and routinely, and demonstrated the degree of trust and commitment, and shared values exhibited among staffs of the APEC Secretariat and the TATF platform.  The PDs credit TATF with highly useful training for themselves and for the leaders of the various committees and fora they support in areas such as medium-term strategic planning and team-building.  Overall improvements in success rates in APEC’s project approval process – especially for developing member economies – are attributed by PDs to TATF training of project proponents and of TATF-designed proposal process improvements.  PDs that the team interviewed all supplied examples of the various ways that TATF had added value to, or facilitated, APEC’s efforts, through deployment of experts performing at the state-of-the-art level and generally functioning as an in-house consultancy.  GoVN Ministry of Industry and Trade representatives expressed great enthusiasm and appreciation for the many forms of assistance they had received from TATF, not least the training in quality proposal writing that TATF had offered in conjunction with SOMs and Economic Committee meetings.  Such capacity-building efforts have led to greater success for Vietnam in its concept paper-writing and project proposal submissions to APEC’s Project Management Unit, and Budget and Management Committee. Occasionally, TATF would contribute to the 28 financing of lower-level Vietnamese officials to attend these meetings in order to receive this proposal-writing training.  One notable negative comment described as less successful TATF’s attempt to serve as alchemists to design an APEC personnel appraisal system for PDs under circumstances where PDs’ chain of command runs back to their home economy, not to executive structures in the APEC Secretariat.  All confirmed that TATF is accountable and responsive, and performs their assistance duties as articulated in the IQC Task Order.  TATF has worked well combining their own staff and externally recruited experts with those provided by U.S. Government agencies to address the full range of development issues associated with regional economic integration.  The overall TATF performance came in for slight criticism on the rare occasion that a consultant allegedly turned in a poor performance or product at a workshop or training event. For example, some clients from member economies expressed disappointment in TATF when it held the “train the trainer workshop on medium-term strategic planning of Bogor Goals’ subject areas,” stating that the planning techniques employed were too sophisticated and perhaps of little utility to APEC’s developing member economies’ end users, especially PNG, Vietnam, the Philippines and Indonesia. Also, more negative feedback was expressed to describe a situation where the TATF staff was engaged to help in drafting a concept note or project proposal that did not make the cut, or required several submissions for approval. Many officials, including APEC’s Executive Director, commented that TATF staff comportment as implementers of U.S. Government policy priorities in support of APEC invariably was interpreted as a plus for the United States. Representatives of other APEC “donor” economies working in the Secretariat were consistent in praising TATF staff cooperation in areas of common pursuit. Rather than reinforcing a view of TATF as an intrusive, possibly manipulative, over-reach of one well-healed member government in the Secretariat, the enthusiasm and professionalism of the TATF staff won admiration and plaudits for the U.S. Government and its demonstrated commitment to APEC’s goals. Summary note on differentiating TATF platform impacts from TATF personnel impacts. As regards contributions by TATF to APEC’s effectiveness, it is difficult if not impossible to quantify with precision, the balance between quality of structure in relation to quality of the contractor and the contractor’s staff. That said, observations, enquiries, stakeholder feedback, and reviews of subject matter presentations suggest unequivocally that TATF’s and APEC’s overall quality of performance would not have been achievable absent the strong professional interaction and shared interests between a well-conceived and intellectually strong and nimble TATF platform and a highly effective contractor oversight supported by well-qualified, enthusiastic and energetic human resources, especially on the TATF long-term team itself. Remove or materially reduce the quality of either of these elements, and the TATF platform – as a source for diverse and responsive state-of-the-art technical assistance and training inputs – could not have been as effective a partner for APEC as it clearly has been reported to be. QUESTION 2: TO WHAT EXTENT DID THE SERVICES PROVIDED BY THE TATF FURTHER INTERNATIONAL TRADE AND INVESTMENT AND ASSIST REGIONAL INTEGRATION? Notwithstanding its twenty-three years of advocacy for REI, APEC does not claim credit for the significant progress that its member economies have voluntarily achieved towards reaching 29 Bogor Goals. However, APEC as an institution does not deny playing a meaningful role in advocating and facilitating incremental changes toward achieving REI within the Asia-Pacific region. Similarly, in the case of the U.S. Government-supported TATF, and especially in light of its relatively short time span of four and-a-half years of delivering focused assistance for accelerating progress in reaching Bogor goals, it would be hyperbolic to ascribe development outcomes solely to the TATF platform and its exceptional performance. Nonetheless, results of evaluation interviews and survey responses showed convincingly that TATF has been instrumental in helping APEC to accelerate the pace of regional integration and international trade, and business development in the Asia Pacific region, particularly as a strategic approach for reducing the development gap among the member economies. There is little question that the diversity and richness of the APEC subject matter addressed in the wide array of seminars, workshops and training events has led to improving institutional performance, procedures, business and ethical practices, corporate governance and transparency, legal and regulatory frameworks, policy development, economic and structural reform, etc., both at sectoral and at whole-of-government levels. As such, all of it is developmental, holding promise for high impacts and benefits to all of APEC’s member economies including an economically stable, peaceful, and secure Asia-Pacific region. Illustrative Interview Results  Respondents were effusive in linking the successes of such initiatives as tariff reductions, and improvements in microfinance, customs procedures, and transparency in rulemaking, competitive procurement processes, ease of doing business, and policy analysis to TATF’s quality technical assistance, analysis, assessments, diagnostics, training, and facilitation.  TATF experts worked with CIEM on the Vietnam case study, reportedly tapping comprehensively into the large Vietnam micro-finance policy and practitioner network to obtain diverse inputs on all forms of small-scale lending including women entrepreneurs; assisted in drafting the case study; and coordinated the workshop agenda and speaker presentations.  As a result of having key GoVN APEC public sector employees in attendance at the various SOMs, technical workshops, and related events organized and implemented by the TATF, subsequent presentations by these participants to the Prime Minister expedited implementation of the recommendations that emerged at these events. Other senior officials facilitated the issuance of a government decree, and passage of a number of new directives and laws for regulating the micro-finance environment and defining the roles of all key players, including the Ministry of Finance, the State Bank, key NGOs and other micro-finance intermediaries.  While direct attribution for these outcomes clearly could not be ascribed to TATF’s involvement, interviewees expressed confidence that the quality of inputs, and the workshop itself, were seminal factors leading to the chain of reforms that eventually were implemented, and resulted in a widening and deepening of financial intermediation in the microfinance sector and improved access to credit by marginalized borrowers.  Substantively, these officials credited TATF assistance in organizing Vietnam’s case study for the public consultation process workshop with encouraging a number of formerly non-participating ministries to establish space on their websites to publish consultative drafts ahead of promulgating new laws, regulations and directives. The Ministry of Industry and Trade, itself, adopted procedures for widespread consultations with other government entities, businesses, academics, civil society organizations, and NGOs to 30 formulate more transparent and better-informed positions on a number of trade agreements.  Similarly, the team’s interlocutors saw the 2011 Assessing Progress on Structural Reform action planning workshop in San Francisco as usefully outcome-oriented. Vietnam attendees credited the small working group and peer review processes facilitated by TATF as helping greatly in devising performance indicators useful for measuring real results rather than mere statistical data gathering.  They also claimed to have gained a greater appreciation for how best practices as might be agreed at such meetings often come into conflict with preferences of domestic policymakers in home governmental structures. In the case of Vietnam, it was interesting to learn how such conflicts – particularly concerning trade – could be influenced by interventions of experts under such USAID bilateral programs as Support for Trade Acceleration in Vietnam (STAR), and Vietnam National Competitiveness Index (VNCI), which on occasion historically had taken place. Survey Results Following are results gleaned from the sample survey supporting the conclusion that TATF’s efforts are contributing to attainment of Bogor Goals:  The overall response to the on-line survey question, “please rate how helpful this facilitated workshop was in helping you to work with colleagues and other organizations in your economy to achieve reform,” led to an overwhelming 96.9 percent of respondents reporting that they found their respective facilitated workshop very helpful or somewhat helpful. As would be expected, response rates were considerably higher for workshops held in 2011 and 2012 than they were for those held in 2009 and 2010, for which period, 98.6 percent of respondents claimed to be satisfied or very satisfied with the workshop they attended.  The link between the TATF-sponsored assessments on Environmental Goods and Services (EGS) and the voluntary agreement by APEC member economies reached at the 2012 Leaders’ Summit to lower tariffs on 54 EGS-related trading items to 5% or less by 2015 is another concrete example of how TATF can have impact. In this case, the TATF’s assessments on EGS formed part of the empirical basis for APEC, the U.S. Government inter-agency representatives, and APEC Canadian delegates to cooperate in advancing the regional integration agenda, and to narrowing the development gap between the developed and developing member economies while mitigating global climate change.  TATF cooperation with the Vietnamese Ministry of Finance and Customs was instrumental in securing GoVN buy-in into the agreement to exempt payment of excise tax and duties on international shipments of goods whose declared value is USD 200 or less, compared to USD 50 previously.  The TATF-provided training and assessments on the feasibility of tariff adjustments for remanufactured goods is spurring cross-border trade, spawning cottage industries, increasing regional trade, and expanding employment opportunities, while addressing environmental concerns – all laudable development outcomes.  All commended the technical and cooperation skills of the core TATF staff and of the extended roster of consultants that the DNG consortium makes available at all project stages, including for execution of assessments, policy studies, preparing concept papers 31 and quality project proposals, and participating in meetings as speakers, experts and facilitators. Similarly, insofar as TATF’s capacity-building activities focus on building the core strengths of APEC as an organization – an organization whose goals are expressed as the Bogor Goals of regional economic integration and reducing development disparities among member economies – there can be little that TATF expends U.S. Government resources on that cannot be characterized as truly developmental.  And though the team began the evaluation neutral on the development question, and in particular on the question of whether whatever development could be measured would be merely as outputs – e.g., number of training workshops held, number of trainees/participants, amounts expended on technical assistance, studies, assessments and consultant contracts – there is now adequate data based on the interviews and survey findings to support the conclusion that the development outcomes attributable to TATF’s efforts have been measurable and robust.  Responses included descriptions of such outcomes as reform initiatives that trainees actually had participated in, best practices they had been successful in negotiating with others to install, establishment of monitoring and evaluation systems to track impacts, all of which they judged were attributable to their participation in TATF-facilitated workshops focused on transferring exactly such technical skills and capabilities. This conclusion is further informed by the many and diverse commitments announced at Leaders’ Meetings in recent years that result at least in part from TATF-facilitated studies and workshops, and the number of attestations from member economy project partners on consensus and reform achievements in the whole range of subject matter areas that TATF events have contributed to. Specifically, these survey responses of participants at TATF events attested to capabilities they acquired that include engaging in processes leading to such outcomes as tariff adjustments, adopting state-of-the-art micro-finance technologies and practices including enhanced women’s access to lending capital, remanufactured goods, reductions in barriers to doing business, improved business competitiveness, consensus-building around regulations, improved disaster planning and disaster resilience, climate change mitigation, and commitments to more active monitoring and evaluation. The survey data yielded a few other findings related to the development question that merit discussion. For analytic purposes, the respondents sample was divided into two subgroups – those from developed member economies and those from developing economies. While it was possible there might be some differences, such as in the sense of being helped in the acquisition of development-outcome-useful skills, in fact few material differences were found. For example, in survey question 3, “to what extent did what you learned in this workshop facilitate your understanding and/or effectiveness in helping your economy to achieve the [three] APEC Bogor Goals,” the proportions of respondents who answered “helped a lot” or “helped somewhat” were not significantly different as between developed and developing groups. This held true across the three categories of Bogor Goals. Thus, the development outcomes and potential impact of the TATF efforts are even larger than if they were to be measured only with respect to the developing APEC member economies. 32 TRADE AND INVESTMENT LIBERALIZATION BUSINESS FACILITATION 33 ECONOMIC AND TECHNICAL COOPERATION It also was demonstrated by the free-form responses to question 7, “please describe the areas and the nature of the reform or technical skills progress that you feel you achieved by your participation in this workshop and related activities stimulated by this workshop,” that both groups expressed their skill acquisitions and achievements in similar terms, thereby demonstrating that development outcomes, at least as revealed in this format, were equally reported by developed and developing economy participants. See the appendix for a summary of these comments. Gender Considerations Derived From the Evaluation Survey There is ample evidence that gender integration and sensitivity to potential gender imbalances are taken into consideration in the planning and execution of the TATF platform’s activities. Nevertheless, some degree of gender imbalance appears evident when the survey data are viewed in aggregate. To illustrate:  The survey results were broken down by male and female to examine whether any systematic differences in gender could be identified with regard to utility, satisfaction, recognition of TATF facilitation or Bogor Goal progress, etc. The gender distribution of survey respondents was 55 percent men, and 45 percent women. The gender breakdown of persons initially surveyed had been 56 percent male, and 44 percent female.  Instances were observed, however, of an overwhelming gender imbalance in favor of females at workshops on microfinance lending, MSE development, public-private partnerships, and public participation in the rulemaking process.  Apart from this small imbalance in the number of respondents, in side-by-side comparisons there were no apparent systematic differences in the way women and men responded to the questions. More generally, to the extent women’s participation is deemed a healthy developmental indicator, the majority of persons interviewed in the APEC Secretariat were female, as were the majority of persons interviewed in the 34 member economies, suggesting that based on direct contact, benefits of TATF activities were made available to more women than men. Also, it should be noted that the TATF project staff in Singapore as well as the Nathan Associates home office support staff were entirely female. In summary, TATF activities directly contribute to measurable progress in implementing APEC priorities for attaining the Bogor Goals, and complement ongoing work by APEC and other donors. Specific activities are developed in close consultation with the APEC Secretariat, member economy representatives, the Department of State, and other U.S. Government agencies engaged in APEC policy support. In addition to the typical benefits to trade growth, tangible benefits are delivered to citizens in the APEC region through increased training and employment opportunities, greater choices in the marketplace, the availability of competitively priced goods and services, and improved access to international markets. QUESTION 3: WHAT ARE THE MOST IMPORTANT AREAS OF FOCUS FOR U.S. ASSISTANCE IN THE FUTURE TO ACHIEVE THE GREATEST IMPACT GIVEN LIMITED BUDGETS AND APEC CAPACITY? Is there a good reason for the U.S. Government to continue participating as a donor “co￾dependent” in an operation that is not now, nor likely in the medium term to become, a financially sustainable operation? Based on the numerous discussions and interviews held, it is clear that, in substantial measure due to the embedded presence of the TATF and its personnel in the APEC Secretariat, the U.S. Government is regarded as a highly valued partner in the Asia￾Pacific region’s continuing economic growth. The United States recognizes that it has compelling economic and national interests in Asia-Pacific economic integration and in the trade and investment and business development environments for which APEC is a gateway.24 While other options exist for enhancing APEC’s institutional capabilities while at the same time servicing the cooperation needs of its U.S. Government stakeholders, the U.S. Government’s support for APEC through a TATF-like structure not only makes many necessary contributions to APEC operations, but gains the U.S. Government respect and credibility as a partner and counterpart. The U.S. Government’s engagement with APEC has shown that influence stems less from deploying power, or from simply providing budgetary resources directly or indirectly, and more from being accepted as a resident part of the institutional foundation supporting the voluntarism and commitment to consensus that define the APEC organization. APEC remains one of the United States’ primary vehicles for advancing both economic cooperation and trade and investment liberalization in the Asia-Pacific region, and the United States therefore should opt for the most robust channel for supporting that national interest. Looking over the horizon, it is expected that implementation of many key actions that emerged from the 2011 and 2012 Leaders Meetings will be continued by the TATF in concert with various APEC Secretariat entities and member economies that have contributed regularly to accelerating the pace of structural reforms for achieving APEC’s vision of regional economic integration within the Asia-Pacific region. The repertoire of rich and diverse experience garnered over the last four years suggests that a possible APEC U.S.-supported follow-on project would be seen by APEC member economies and the Secretariat as a welcome, appropriate, and timely affirmation of the United States’ interest and commitment to a prosperous and peaceful Asia-Pacific region. The possible new support to APEC should seek to 24 President Obama’s Closing News Conference at APEC Hawaii. The White House Office of the Press Secretary, November13, 2011. 35 build on past successes and lessons learned, while responding with continuing flexibility to rapidly emerging opportunities, especially as they relate to:  Implementation of protocols and policy frameworks for free and open trade and investments.  Assisting developing member economies to strengthen their capability to monitor progress towards reaching Bogor Goals, and to improve their ability to carry out actions in support of their voluntarily pledges to embrace select reforms because it is in their best economic interest to do so.  Continuing and new opportunities requiring U.S. technical and training support that will undoubtedly stem from the Indonesia-hosted APEC Leaders’ Meeting to be held in Nusa Dua, Bali, at the end of 2013. Specific program areas are articulated in the following Chapter, including both PMP category 1 and category 2 types of institutional strengthening activities, which have proven so effective in enhancing the capacity of the APEC Secretariat to be a catalyst for transforming APEC and its member economies into the world’s fastest growing trading bloc. 36 V. RECOMMENDATIONS The team’s enquiries, the preceding analyses, and the findings and conclusions in this report suggest a number of ways forward for consolidating gains over the past four years, improving the performance of the current APEC TATF in its final operational year, and more importantly, for informing directions in possible future U.S. Government support to APEC. While policy decisions and appropriations legislation for upcoming years have yet to be finalized, there is a high likelihood that the U.S. Government will wish to continue not just diplomatic and security cooperation as foundations for its larger Asia-Pacific re-balancing strategy, but also provide adjunct support for the performance and development of supportive dynamic regional organizations such as ASEAN, TPP, and certainly APEC. In the cases of ASEAN and APEC, USAID has for many years served as the State Department’s partner in administering this institutional-strengthening activity, primarily for two reasons: 1) USAID’s unique authorities and experience in administering such support projects make it uniquely fit to play this complementary implementation oversight and accountability role in conjunction with State’s policy leadership and oversight; and 2) the organizational and institutional developmental objectives that are at the heart of the assistance traditionally provided to the ASEAN and APEC Secretariats. Also, importantly from USAID’s institutional point of view, the clear contributions of these bodies to development outcomes in the region – human and institutional capacity-building, economic integration, environmental and health improvements, reductions in regional development disparities, etc. – all are consistent with the core objectives of USAID as the U.S. Government’s principal development assistance agency. As a capstone to this TATF mid-term contractor performance evaluation, this chapter will offer a number of recommendations in the areas of 1) U.S. Government agencies’ cooperation in management and oversight, and 2) project content, goals and objectives, that are intended to inform adjustments in current project operations and to inform the design of a possible new APEC assistance project. A. RECOMMENDATIONS FOR USAID OVERSIGHT This sub-section contains five recommendations of current approaches and practices that merit keeping, and changes that USAID and other U.S. Government inter-agency partners can consider taking that in the short-run can improve performance and outcomes in the remaining operational life of the current APEC assistance project. In addition to their potential utility in the current project, all can be carried forward equally as design considerations for a possible future APEC assistance project. In short, these first five recommendations are: 1. USAID and State should streamline and formalize the work planning process. 2. USAID needs to be better represented at the Washington-based inter-agency table. 3. RDMA should take immediate steps to ensure quality control in TATF through more regular and more intensive oversight. 4. TATF should inform Embassies and USAID Missions of activities relevant to their project, reporting or diplomatic efforts. 37 5. TATF, RDMA and State EAP/EP should install a real-time calendar and dedicated messaging system to ensure up-to-date information sharing. Recommendation 1: State and USAID should move to ensure completion and optimization of the current final-year TATF work plan, by an improved process that leads to more certain and timely outcomes and which also can serve as a model for annual work planning in any possible follow-on APEC support project. Context: As occasionally noted in the preceding chapters, many stakeholders in TATF’s work – including the U.S. Government’s inter-agency group, APEC Secretariat constituencies and the extended community of TATF clients and partners in the member economies – have expressed frustration with the uncertainty that has surrounded the decision-making process that establishes TATF’s annual work plan. As a result, commitments to projects with long lead-times may experience delays, lock-in of consultants and venues becomes problematic, individual participants’ time- and travel-planning cannot be confirmed, and as a result, the ability of all these stakeholders –notably including TATF – to achieve their near-term and even long-term objectives can be hindered. To be clear, for the most part, this process works. But adherence to timetables should be improved, in order to provide more certainty for those dependent on it to make downstream decisions. Recommendation: Driven by the urgency of the project entering its final year and the shared objective of maximizing attainment of all participants’ shared goals, as early in 2013 as possible, State, USAID and TATF should convene to bring closure to the project’s year-five work plan, not just to ensure optimum outcomes during this final year, but to conceptualize a possible prototype for an improved and more predictable process that can be carried forward for consideration and incorporation into a possible follow-on TATF-like APEC assistance project Recommendation 2: USAID’s Asia Bureau, Washington, must be more present and more assertive in Washington-based APEC inter-agency processes. Context: This issue is in part a corollary to the preceding issue. While in most respects, a sensible division of labor and responsibility among USAID, State and the other U.S. Government participants in the inter-agency planning and governing process exists, it was clear that some of these stakeholders were unaware of USAID’s role in implementing this assistance to APEC, were not sensitive to the regulations that govern USAID’s administration of the TATF contract (and to some extent were unaware of the project’s development objectives), and as a result the shared knowledge that might optimize outputs and outcomes was lacking. Considering the infrequent attendance of USAID at the inter-agency table for most of the project’s life, it is unsurprising (though surely not helpful) that this shared awareness was lacking. Recommendation: Bearing in mind that USAID managers of the APEC support project are located predominantly in the field near to its implementation sites, while the rest of the whole￾of-government players are based primarily in Washington, USAID’s Asia Bureau and RDMA should agree no later than early 2013 on an arrangement under which a senior manager in the Asia Bureau is empowered as a member of the RDMA APEC TATF oversight team and delegated authority (and responsibility) to engage fully in inter-agency APEC TATF meetings and related processes and thereby strengthen the presence of, and more fully represent, USAID in these Washington-based councils. By taking this action immediately, better-shared understanding will result and contribute to final-year performance. Also, USAID will gain itself a stronger voice as the State Department’s partner in generating consensus in the inter-agency group around any new APEC support project design, and will thereby ensure that its 38 administrative expertise and the development dimension of the assistance are understood and established at the outset. Recommendation 3: RDMA should take immediate steps to ensure quality control in TATF through more regular and more intensive oversight. Context: In interviews in Singapore with more than 50% of the APEC Secretariat’s staff, and with member economies’ representatives, a very small percentage of interviewees questioned whether there was adequate U.S. Government oversight and APEC institutional ownership in some of the capacity-building activities that have been carried out at the behest of the APEC Secretariat’s leadership. For example, a question was raised whether USAID managers had sufficient insight into such activities as development of the project performance and highly technical data tracking and on-line searchable data-base systems under construction with TATF’s support, and whether they were producing systems that will be used and will be sustained over the long-term by the regular budget after TATF design support comes to an end. Although no instances were identified in which TATF’s efforts, resources, and work planning were deficient, duplicative, or misdirected, it is clear that there are potential vulnerabilities to TATF’s effectiveness, which, if they led to flawed outputs or outcomes, would potentially reflect poorly on the U.S. Government and the quality of its allocation of scarce resources. RDMA staff contributes to the technical dimensions of the program. However, these contributions are thin and staff is subject to frequent turn-over. RDMA should prioritize increased level-of-effort in order to ensure critical quality control over project implementation. During the life of the project, due to office turnover, the COR has changed five times between four individual officers. Currently, because of other work assignments, APEC TATF implementation has not had a dedicated, full-time COR overseeing the project. The substantial amount of continuing and new TATF activities, the need to carry out USAID standard program management and program performance reporting requirements, the uniqueness of the TATF with respect to its complex work plan development process, the many TATF stakeholders and broad spectrum of program operations, and the region-wide implementation architecture, call for an extraordinary amount of real-time communications initiated by the COR, to the diverse stakeholders, partners and clients of TATF. As well, the significant program resources allocated to convening training workshops, SOMs, and assessments would, in an ideal world, require the attendance by the COR at a minimal number of these events in order to glean insights on the relevance and effectiveness of the events, assess whether they optimally utilize U.S. Government resources, and to gather information on lessons learned. Recommendation: Given the high profile of the United States assistance to APEC, the deep and rich diversity that characterizes the TATF platform’s services and the need to adhere to USAID’s accountability requirements, steps should be taken immediately to ensure greater overall management oversight of TATF. This supplementation will lead to regular periodic, in￾depth internal reviews, and regular site visits to assess the continuing appropriateness of long￾gestation policy projects, and the relevance of new requests emanating from the APEC Secretariat for advancing the objective and goals of U.S. Government support. Also, site visits to developing member economies, especially the power users of the TATF platform services, should be considered. Implementation of this recommendation could be facilitated by recruiting additional host country professional program management personnel who are familiar with economic integration principles, policies, reform protocols, and USAID rules and operating 39 systems. These actions will also minimize the potential for vulnerabilities to the U.S. Government. Recommendation 4: TATF should inform Embassies and USAID Missions of activities relevant to their project, reporting or diplomatic efforts. Context: Face-to-face interviews with key APEC interlocutors in developing member economies’ capitals indicated that TATF’s efforts have a greater chance to endure and produce fruits if certain downstream support functions can be negotiated to be taken on by U.S. Government agencies’ managers that already operate in those member economies. While TATF does actively “push” information to various stakeholders on their activities and on lessons learned in the course of carrying out studies and assessments – many being project partners with whom TATF has had long-standing and repeat involvements – there have been a number of missed opportunities to systematically put timely and more-targeted APEC-related information that might mobilize action into the hands of other potential beneficiaries. For example, in both Indonesia and Vietnam, two developing member economies also served by USAID bilateral Missions, discussions with key TATF seminar and workshop partners described situations where the member economy proponents had become frustrated that the highly valuable products that had resulted from TATF-facilitated projects were not able to be pushed through member economy bureaucracies to obtain full reform impact or other fruitful outcomes. At the same time, discussions with counterparts in the USAID Missions revealed a lack of awareness that the USAID-supported TATF had generated a number of outputs that might be accelerated in achieving their objectives if they had the support of other policy players whose interventions might be able to relieve logjams in consensus-building or actions on the TATF-facilitated rest practices or reform initiatives. Specifically, had USAID managers of portfolios operating in the same sectors known of these TATF activities – activities not engaging known USAID host country counterparts – they, and/or the Mission’s highly effective technical assistance partners (such as SEADI in Indonesia, or Vietnam National Competitiveness Index Project in Hanoi, both of which possess considerable flexibility for ad hoc interventions) could have been teed up to provide important leveraged support. Similarly, discussions with other Embassy elements such as POL/ECON, Commercial Service, FAS – and not only those located in USAID countries in the APEC region – confirmed that many Embassy officers have diplomatic or other interests and responsibilities closely aligned with areas of TATF cooperation in their host economy. Most, of course, knew generally about APEC and/or routinely monitored or reported on APEC from their respective host countries’ point of view. Yet few of the officers that were interviewed claimed to receive timely or regular reports from TATF on activities with material implications for their country. All these officers acknowledged that such information offered potential benefits to them in doing their jobs better. A few asked the evaluation team to convey a request to TATF for heads-up notifications on activities in their countries in which TATF was engaged. Recommendation: USAID should authorize TATF resources to be expended for the purpose of targeted or broadcast communications concerning their substantive activities involving member economies to be “pushed” to USAIDs and U.S. Embassies located in all APEC economies. Recommendation 5: TATF, RDMA and State EAP/EP staff should install a real-time calendar and dedicated messaging system to ensure up-to-date information sharing. 40 Context: In the project management process, there are robust two-way flows of approval requests, advice on activity modifications, extraordinary requests for clarification, and routine coordination memoranda. While for the most part TATF complies in a timely fashion with the scheduled reporting requirements and responds well to irregular requests amidst an otherwise highly pressured agenda of daily action items, USAID and TATF managers admitted to some shortcomings in real-time information flows that have material effects on the quality of cooperation among USAID and TATF and occasionally effect State Department and U.S. Government inter-agency interests. Non-notification of delays in implementation or re￾programming of near-term scheduled events, abrupt changes in personnel arrivals, unexpected requests from partners requiring extraordinary TATF team support, for example, can be disruptive. There are new, simple and reliable ICT tools available – e.g., Google Calendar – that could substantially reduce such stress and improve real-time communications, which the key actors have agreed ought to be installed immediately. Recommendation: TATF, USAID, and State/EAP/EP should begin immediate planning to install a real-time calendar and dedicated messaging system to ensure up-to-date information sharing. B. RECOMMENDATIONS FOR POSSIBLE FUTURE U.S. FOREIGN ASSISTANCE TO APEC The following nine recommendations apply to design options that are judged to be important considerations while vetting the structure and content of a possible new assistance project for APEC, growing out of the lessons learned during the contractor performance evaluation of the TATF experience, four years into its life-of-project. Recommendations 6 through 14 are summarized as follows: 6. The TATF concept as an embedded technical assistance and training platform should be a starting point for consideration of a follow-on project. 7. The profile of monitoring and evaluation should be elevated in any follow-on project. 8. A higher priority should be given in any follow-on project to down-stream implementation and institutionalization of reforms in member economies. 9. Focus should be on incorporating ways to initiate and support innovative high-pay￾out policies and practices that enhance REI in a developmentally sound way. 10. U.S. Government resources in APEC should be better leveraged by establishing more conscious and systematic APEC donor coordination. 11. Consider shifting project balance from capacity-building activities toward consolidating APEC efforts that directly support REI and reduce the development gap. 12. Concretize greater technical cooperation between APEC and ASEAN. 13. Balance sustainability considerations of APEC support with a clear-eyed calculus as to benefits to United States national interests. 14. More flexible procedures should be devised in order to expeditiously accommodate short lead-time contractor work-load changes and skill-mix requirements. 41 Recommendation 6: TATF’s embedded presence, its mode of operations, and the quality and flexibility of its staff are effective, good for APEC, good for the United States, and therefore the TATF “concept” should be a starting point for consideration of a follow-on project. Context: Many assistance models – from simple cash transfers without a technical presence, to short-term externally supervised episodic supervised level-of-effort task orders, to reprogramming ESF resources directly from the State Department to the other inter-agency stakeholders – can be conceptualized to attempt to do what TATF has done for APEC. The evidence of this evaluation, however, has demonstrated that as a platform embedded within and co-located at the APEC Secretariat in Singapore, both the concept of TATF, and the TATF staff, have received approbation from APEC’s diverse stakeholders and clients alike. TATF’s staff have earned such high respect for the technical prowess and knowledge, and operational effectiveness they have demonstrated as the TATF responded to the gamut of requests aimed at: 1) strengthening the APEC Secretariat’s ability to effectively carry out its day-to-day functions and communicate with its member economies; 2) elevating to a much higher profile the intersection of quality policy-based project proposals and an incisive understanding by APEC developing member economies of the critical role of deep structural reform in the processes leading to regional economic integration; 3) coordinating effectively with the U.S. Government inter-agency working group to incorporate United States Foreign Policy priorities into APEC’s annual work plan Agenda; and 4) smoothly informing senior level decision-making processes of APEC’s lesser-developed member economies on reform measures and their implementation – so critical to accelerating the pace of REI in the Asia-Pacific region. This is no mean record of achievement. One inter-agency stakeholder remarked that as the competitively selected implementing partner for the U.S. Government’s priorities and national interests within APEC, the TATF is very flexible, very solutions-oriented, is cost-effective, and very supportive of the United States Government’sstrategy and vision. This significant contribution was similarly recognized by the APEC Executive Director when he mentioned that the United States, through the TATF, “adds value to APEC operationsbecause of its experts’ on-site presence, especially during frequent brainstorming sessions with project overseers and working groups, where ideas for moving the agenda forward to achieving the Bogor Goals are born and developed into projects.” Further, TATF’s demonstrated strengths at collaborating with other developed donor member economies elicited the following comment from a senior-most member of the APEC Secretariat’s Project Management Unit: “It is good to have TATF here at the Secretariat. Now, the review and approval process for concept notes and project proposals, although more stringent, tends to be smooth as project proposals are of much higher quality than before, and demand now outstrips funding availability.” Another long-term project director, eminently familiar with TATF’s genesis and performance commented: “TATF sends out a strong message that the United States quietly and with dignity has been helping APEC, while advancing U.S. national security interests.” Recommendation: The structure of the TATF and the excellent relationship its staff has built with the APEC Secretariat, stakeholders, and clients should be nurtured, unchanged, in the final year of project operations. Not embedding such a resource, in its current form, at APEC headquarters, will result in a loss of effectiveness. Also, under any envisaged follow-on support project for APEC, the staff composition of TATF should be examined to ensure correspondence between the mix of services demanded and the set of staff skills to be provided (in-house and through short-term consultant availabilities) that will lead to continued high performance levels 42 and adequate technical capacity to respond in a timely fashion to a potential surge in demand for REI structural reform, analytical, training, and implementation advisory services. Recommendation 7: Consideration should be given in a possible follow-on project to narrowing and deepening program focus to maximize benefits for APEC and the United States that will be measurable through an active M&E effort. Context: In connection with working in diverse institution-building, policy and program areas, the APEC TATF has chalked up numerous achievements that contribute to the Bogor Goals. TATF program activities are mainly demand-driven by decisions that emerge from the annual Leaders’ Meetings. As pointed out earlier, activities range from specialized workshops where renowned speakers present state-of-the-art information on trade and investment issues, to reform-centered diagnostics and assessments for uniquely informing member economies’ decisions on select REI issues. Such outputs may be voluntarily embraced and implemented because it is in the economies’ interest to do so; they may spawn new ideas for narrowing the development divide between the richer and poorer member economies; and they may facilitate an understanding by member economies of how case studies’ findings can benefit their economies. While the numerous achievements of TATF are indisputable, and benefits clearly discernible, there is a dearth of available quantitative baseline data and information related to program outcomes. Only in 2012 have resources been tapped to institutionally move forward with ascertaining contractor performance, strengths and weaknesses. Further, with the new emphasis in APEC on multi-year projects for ensuring greater program coherence, continuity, and development impacts, and given APEC’s objective of taking stock in 2013 of progress by developing member economies toward delivering on their voluntary commitments to reach critical components of the Bogor Goals, the stage has been set for narrowing and deepening the TATF strategic program focus in 2013. This would allow for concentration of program resources, which, in turn would facilitate more accurate tracking and reporting on the input￾output-outcome continuum, and attest to the validity of the underlying development hypothesis. Recommendation: Any possible future APEC assistance project should, with sufficient technical clarity and in consonance with USAID policy, carefully weigh the options for narrowing and deepening program focus during the design and approval of new activities. Such an analysis should incorporate the elements of a set of robust selection criteria that will give priority to focusing the resulting program on the U.S. Government’s topmost development objectives for APEC, and that will establish a robust, pragmatic and viable monitoring and evaluation system, that effectively guides the data collection efforts of the assistance contractor, and complies with USAID’s policy requirements on this subject. Recommendation 8: A higher priority should be given in any follow-on project to down-stream implementation and institutionalization of reforms in member economies. Context: Given the historical agreement that emerged at the 2012 Russia-hosted Leaders’ Meeting in Vladivostok concerning the new tariff regime for environmental goods and services by 2015 – an agreement that still eludes the WTO for countries outside of APEC – the rationale is even stronger today for focusing TATF efforts on downstream implementation steps. This evaluation has reported on the frequent absence of follow-on action after such agreements are struck. Consideration should be given to incorporating in any follow-on APEC assistance project an intensified effort to institutionalize the reforms by helping to accelerate the 43 consensus-building process leading to formal implementation of trade policy agreements on a limited number of key issues such as those concerned with EGS, de minimus, ease of doing business, tariff nomenclature, among others. Such efforts can be particularly value-adding in the case of assisting less-developed member economies that lack sufficient own-resources for moving forward with implementation of Bogor Goals’ components. For example:  The team learned of some of the frustrations experienced in the course of enjoying TATF support, when an initiative would come to a close, leaving behind a good product and/or some strong member economy supporters prepared to take on a reform initiative, but where implementation of change could not be made to materialize. In the case of Indonesia, the team was informed of some material advances in understanding of the rule-making process and on starting a business that TATF was instrumental in developing. In the specific case of Indonesia, with so much fragmentation and competition among ministries with overlapping jurisdictions particularly on economic matters, TATF-client champions of reform often found it difficult not only to win the support of other ministries with a stake in a particular sector, but equally achieved limited success in building consensus up the line to their own ministries’ executive levels.  Conversations with long-time observers of Indonesia in the U.S. Embassy confirmed that even with further support from TATF and experts from other member economies such as Australia, New Zealand and Korea (such additional support was provided in a few cases), institutional and political forces affecting ministerial cooperation could be limiting factors on reform outcomes, no matter the quality of the support from these supplementary outside forces. Notably, however, the team formed an impression that interventions to support such reform activities could be positively influenced by systematic interventions of high-level U.S. Government officials in USAID/Indonesia and the Embassy, or through targeted project support from a well-regarded and flexible research and development support entity such as the USAID/Indonesia SEADI project working in GoI institutions but carrying out their dialogs through parallel informal channels. Recommendation: Potential follow-on assistance should include in its Program Description and Statement of Work, a sound rationale and justification for helping developing member economies, upon their request, with the development of implementation frameworks, protocols, and public outreach communications, as appropriate, and for drafting legislation to legalize implementation of the concerned policy reform, as may be required by their constitution. Recommendation 9: In a potential follow-on project, consider providing resources and encouragement for the project to initiate and opportunistically support innovative high-pay-out policies and practices that enhance REI in a developmentally sound way. Context: To the extent possible, in the process of carrying out their work plan, the TATF has ensured additional value is added by supporting ideas and project proposals originating from the developing economies. This assistance has occasionally taken the form of providing resources to those member economies that have become interested in non-traditional (“Pathfinder”) initiatives that have often matured into highly useful developmental forces, such as trade in remanufactured goods, environmental goods and services, and others. Such leadership 44 initiatives are consistent with APEC’s core objective of achieving an open regional trade and investment system that may be enhanced and developed in creative and innovative ways. These aspirations, and the sectors in which they have been applied such as in environmental trade and small-scale manufacturing, also are consistent with U.S. Government advocacy for trade and development in the Asia-Pacific region (as in others). Recommendation: Consider narrowing the range of policy-based activities to focus only on the most innovative policy and programming priorities that emerged at the 2011 and 2012 Leaders’ Meetings. And, in anticipation of developmentally sound decisions that emerge from the Indonesia-hosted summit in 2013 and the out-years, allow for the mobilization of additional program resources in a timely fashion to pursue additional work in innovative program areas. Recommendation 10: Consider establishing more conscious and systematic APEC donor coordination in the new project in order to more efficiently leverage U.S. Government resources. Context: There is a diverse array of voluntary assistance coming to APEC from developed member economies. Some of it has been coordinated among the donor economies; some support was implemented independently without regard to the operations or objectives of others. While there appeared to be little such assistance that was conflictive, a question still remained whether outcomes could have been more robust if donors had more consciously coordinated their assistance. Itself being a major component of United States assistance to APEC – and thereby an interested party – TATF itself perhaps would not have been the best mechanism for carrying out such coordination. Given the predominance of United States assistance to APEC, it might make sense for the U.S. Government to initiate independent of TATF a more conscious and systematic process of donor coordination in order to more efficiently leverage developmental outcomes. For example, such coordination might take the form of closer cooperation with the APEC Policy Support Unit, which currently works well on a parallel track with TATF, but whose rigorous economic analysis products could be more integrated into support for the training and technical assistance activities of TATF through economic analysis, impact evaluation, and sponsorship of economic integration and trade literacy components in TATF-facilitated events. Recommendation: Options should be assessed for establishing and supporting an element of a new APEC assistance project – possibly in addition to a continuing TATF-like function – whose conscious and primary objective would be to identify, promote and supervise such cooperation opportunities and other joint donor activities. Recommendation 11: Shift project balance from capacity-building activities toward consolidating APEC efforts that directly support REI and reduction of the development gap. Context:The TATF has contributed immensely to the improved efficiency and effectiveness of the day-to-day operations of APEC. In some instances, however, requests by the Secretariat’s leadership for the TATF to carry out such tasks as the development of an employee performance standards and appraisal system, and convening of annual team-building exercises for enhancing staff performance might have yielded greater outcomes had the Secretariat carried them out under another mechanism that would have fostered greater ownership and institutionalization and promise for sustainability. Also, this would have enabled TATF to dedicate more effort to providing assistance with reaching the Bogor Goals. 45 Recommendation: Although subsidies for such investments may not be the issue, consideration should be given in a follow-on project design to assessing the pros and cons of rebalancing level-of effort in the PMP Category 1 type of capacity-building activities in relation to that devoted to the external projects side, in order for the new project to consolidate past REI achievements under the TATF. If such a decision is merited, the potential for sustainability of trade reforms, opportunities to optimize/maximize the outcomes of future policy-focused analyses, and implementation of structural and regulatory reforms by those developing member economies that have voluntarily embraced them could be enhanced. Recommendation 12: Concretize greater technical cooperation between APEC and ASEAN. Context: ASEAN and APEC pursue similar goals of regional economic integration and broad￾based economic growth. ASEAN’s and APEC’s membership comprise ten and twenty-one economies respectively. The staff complement of the ASEAN and APEC Secretariats approximate 350 and 62 respectively, and almost 35 percent of APEC’s senior-most staff consist of officers who are generalists attached primarily to their respective economies’ Ministry of Foreign Affairs, and deployed to Singapore on two-year secondments. The U.S. Government’s assistance to the respective Secretariats of both institutions utilizes proven TATF platforms to effectively and efficiently deliver specialized technical assistance and training services. As stated earlier, APEC TATF’s work plans are driven primarily by a U.S. Government inter-agency committee, while ASEAN’s work plans are driven by its ten member states. Unsurprisingly, there is significant overlap in the economic integration themes pursued by these organizations, and opportunities exist for sharing the lessons learned and best practices of both. Toward this end, a “mapping exercise to identify potential areas for technical cooperation between APEC and ASEAN” was jointly conducted in 2011 in Jakarta, attended by senior officials and analysts from both organizations, including the APEC Executive Director and the ASEAN Secretary General. The preliminary results pointed to key areas of shared interests and priorities including food security, trans-national health management, supply chain connectivity, disaster preparedness and resilience, structural and regulatory reform, SME development, customs procedures/regimes, harmonized tariff nomenclature, and green buildings. Opportunities abound for taking this effort to the next stage by developing individual action plans for the medium-term – both at the thematic and country implementation levels – by building on the APEC medium-term strategic planning exercise that the APEC TATF has spearheaded for the thirty or so technical working groups within APEC. Recommendation: A possible follow-on APEC assistance project should include a mechanism for concretizing greater technical cooperation between APEC and ASEAN, the two apex regional institutions within Asia and the Pacific Rim for economic integration. Further, the U.S. Government-supported TATFs working in both organizations and the two Secretariats should convene urgently to develop an action plan for building bridges to link the efforts of ASEAN and APEC. Recommendation 13: Balance sustainability considerations of APEC support with a clear￾eyed calculus as to benefits to United States national interests. Context: As discussed in Chapter IV, there is clear evidence of what could be termed non￾sustainability in APEC Secretariat operations and on the projects side. The issue arises out of a number of important core values governing APEC’s operations, the most important being voluntarism. Voluntarism in APEC applies to such critical principles as non-enforceability of 46 agreements reached, the value of consensus among the highly economically differentiated member economies, and the commitment to a small core Secretariat staff operating under a regime of limited budgetary contributions, allowing decentralized initiative and power to remain with the member economies. However, as noted earlier, consequently, APEC is highly dependent on voluntary contributions from developed member economies – not least the U.S. Government through the TATF – without which it is arguable that certain institutional capabilities would not exist at all, and others that exist might not be sustained. While such non-sustainability ordinarily is regarded with disfavor and generally is rhetorically not supported by most development donors, the high level of buy-in to the voluntarism concept that exists among main APEC donor economies such as the United States, Australia, Singapore, Japan, New Zealand, and Korea cannot be ignored and must be respected. And rather than imposing a limit on the organization’s effectiveness, it appears to contribute to a certain ennobling esprit within the entire stakeholders’ community attached to this organization. It is as if they all agree with the paraphrase, “we not only can live with this awkward and counter￾intuitive situation, but it drives the organization to more consensual and therefore more durable outcomes, even if it requires subventions to attain them.” Recommendation: Since the United States has major national interests and equities in the successful performance of APEC in the increasingly important Asia-Pacific region, and since those national interests outcomes have real value as investments at least equal to the subsidies provided through the existing TATF or a future support project, sustainability considerations that would normally be applied to design decisions in a development project should be removed from the table in considering the future design of an APEC support project that would operate in an organization such as APEC where the voluntarism principle remains so central and seminal. Recommendation 14: Recognizing the permanent technical assistance team’s need to be responsive to changing work-load and the mix of skills required, any new project design should incorporate expedited mechanisms and authorities that accommodate quickly to such requirements. Context: Over the life of the current TATF project, the core team based at APEC headquarters in Singapore has always been lean in relation to the amount and diversity of activities on their agenda at any given moment. There has been a strong commitment from DNG headquarters in the United States to supporting its field team, providing not only head office oversight and support, but surge capacity to accommodate episodic short-term requirements and occasions such as the U.S. host year. Similarly, the skills distribution of permanent staff – including trade, finance, training and development generalist capabilities – has been an appropriate mix for carrying out the main corpus of TATF’s work. And where skills deficiencies have occasionally arisen, they have been ably accommodated by short-term specialist TA recruited from DNG’s worldwide network. However, owing to a combination of communications problems among the team, USAID and State, and some policy differences between State and USAID concerning the appropriate balance of costs between staff support and project expenditures, there have been moments when the “lean” staff became a “too-lean” staff, and agreement on how to supplement it and for what duration remained elusive. It is testament to the commitment of the contractor staff that these moments never led to crises, yet responsible management might have prevented these occasional staff imbalances from arising. An agreement on staff flexibility at the front end of project implementation – for example, through contract language authorizing that by exchanges of memoranda between the USAID COR and the CoP agreeing on such changes up to a 47 particular dollar amount or time duration – might constitute a sufficiently expedited mechanisms for implementing staffing changes that could materially reduce this vulnerability. Recommendation: Reflecting an awareness that human resource requirements may be highly variable over the life of any project that is intended to be short-term demand-responsive as the current TATF activity is, a new APEC support project intended to incorporate similar flexible training and technical assistance objectives should incorporate in its design expedited mechanisms and approval authorities that accommodate quickly to the ebb and flow of activity work-load. 1 APEC TATF Mid-Term Contractor Evaluation Volume 2 – Appendices LIST OF APPENDICES APEC TATF STAKEHOLDER SUMMARY ...................................................................................2 STRUCTUREOF APEC COMMITEES, SUB- COMMITEES, AND OTHER FORA .....................3 APPROVED APEC TATF MID-TERM EVALUATION STRATEGY AND ACTION PLAN .......4 INTERVIEW GUIDES AND QUESTIONAIRRE TEMPLATES ..................................................13 INTERVIEW FOR APEC PROJECT DIRECTORS ....................................................... 13 INTERVIEW FOR MEETINGS WITH PMU, PSU AND IT .......................................... 13 INTERVIEW FOR MEETINGS WITH RESPONSIBLE PUBLIC OFFICIALS IN APEC MEMBER ECONOMIES ................................................................................................ 14 GENERIC QUESTIONNAIRE TEMPLATE USED FOR USG INTER-AGENCY INTERVIEWS ................................................................................................................ 14 PARTICIPANTS SURVEY QUESTIONNAIREAND RESULTS ..................................................16 SUMMARY OF FIELD VISITS AND INTERVIEWS ....................................................................34 INTERVIEWEES FOR TATF EVALUATION .............................................................................43 U.S.-BASED CONTACTS INTERVIEWED .................................................................. 43 CONTACTSINTERVIEWED IN ASIA-PACIFIC REGION, Sept-Oct 2012 ................ 44 EVALUATION STATEMENT OF WORK .................................................................................48 2 APEC TATF STAKEHOLDER SUMMARY State Department: In connection with its provision of funds and chairing of the inter-agency committee that oversees U.S. Government agencies’ participation in APEC committees, working groups and other APEC fora, the State Department approves funded tasks that make up the TATF work plan. Execution of this work plan contributes to achievement of U.S. national interests in the Asia-Pacific region through U.S. participation in APEC. U.S. Government (USG) Agencies: State Department organizes and chairs an inter-agency committee comprised of USG technical departments and agencies whose staff interacts through APEC with other member economies’ staffs in defined technical areas to achieve economic integration and reforms in member economies as called for by APEC’s Bogor Goals. Such inter￾agency stakeholders are clients of TATF, setting the agendas for “projects” that TATF may contribute to executing or facilitating in conjunction with other member economies. USAID: USAID supervises the TATF’s performance and is the accountable administrator of State Department ESF funds that are allowed to the Agency to implement APEC assistance through the TATF. As the USG’s primary development assistance agency, USAID has a stake in the developmental outcomes of TATF capacity building and project implementation outcomes. APEC Secretariat – Executive Level: Through provision of technical assistance to strengthen the organizational capabilities of entities in the APEC Secretariat, APEC assists in implementing organizational and administrative reform and capacity building initiatives requested by the APEC Executive Director and other senior officials. APEC Secretariat – Permanent and Seconded Staff: TATF direct training of Secretariat staff enhances individual capacity to carry out appointed duties, as well as the capacity of APEC Secretariat corporately to achieve its objectives. APEC Member Economies: Through participation in TATF-facilitated events, and through training and other support they receive from TATF in the project proposal process, officials and organizations from APEC member economies are assisted in the processes supporting progress in meeting Bogor Goals and related reform and regional economic integration activities. Private Sector and NGOs in Member Economies: Considering the weight of commerce, trade, investment, SME development and related activities geared by reform and development actions advocated in committees and other APEC fora assisted by APEC TATF, private businesses', ABAC members', business advocacy and assistance NGOs' and other private actors' interests may be directly affected by the quality of TATF facilitation, training and technical assistance inputs and outcomes. 3 STRUCTUREOF APEC COMMITEES, SUB- COMMITEES, AND OTHER FORA 4 APPROVED APEC TATF MID-TERM EVALUATION STRATEGY AND ACTION PLAN Introduction: What is APEC and why does it matter? Asia-Pacific Economic Cooperation (APEC) is the premier forum for facilitating economic growth, cooperation, trade and investment in the Asia-Pacific region. APEC is the only inter-governmental grouping in the world operating on the basis of non-binding commitments, open dialogue and equal respect for the views of all participants. Unlike the WTO or other multilateral trade bodies, APEC has no treaty obligations required of its participants. Decisions made within APEC are reached by consensus, and commitments are undertaken on a voluntary basis. APEC has 21 members – referred to as "member economies" – which account for approximately 40 percent of the world's population, approximately 56 percent of world GDP, and about 48 percent of world trade. Since its inception in 1989, the APEC region has consistently been the most economically dynamic part of the world. In its first decade, APEC member economies generated nearly 70 percent of global economic growth, and the APEC region consistently outperformed the rest of the world, even during the 1997 Asian financial crisis. APEC member economies work together and sustain this economic growth through a commitment to open trade, investment and economic reform. By progressively reducing tariffs and other barriers to trade, APEC member economies have become more efficient, and exports have expanded dramatically. TATF’s Objectives: The APEC Technical Assistance and Training Facility (APEC TATF) is a four-and-a-half-year $17, 657,720 project managed by the USAID Regional Development Mission Asia (USAID/RDMA) with funding from the U.S. Department of State’s Bureau of East Asian and Pacific Affairs, Office of Economic Policy (State EAP/EP). It began operation in September 2008, and has a projected end date of 1 March 2013. APEC TATF is embedded in APEC headquarters in Singapore. The project is implemented by a consortium of TA firms co-led by Development Alternatives, Inc. and Nathan Associates (DNG). The purpose of the project is to help the APEC Secretariat become a strategically managed institution that achieves policy objectives more effectively and efficiently. The project also provides capacity building assistance to APEC member economies so they can contribute to strategic goals for regional economic integration and meet the Bogor Goals of free and open trade and investment in the Asia-Pacific region by 2010 for developed economies and 2020 for developing economies. Purpose of Evaluation: An external participatory mid-term evaluation (Contractor Performance Review) of APEC TATF activities has been called for by USAID/RDMA, and will be conducted from September - November 2012. The objectives of the mid-term evaluation of APEC TATF are to: 1) evaluate contractor performance with respect to compliance with the award by meeting agreed-upon project objectives and intermediate results; and 2) develop a lessons￾learned package of information around which the current program can be shaped and future projects can be designed. Towards this end, the evaluation will identify TATF’s approaches, programs, and interventions that have been successful and should be duplicated or continued in future program design. The evaluation also will identify key weaknesses that USAID and State/EAP/EP should address in their future APEC project design. The lessons learned through this evaluation may also inform additional projects with the APEC Secretariat and/or other USG engagement with regional bodies. 5 Following the completion of the current DNG contract, it is expected that the APEC Secretariat will continue to merit U.S. Government support for accelerating the pace of adoption of critical economic reforms and behavioral changes for achieving U.S. foreign policy priorities in the Asia￾Pacific region and robust economic integration of the lesser-developed member economies of APEC. It is important to note that this is not an evaluation of the APEC Secretariat itself. Rather it is an evaluation of the contractor’s implementation of the Facility and the direct support it provides to the Secretariat; and through its support for the APEC Secretariat’s efforts to achieve Bogor Goals, to promote development progress in assistance-eligible member economies. Evaluation team and members: The team consists of a team leader who will coordinate all activities (Michael Farbman), and a senior analyst (Abdul Wahab). Others from USAID/RDMA/GDO and State/EAP/EP may participate as time permits, and APEC member economy observers will be welcomed, if interested. The evaluation also is being structured to accommodate participation by two observers from the PROC who have been nominated by Chinese MOFCOM.1 The mid-term evaluation will be conducted over five phases. The first phase – the preparatory phase – was implemented by the consultants in the United States, and involved completion of the following actions: Phase 1 – Getting grounded  Review of project development background material and USAID approval documents.  Review of work plans and select project-supported activities that have been completed or are continuing.  Review of available USAID project Performance and Management Plan (PMP), monitoring, implementation, and management reports.  Establish initial contacts with RDMA/GDO APEC TATF project team.  Establish initial contacts with APEC TATF lead Technical Assistance Consortium Contractor.  Establish initial contacts with the USG’s principal interlocutor for APEC TATF activities, State EAP/EP, and coordinator of a whole-of-U.S Government effort that aims at identifying and advocating for policy-based changes that are intended to accelerate the pace of regional economic integration among the 21-member APEC community and that are consistent with, or that advance, U.S. interests in the Asia-Pacific region.  Hold discussions with key U.S. Government inter-agency representatives whose program strategic decisions are informed by the incumbent Administration’s foreign policy priorities. The inter-agency group convenes regularly to identify and propose project and policy-related initiatives that draw on TATF facilitation to implement their cooperation activities in APEC fora and working groups. Comprised of U.S. Government Departments and entities including USTR, Department of Commerce, USDA/FAS, inter al., this group approves and plans budget levels for supporting TATF activities, and in these ways influences the structure, content, and strategic focus of the annual TATF work plans. 1 The participation on the part of the PROC is an extraordinary action designed to exchange ideas and best practices on evaluation; engage with Chinese colleagues on project design and evaluation; and to better understand mutual areas of evaluation interest and deepen U.S.-China relations in the area of foreign assistance and project implementation. 6 Phase 2 – Evaluation Hypothesis and Approaches to Data Collection and Trends Analysis The second phase of the evaluation will be held on site at RDMA/GDO in Bangkok. It will consist of an inception briefing with RDMA/GDO APEC TATF project staff, and wider meetings with other Mission stakeholders. The expected outcomes of these meetings, which are envisaged to last for 5-10 days, beginning on Sept 24, are confirmation of the evaluation hypothesis; the specification of instruments and methodology(s) for developing the data for assessing contractor performance; and preliminary design of the approach to analyzing these data to confirm (or not) the evaluation hypothesis. Note on Performance Measures: As normally understood, human and institutional capacity development interventions include training to address skill and knowledge gaps, and to deal with other impediments such as the inadequacy or lack of necessary tools and incentives.2 Because this characterization (“human and institutional capacity development”) covers TATF’s core objectives for APEC member economies and associated human resources, it might be expected that “outcomes”, rather than outputs and other intermediate results, might be sought as performance measures. However, because member economies of APEC, as voluntary participants in the community, do not enter into binding or enforceable agreements around reforms, policies and other changed behaviors that APEC aims to influence, “outcomes” as defined by USAID are not appropriate as measures of TATF (or APEC, for that matter) performance. Therefore, this mid-term evaluation3 will link TATF performance in providing training, TA, and advisory services to the APEC Secretariat as well as through the Secretariat to launch projects and activities in member economies (primarily lesser developed), to the following measures: 1) measures of improved organizational and management performance by the Secretariat; 2) demonstrated evidence of education and consensus-building around improved economic and trade practices and protocols for achieving the Bogor Goals; and 3) inferentially derived measures of success in achieving U.S. foreign policy objectives in the Asia-Pacific region. Evaluation Hypothesis: The Evaluation Statement of Work (pp. 18-19) identified three key evaluation questions to be answered, which we have mapped into the following two-part hypothesis:  Through demand-driven training for Secretariat staff, and the identification of priority management operations improvements, U.S. APEC TATF has strengthened the institutional ability of the APEC Secretariat to carry out its mandate; and  U.S. APEC TATF-supported assistance is furthering the aims of APEC member economies to achieve the Bogor Goals of regional economic integration and free/open trade, which materially support U.S. strategic interests in the Asia-Pacific region. As noted, TATF is constructed as a platform for delivering specialized institutional- and capacity￾strengthening services targeted at the APEC Secretariat per se, and is as well a source for supporting the Secretariat’s capacity to facilitate achievement of the APEC mandate. Therefore, 2 USAID Human and Institutional Capacity Development (HICD) policy paper, p. 6. 3 Performance evaluations focus on descriptive and normative questions: what a particular project or program has achieved (either at an intermediate point in execution or at the conclusion of an implementation period); how it is being implemented; how it is perceived and valued; whether expected results are occurring; and other questions that are pertinent to program design, management and operational decision-making. Performance evaluations often incorporate before-after comparisons, but generally lack a rigorously defined counterfactual (relating to or expressing what has not happened or is not the case). 7 to the extent possible, the evaluation team will examine the appropriateness of the TATF’s structure for facilitating the capacity-building needs of the Secretariat and the effect of the structure on enabling the Secretariat to effectively contribute to achievement of the APEC mandate. There is a separate and equally important dimension that also must be considered – that is, the quality and effectiveness of the contractor and the contractor’s staff. These two dimensions – structure and staff quality – must be differentiated, as they may function independently to determine the quality of the overall performance and effectiveness of the Secretariat. Data Development and Analysis: As noted above, data will be gathered on TATF’s contributions to the APEC Secretariat’s demonstrated ability to effectively discharge their core functions, including education and training relative to achievement of Bogor Goals; information and communications technology; streamlining the project review process; monitoring and post￾project evaluation processes (the Secretariat is mid-way in a three year strategic plan [2011 – 2013] focusing, inter al., on internal M&E skills development, learning lessons and articulating expected results); public outreach; inter al. These data will be sourced through document reviews, administering mini-surveys (see the example at Appendix 1) and holding key informant interviews with internal Secretariat staff. External validation of findings also will be sought through targeted enquiries to member economy officials and other key stakeholders who have insight into such performance by the Secretariat. Data on effectiveness of member economies’ embrace of behaviors, practices, and protocols consistent with achieving Bogor Goals will be gathered through enquiries of a relatively large randomized sample of seminar and workshop participants in select technical and reform areas such as trade and investment facilitation, economic integration, ease of doing business improvements, tariff reductions and harmonization, inter al. These data will be collected through e-mail surveys of – and visits to – a representative sample of member economies to determine the degree to which better practices consistent with achieving Bogor Goals have been understood, embraced and put into effect as judged by respondents. Measures of applicability of training obtained through such enquiries will by necessity be subjective (though possibly documentable in metrics such as increased trade or reduced tariffs), as objective or numeric measures of such progress can be difficult to estimate. Where appropriate, the data will be disaggregated to take gender considerations and gender biases as appropriate into account. In order to assess whether TATF support is contributing to U.S. strategic interests, targeted interviews will be held with U.S. Government officials (primarily under the State Department-led inter-agency working group which drives many TATF activities). See Appendix 3 for the generic questionnaire template proposed for these interviews. Many participants on the inter-agency group are actively involved in TATF activities, attend Senior Officials Meetings and Leaders’ Meetings, and are in a position to define through their respective U.S. Government agencies’ optic, the objectives of the U.S. Government’s interest in and support for APEC as a key instrument for achieving the rebalancing of U.S. priorities in the Asia-Pacific region. A suggested format/matrix that summarizes data sourcing by key performance areas and survey respondents is presented in Appendix 5. Phase 3 – Field Work and Country Visits The RDMA/GDO meetings will be immediately followed by a round of field visits to selected member economies. The visits are intended to validate and supplement available project 8 information, and to obtain feedback from key individuals on their perceptions of which TATF contributions to APEC activities have, and have not, worked. The ultimate objective will be to distil and analyze these informed interlocutors’ opinions on areas of support to APEC that merit continued U.S. Government support; their suggestions on ways to improve performance; and to determine new priorities to be considered under a follow-on effort. Country Visit Methodology: In parallel with the initial foundational reading-in period, which began on September 5, information was acquired through interviews of main stakeholders, including RDMA, Nathan Assoc./Arlington, APEC TATF/Singapore, and USG inter-agency representatives. Subsequent to initial interviews, and based on guidance from early respondents, targeted former and current participants in member economies will be identified to be addressed as individual respondents and in small groups. They will be asked a number of questions from prepared agendas of questions evaluating their experiences with TATF. Similarly, small group interviews will be held during the Singapore visit with APEC Secretariat officials such as administrative personnel, “Project Directors,” (Executive Directors?), IT managers, budget, and project management, and others who have been recipients of capacity building-type activities. In the visited member economies, depending on access, former participants/trainees, APEC oversight officials and other partners that had engaged on TATF-supported APEC initiatives, (e.g., related to reaching Bogor Goals), will be interviewed, either as individuals or in small groups. See appendix for a sample interview guide. Similarly, advanced countries involved in joint assistance to APEC will be canvassed as to their experience working with TATF and the effectiveness they perceive in TATF interventions in behalf of the Secretariat. Singapore, headquarters for APEC, within which the APEC TATF implementing office resides, will be the first country visited after the inception phase in RDMA. Singapore is one of APEC’s more developed member economies, and furthermore is a demonstrated strong U.S. partner and co-sponsor of specialized APEC-endorsed initiatives. As the U.S. did in 2011, Singapore served as the host economy and chaired the APEC Ministerial Meetings in 2009. Therefore, interviews will also be scheduled with Singapore Government officials who are secondees to the APEC Secretariat, as well as with interlocutors embedded in the Singapore Ministries that engage with APEC. Stemming from Australia’s founding efforts in the creation of APEC; AusAID also has been an enduring and prominent supporter and donor member economy of APEC. It has long stationed a senior-level permanent resident delegate based at the APEC secretariat. While in Singapore, the team will endeavor to learn of Australia’s experience with TATF and how U.S.-supported programming assistance complementarities with AusAID might be further strengthened in the future. Japan is another strong supporter of APEC and has been a recent host of the APEC Ministerial Meeting (2010). It has sponsored many policy studies, training events and workshops on specialized APEC themes, which have placed emphasis on the economic performance and policy gaps and opportunities of the lesser-developed member economies. Thus, the evaluation team intends to hold discussions and conduct interviews with Japanese APEC counterparts who are resident delegates at the APEC Secretariat. The team considered paying calls on APEC-related Government Officials in Tokyo from the various APEC-concerned Ministries, but has determined that the benefit-to-cost ratio does not justify such a visit. 9 In addition to holding consultations with the APEC TATF core staff on defined evaluation elements as called for under the Statement of Work, the evaluation team will hold discussions with small groups, and will administer questionnaires to APEC Secretariat staff and others working in APEC with organizational and functional connections to TATF. These encounters are designed to complement and validate the reporting data sourced from other respondents as well as that reported by the APEC TATF project implementation team. Also, further insights into any future U.S. Government support will be sought. Collectively, the data and information, and input by member economies’ resident delegates (Project Directors) to APEC will be analyzed and the findings added to the information pool for ascertaining the effectiveness of the TATF in enhancing the APEC Secretariat’s institutional ability to discharge their myriad administrative, operational, and technical functions for supporting attainment of the Bogor Goals, including single- and multi-country studies and projects. Please see Appendix 1 for a model questionnaire template and a list of groups to be sampled for feedback through surveys and small group interviews. Country Selection: The Singapore field visit will be followed by visits to selected APEC member economies. As USAID naturally is concerned with developmental content of TATF activities, and as developing member economies are recipients of substantial TATF services, developing economies will be targeted for team visits. Specific economies to be visited have been identified in accordance with the following selection criteria:  Member economies that have received training and other assistance from TATF.  Member economies where it is already known that impacts have been perceived and are measurable.  Member economies having a substantial number of identifiable participants in APEC TATF training and workshops.  Member economies in which data have been disaggregated by gender and where women-focused assistance has been offered.  Member economies that have demonstrated marked progress in achieving Bogor Goals.  Member economies that may be notorious for not making progress despite known disproportionate efforts directed at them.  Member economies that have hosted relatively large numbers of USG agency￾supported participatory events.  Member economies where the regional integration progress is tied to U.S. priority interests.  Member economies whose APEC-assisted benefits are well circumscribed and therefore measurable.  Member economies that have a well-developed data base on development impacts. Document review by the evaluation team and conversations between the team and stakeholders including RDMA/GDO, State EAP/EP, TATF staff in Singapore and Nathan/HQ overseers have led to selection of three developing member economies that we believe will justify the cost and effort to visit, and which conform to the selection criteria enumerated above. They are: Indonesia, Vietnam, and Thailand. As the upcoming (2013) APEC host country for the next Ministerial Meetings, Indonesia will have a large hand in setting priorities (including for TATF or its replacement) for future APEC activities. 10 For this reason, its perspectives on the interactions it has had with TATF would be timely and useful. Indonesia has participated actively and was a major proponent for a number of years in TATF-assisted APEC rulemaking policy initiatives. Indonesia has made sufficient progress in building its capacity in promoting structural reform to assume a leadership role in advocating and providing technical assistance for structural reform in other developing member economies. Also, it should be noted that Indonesia is the home of the ASEAN Secretariat, with which stronger cooperation with APEC is a priority U.S. objective. Vietnam also has participated in APEC workshops and case studies on improving public consultation in the rulemaking process, and on issues concerning cross-border services (the latter also including participation by Thailand and Indonesia). The Vietnam Ministry of Industry and Trade has requested and received TATF assistance in understanding OECD principles on the role of corporate governance as it contributes to the development of SMEs. Vietnam also has received assistance relating to empowering marginalized business women by easing the rules on establishing businesses and by mobilizing lending capital. Vietnam has been participating in the “de minimis initiative.” They also were the subject of a TATF-assisted case study on environmental goods and services markets and industries. It also will be valuable to include Vietnam from the perspective of it being still largely a command economy, for all that implies regarding openness to trade and investment, and economic integration through liberalized markets. With respect to the choice of Thailand, apart from it being opportunistic while the team is in Bangkok to visit Government of Thailand APEC oversight officials on general matters, TATF has recently completed a diagnostic survey on environmental goods and services, and the GoT participated prominently in TATF-supported EGS workshops. As noted above, GoT officials also participated in consultations in the rulemaking process workshops. They also participated in workshops on regulatory issues in cross-border services trade. Beginning with the inception period at RDMA and ending with the drafting and vetting of the final evaluation report in Bangkok, field visits will occupy fully the month of October. For details on the timing of country visits, refer to the task and travel agenda incorporated in this paper as Appendix 2. Phase 4 – Evaluation Report Preparation This phase will focus on validation of the evaluation hypothesis, vetting the analyses of all data collected to this point, principal findings of which will be jointly reviewed with TATF, the APEC Secretariat, and RDMA/GDO and other Mission stakeholders for feedback and concurrence. Should errors of omission or material differences in interpretation be identified, follow-up efforts with respondents will be carried out to the extent possible to validate the accuracy of the findings and to agree on defensible changes. This phase of the evaluation will include discussions among core evaluation participants and stakeholders relating to the evaluation findings and recommendations. Discussions will incorporate key conclusions that emerge from the assembled data. To the extent possible, and based on respondents’ feedback to survey questionnaires, the results and outcomes of strategically essential Contractor-implemented activities will be disaggregated to align with: 1) APEC’s three pillars for achieving the Bogor Goals – trade and investment liberalization, business facilitation, economic and technical cooperation; 2) institutional strengthening and capacity building within the APEC Secretariat; 3) APEC TATF successes and challenges in 11 supporting member economy actions aimed at achieving Bogor Goals; and 4) progress towards achieving U.S. foreign policy interests and priorities under APEC. Before moving toward finalization of the report, with the concurrence of RDMA, the data, as well as preliminary findings, lessons learned and draft recommendations for future directions for support to APEC will be shared and vetted with partners at State EAP/EP. The objective will be to obtain feedback, and as a precursor for developing wider inter-agency concurrence in the anticipated final report’s conclusions. The conduct of this phase will require guidance from RDMA as to when to solicit Washington agency input and reactions to draft conclusions derived from the field enquiries and for filling in supplemental information that may be sourced from U.S. Government agencies. For example, it is possible that information gathered from field contacts with TATF recipients and cooperators will give rise to a need to fill gaps and to revalidate initial assumptions and observations by contacting principals from U.S. Government agencies (e.g., Commerce, USTR, USDA/FAS) affiliated with various APEC fora that have been participants in, or proponents of, various TATF-supported activities. Such conversations with Washington agencies, if necessary, can be done while the team is in Bangkok, or may be held in abeyance for addition when the team returns to the U.S. We seek RDMA’s guidance as to when best to resolve any such issues. Phase 5 – Enumerating Principal Conclusions and Recommendations This section of the report will focus on future aspects of U.S. Government engagement with APEC. The discussion will include lessons learned and an articulation of implications for a continued whole-of-government (inter-agency) approach to and deepening engagement with APEC through an intermediary TA/training mechanism such as the TATF. The discussion is expected to focus on re-justifying and/or introducing adjustments to priority areas for affecting and supporting the major policy and administrative objectives of APEC and its Secretariat. The report will also discuss the current and future implications for this evaluation under the USAID Forward Agenda and USAID assistance. 09/26/2012 Format/Matrix for Summarizing Evaluation Methodology Key Elements Document Review Stakeholder and key Informant Interviews Rapid email Survey Small Group Encounters In APEC Secretariat Site Visits Other4 Validity and effectiveness of approach X X X X Confirmation of results; performance measurement & targets X X Responsiveness to stakeholder needs X X X 4 To be defined if needed 12 Challenges to implementation X X X APEC Secretariat Capacity X X X Lessons learned, guidance, recommendations X X X X X X Timeline for Field Portion of APEC Mid-Term Evaluation 22 Sept – Consultants depart Washington and Atlanta for travel to Bangkok. 23 Sept – Consultants arrive Bangkok. 24 Sept - 6 Oct – Inception meetings with RDMA, refining and approval of APEC Evaluation Strategy, and finalization of logistics. Interviews with Thai officials. 7 - 13 Oct – In Singapore, meeting with APEC officials, TATF staff, Singapore, Australia and Japan APEC-oversight officials. 15 - 18 Oct – In Indonesia for meetings with GoI APEC-oversight officials, former project participants and trainees, ASEAN TATF staff, USAID staff. 19, 20 Oct – In Bangkok for intermediate review of Singapore and Indonesia findings. 21 - 24 Oct – In Vietnam for meetings with GoV APEC-oversight officials, former project participants and trainees, USAID staff. 25 Oct – Return to Bangkok to begin work on Phases 4 and 5. Engage with GDO and other RDMA stakeholders on validation and precision of findings, drafting of evaluation report. 3 Nov – Consultants depart Bangkok to return to USA TBD in U.S. – Polish final report as required; review/present findings to State EAP/EP and inter-agency stakeholders. 13 INTERVIEW GUIDES AND QUESTIONAIRRE TEMPLATES INTERVIEW FOR APEC PROJECT DIRECTORS Name Member economy of affiliation Months serving in Singapore Forums with which you are most closely associated  What has been the main way you have related to APEC TATF?  Is it predominantly on the projects side, or have you been recipients of some capacity building or other training activities supported by TATF?  How would you rate any such training you received? Has it fed into your work at the Secretariat?  On the projects side, please describe how you interact with the TATF. E.g., do you cooperate on conferences/seminars/workshop design or implementation? Logistics cooperation (venues, traveling speakers and participants)?  Please describe some instances of what you feel worked best in these encounters between the TATF and you/your economy, in terms either of relevancy and timeliness, impact or quality of process.  Any take-aways or anecdotes of highly successful events having impacts on advancing Bogor Goals?  Do you have any guidance as to how to improve TATF performance in the type of events in which you or others from your economy encountered their work?  Have you any thoughts on how we might re-design for greater effectiveness the next iteration of a TATF-like entity in APEC? Is the embedding of such a TA/Training platform in the APEC Secretariat an appropriate/effective/efficient way to go? Is there an alternative? INTERVIEW FOR MEETINGS WITH PMU, PSU AND IT Name Member economy of affiliation Months serving in Singapore  Could you tell us a bit about your principal responsibilities?  What has been the main way you have related to APEC TATF? Is it mostly through capacity building of your operation, personal training, linkages to APEC projects?  Please describe in particular the capacity building that you are aware TATF has done for your operation, and elsewhere in the Secretariat.  How would you rate the quality and impact of the capacity building and training? In particular, how much can be attributed to the quality of the work the TATF does in organizing such activities? Can you give us some examples?  To what degree to you think the progress you/your unit has made through TATF support might not have been possible had there not been a TATF (and its funds) in the Secretariat?  Are there other TA and training options you and the Secretariat could have pursued? 14  What do you think are the pros and cons for the TATF being embedded in the Secretariat?  Have you any thoughts to offer us about any adjustments or areas of additional and continuing focus for the future TATF-like assistance by the USG? INTERVIEW FOR MEETINGS WITH RESPONSIBLE PUBLIC OFFICIALS IN APEC MEMBER ECONOMIES Name Gender Title and responsibilities Agency or private affiliation Member economy  Could you tell us a bit about your principal responsibilities, especially as they relate to APEC?  What has been the main way you have interacted with the APEC TATF?  What is the subject matter (corresponding to TATF encounter; e.g., rules-based policy, EGS, EoDB, structural reform, etc.).  Please describe how you encountered TATF (APEC Working Group, case study, seminar, workshop, training, etc.) When, and on how many occasions did you engage with them?  What aspects of your contact with TATF facilitation do you recall?  Are your recollections mostly of strengths or mostly of weaknesses of TATF, and in what ways?  How did the subject matter of your encounter relate to your official responsibilities? Describe the special interest area, objective or best practice knowledge that you hoped to take away from the seminar, workshop or study in which you encountered TATF.  Follow-up actions: What did you do with this knowledge – educate others, put it to work in a personal advocacy effort for reforms, use it at a negotiating table?  To what end? Were efforts to establish better practices/processes advanced? Were new regulations developed or old ones changed? Through this learning, was progress made in consensus-formation around some initiative?  You are aware of APEC’s “Bogor Goals” – free and open trade and investment in Asia￾Pacific by 2010 for developed economies, and 2020 for developing economies (expand in detail as appropriate). Can you link your encounter facilitated by TATF to any progress in meeting your or any other Asia-Pacific economy’s Bogor Goals? GENERIC QUESTIONNAIRE TEMPLATE USED FOR USG INTER-AGENCY INTERVIEWS  How long have you been engaged in the inter-agency process through State in doing work with/through APEC?  As a management and liaison forum, does the existing inter-agency process and the communications it sets up meet ______’s needs?  Describe the nature of your typical interaction with TATF as an agent or facilitator of your work with APEC member economies. 15  What subject matter and what (sub-)committees or fora do you mostly work with in APEC?  Please give us some examples of priority S-T and L-T initiatives of USDA in APEC. Has TATF been involved in some/most/all of them?  Please share your opinion as to how effective TATF is as your partner in your project, policy and reform initiatives.  Can you identify certain strengths of the TATF as a contributor to meeting _____ objectives in APEC?  How about some weaknesses, or things you wish they could do differently, better or in addition?  For a likely future follow-on project, what small or structural changes in the facilitation process might you suggest be incorporated?  Any other comments touching on other issues that you would wish to advise us as we engage more deeply in this evaluation? 16 PARTICIPANTS SURVEY QUESTIONNAIREAND RESULTS Contemporaneously with the field visits, a survey was e-mailed to 100 percent of all participants who had registered e-mail addresses when they had attended a sample of APEC seminars, workshops, conferences and training events that TATF had supported since the project’s inception. In all, 1327 persons were invited to respond to a brief on-line survey designed to assess their satisfaction with the event they attended; their perceptions of TATF’s facilitation of, or contribution to, the event; and the value to them of the content insofar as it might help them with their responsibilities for promoting reform, especially in relation to the Bogor Goals. In order to measure development impacts as much as possible, questions were designed to prompt feedback that addressed the extent to which participants believed these seminars and workshops developed their various skills to plan and to measure progress in implementing reforms; to define and to administer the technical assistance required to support a reform process; and the ability to build consensus among the diverse stakeholders who must be included in change processes in home economies. Through one unstructured question, respondents also were provided an opportunity to describe notable reform achievements that they participated in that could plausibly be attributed to their participation in these events. In all, 210 responses (15.8 percent) were received. The attrition rate was due primarily to two factors: 1) e-mail addresses that were no longer valid (measurable by bounced and undeliverable surveys); and 2) inferentially, to the amount of time that had elapsed since many of the targeted recipients had attended their event. Generally, as would be expected, response rates were considerably higher for workshops held in 2011 and 2012 than they were for those held in 2009 and 2010. Demographic information – gender, organizational affiliation, member economy and job title – for each recipient also was collected. Following are some key findings:  49.5 percent of respondents claimed to be “very satisfied,” and 49.0% were “satisfied” with the workshop they attended.  26.7 % acknowledged that they were “very familiar,” and 52.9% were “familiar” with TATF as organizers/facilitators of the workshop.  Respondents were 55.2 % male, and 44.8 % female (though 17.5 % of recipients skipped the question!) To the question “to what extent did what you learned in this workshop facilitate your understanding and/or effectiveness in helping your economy to achieve the following APEC Bogor Goals,” responses were as follow: Helped A Lot Helped Somewhat Did Not Help at All N/A Trade and Investment Liberalization 60 96 11 28 Business Facilitation 79 89 10 19 Economic and Technical Cooperation 98 90 2 9 The responses immediately above, indicating that the vast majority of respondents believe they profited from attending these events, demonstrate at the minimum that APEC’s intent in 17 supporting these projects to achieve the Bogor Goals, as assisted in implementation by TATF, was largely achieved. With respect to the question “how helpful was the workshop in building your ability to advance the following reform objectives,” respondents clearly indicated that they perceived they had received benefits across the broad array of skills helpful to the reform process. Very helpful Somewhat helpful Not at all helpful N/A Planning a strategy for reform 90 97 5 10 Developing qualitative and quantitative indicators for measuring progress toward achieving reform objectives 83 96 13 10 Building consensus around reform objectives among other stakeholders in your economy 73 110 8 12 Defining technical assistance activities to help move forward the reform process 94 94 8 7 Implementing technical assistance activities to help move forward the reform process 71 108 13 11 Complementary to the above responses on skills acquired, on the question, “please rate how helpful this facilitated workshop was in helping you to work with colleagues and other organizations in your economy to achieve reform,” 96.9 percent of respondents indicated that they found their respective facilitated workshop very helpful or somewhat helpful. Finally, and perhaps most importantly, 55 percent of respondents elected to complete the open￾ended question, “please describe the areas and the nature of the reform or technical skills progress that you feel you achieved by your participation in this workshop and related activities stimulated by this workshop.” Responses included descriptions of such outcomes as reform initiatives that they actually had participated in, best practices they had been successful in negotiating with others to install, establishment of monitoring and evaluation systems to track impacts, inter al., which they judged were attributable to their participation in TATF-facilitated workshops focused on transferring exactly such technical skills and capabilities. A facsimile of the four-page survey document follows this page. 18 19 20 Summary of Full Sample Survey Responses Q1. How satisfied were you overall with the APEC Workshop you attended? Answer Options Response Percent Response Count Very Satisfied 49.5% 105 Satisfied 49.1% 104 Dissatisfied 1.4% 3 Very Dissatisfied 0.0% 0 Answered Question 212 Skipped Question 0 Q2. How familiar are you with the APEC Technical Assistance and Training Facility (APEC TATF) that organized and facilitated the workshop? Answer Options Response Percent Response Count Very Familiar 26.9% 57 Somewhat Familiar 52.4% 111 Not at all Familiar 20.8% 44 Answered Question 212 Skipped Question 0 49.5% 49.1% 1.4% 0.0% 0% 10% 20% 30% 40% 50% 60% Very Satisfied Satisfied Dissatisfied Very Dissatisfied Q1. How satisfied were you overall with the APEC Workshop you attended? 21 Q3. To what extent did what you learned in this workshop facilitate your understanding and/or effectiveness in helping your economy to achieve the following APEC Bogor Goals? Answer Options Helped A Lot Helped Somewhat Did Not Help at All N/A Response Count Trade and Investment Liberalization 60 98 11 28 197 Business Facilitation 79 90 11 19 199 Economic and Technical Cooperation 99 92 2 9 202 Answered Question 205 Skipped Question 7 26.9% 52.4% 20.8% 0% 10% 20% 30% 40% 50% 60% Very Familiar Somewhat Familiar Not at all Familiar Q2. How familiar are you with the APEC Technical Assistance and Training Facility (APEC TATF) that organized and facilitated the workshop? 22 Q4. How helpful was the workshop in building your ability to advance the following reform objectives? Answer Options Very helpful Somewhat helpful Not at all helpful N/A Response Count Planning a strategy for reform 91 98 6 10 205 Developing qualitative and quantitative indicators for measuring progress toward achieving reform objectives 84 98 13 10 205 Building consensus around reform objectives among other stakeholders in your economy 74 111 9 12 206 Defining technical assistance activities to help move forward the reform process 95 96 8 7 206 Implementing technical assistance activities to help move forward the reform process 72 109 14 11 206 Answered Question 207 Skipped Question 5 60 79 99 98 90 92 11 11 28 2 19 9 0 50 100 150 200 250 Trade and Investment Liberalization Business Facilitation Economic and Technical Cooperation Q3. To what extent did what you learned in this workshop facilitate your understanding and/or effectiveness in helping your economy to achieve the following APEC Bogor Goals? (Answer Count) Helped A Lot Helped Somewhat Did Not Help at All N/A 23 Q5. Please rate how useful you found the following Answer Options Very Useful Somewhat Useful Not at All Useful N/A Response Count Plenary Sessions 134 55 0 8 197 Presentations by Speakers 142 52 1 0 195 Panel Presentations 125 67 0 3 195 Small-group breakout Sessions 106 66 3 17 192 Answered Question 197 Skipped Question 15 91 84 74 95 72 98 98 111 96 109 6 13 9 8 14 10 10 12 7 11 0 50 100 150 200 250 Planning a strategy for reform Developing qualitative and quantitative indicators for measuring progress toward achieving reform objectives Building consensus around reform objectives among other stakeholders in your economy Defining technical assistance activities to help move forward the reform process Implementing technical assistance activities to help move forward the reform process Q4. How helpful was the workshop in building your ability to advance the following reform objectives? (Answer Count) Very helpful Somewhat helpful Not at all helpful N/A 24 Q6. Please rate how helpful this facilitated workshop was in helping you to work with colleagues and other organizations in your economy to achieve reform objectives Answer Options Response Percent Response Count Very helpful 57.1% 113 Somewhat helpful 39.9% 79 Not at all helpful 2.0% 4 N/A 1.0% 2 Answered Question 198 Skipped Question 14 134 142 125 106 55 52 67 66 0 1 0 3 8 0 3 17 0 50 100 150 200 250 Plenary Sessions Presentations by Speakers Panel Presentations Small-group breakout Sessions Q5. Please rate how useful you found the following(Answer Count) Very Useful Somewhat Useful Not at All Useful N/A 25 Q8. From the drop-down list below, please select the name of the Workshop or Seminar you attended (this is available in the email you received) Answer Options Workshop: APEC Micro-Credit Workshop and Women Entrepreneurs August 2009, Singapore 2 APEC Workshop on Human Resources Impacts of the Global Economic Crisis July 2010 in Jakarta, Indonesia 6 The Establishment of Use of Accountability Agents in the APEC Cross Border Privacy Rules System, February 2010 in Hiroshima, Japan 5 Workshop on Regulatory Approaches to Smart Grid Investment/Deployment, May 16-17, 2012 in Quebec City, Canada 3 APEC Bus Anti-Terrorism Pre-Meeting & Workshop APEC Coping with Bus Terrorism: Learning and Sharing Bes Practices, 24-26 July 2012 in Manila, Philippines 7 Seminar on Regulatory Issues in Cross-Border Services Trade: Ensuring Protection of Consumers and Service Suppliers, Singapore, 27 July 2009 5 APEC Workshop: Improving Public Consultation in the Rulemaking Process, Jakarta, Indonesia, October 29-30, 2009 9 Workshop on Reducing Start-up and Establishment Time of Businesses, Hiroshima, Japan, 1-2 March 2010 6 Policy Round Table on Low Level Presence of Products of Agricultural Biotechnology in Food, Sapporo, Japan, 27-28 May 2010 5 Developing Food Safety Plans for the Supply Chain Module, Beijing, 5-7 November 2010 1 Sixth Conference on Good Regulatory Practice, Washington, 1-2 March 2011 (e-mail is prepared) 12 Using Regulatory Impact Analysis to Improve Transparency and Effectiveness in the Rulemaking Process, Washington, 3-4 March 2011 10 Workshop on Microfinance Best Practices Hanoi, Viet Nam, 7-8 April 2011 7 Conference on the Framework and Options for Public and Private Financing of Infrastructure, Washington, DC, 22-23 June 2011 5 APEC Workshop on Private Sector Emergency Preparedness, Sendai, Japan, 1-3 August 2011 6 Workshop on Low Emissions Development Strategies, San Francisco, 11-12 September 2011 2 Green Buildings and Green Growth: Approaches to Encouraging a Positive Green Building 15 57.1% 39.9% 2.0% 1.0% 0% 10% 20% 30% 40% 50% 60% Very helpful Somewhat helpful Not at all helpful N/A Q6. Please rate how helpful this facilitated workshop was in helping you to work with colleagues and other organizations in your economy to achieve reform objectives 26 Climate, Singapore, 12-13 September 2011 Ease of Doing Business Seminar on Women’s Entrepreneurship, San Francisco, 14 September 2011 (e-mail is prepared) 9 Workshop on Approaches to Assessing Progress on Structural Reform, San Francisco, 19-20 September 2011 (e-mail is prepared) 4 Drug Safety and Detection Workshop, Beijing, China, 27-28 September 2011 8 Workshop to Assess and Improve Agricultural Data Collection and Dissemination by APEC Member Economies, Manila, Phils., 27-28 October 2011 11 Industrial Science and Technology Working Group Strategic Planning Workshop, Kazan, Russia, 28 May 2012 1 Aligning Energy Efficiency Regulations for ICT Products: Developing a Strategic Approach, Seoul, Korea, 18 July 2012 14 APEC’s New Strategy for Structural Reform (ANSSR) Regional Project Development Training Workshop, Lima, Peru, 1-3 August 2012 6 Workshop on Climate Change Adaptation in the Asia-Pacific: Observations and Modeling Tools for Better Planning, Singapore, 16-17 August 2012 4 Technical Assistance Seminar – The Establishment and Use of Accountability Agents in the APEC Cross Border Privacy Rules System, Sendai, Japan, 15 September 2010 7 Response Count 170 Q9. Gender: Answer Options Response Percent Response Count Male 54.9% 96 Female 45.1% 79 Answered Question 175 Skipped Question 37 Q10. Member Economy that You Represent AnswerOptions Economy Australia 1 Brunei Darussalam 3 Canada 3 Chile 11 People's Republic of China 1 Hong Kong, China 2 Indonesia 22 Japan 4 Republic of Korea 8 Malaysia 15 Mexico 11 New Zealand 3 Papua New Guinea 5 Peru 10 The Philippines 14 Russia 4 27 Singapore 9 Chinese Taipei 9 Thailand 10 The United States 9 Viet Nam 17 Response Count 171 Please describe the areas and the nature of the reform or technical skills that you feel you achieved by your participation in this workshop and related activities stimulated by this workshop. APEC Micro-Credit Workshop and Women Entrepreneurs August 2009, Singapore  I acquired some good leadership traits. It also gave me the courage to be more independent in decision making.  Create awareness stage. APEC Workshop on Human Resources Impacts of the Global Economic Crisis July 2010 in Jakarta, Indonesia  This activity is very useful for the progress of Indonesia, both technically as well as inter-state relations.  Area. Innovation in higher education Activities. Research and publication in exploring the nature and effects of innovation in the education program using ALFA-TUNING competence. The main focus is in human capital at higher education.  Benchmarking best practices in managing economic crisis, particularly frontloading of resources to emergency employment programs.  The workshop did provides more insight of other economy on how they responding to their economy especially during the downturn through the development of human capital.  Alert system on global crisis. The Establishment of Use of Accountability Agents in the APEC Cross Border Privacy Rules System, February 2010 in Hiroshima, Japan  Reforms regarding privacy issues and data protection, considering the legal framework, the enforcement authorities, the users and the businesses. I found very helpful the nature of the workshop, as well as other APECs sessions, that includes all the actors I described before; different perspectives, interests and objectives based on the right use of information and consumers protection.  Areas: telecommunications, electronic commerce, data privacy, liberalization. Workshop on Regulatory Approaches to Smart Grid Investment/Deployment, May 16-17, 2012 in Quebec City, Canada  The workshop reinforced the need to engage "local" (state-level) regulators in Smart Grid discussions, not just Federal/national regulators. 28 APEC Bus Anti-Terrorism Pre-Meeting & Workshop APEC Coping with Bus Terrorism: Learning and Sharing Bes Practices, 24-26 July 2012 in Manila, Philippines  I could applied many subjects that I learnt from the seminar to be the Transportation security policy and applied lesson learns to be practices drill my troops.  The presentations at the workshop have provided us with valuable inputs and enabled to learn basic and advance ideas to ensure public transport security. The new ideas provided by the workshop have indeed created awareness among security and public transportations authorities on the important of public safety. This in turn will provide greater opportunities for the continuous development together with good business and security environment.  Risk identification and stakeholder cooperation.  Bus terrorism tactics and procedures. APEC Workshop on Climate Change Adaptation in the Asia-Pacific: Remote Sensing and Modeling Tools for Better Planning, August 2012 in Singapore  Understanding of the available information and resources that may be tapped for defining plans and programs to address climate change issues.  Remote sensing is a good tools for change detection of climate variability.  Need to have a follow up opportunities/key facilitators to drive whatever is suggested and concluded in the breakout sessions. I remember we were to receive a summary report, but haven’t got that as well.  I have learned some useful observation technologies on climate change and its impacts. It is very helpful for our further research plan and work.  Established a network of experts working on the same field from which future cooperative activities may be formed. Seminar on Regulatory Issues in Cross-Border Services Trade: Ensuring Protection of Consumers and Service Suppliers, Singapore, 27 July 2009  Integrity anticorruption.  Best practices that we can look at for domestic implementation.  Trade facilitation. APEC Workshop: Improving Public Consultation in the Rulemaking Process, Jakarta, Indonesia, October 29-30, 2009  Biotechnologies  I gained additional knowledge in writing papers and in teaching.  My ability to create and get involved in the area of public consultation particular to contribute an academic paper in local regulation. Workshop on Reducing Start-up and Establishment Time of Businesses, Hiroshima, Japan, 1-2 March 2010  The workshop I attended was on Starting a Business. One of the things I learned from the workshop is the importance of engaging and getting the commitment of stakeholders in a reform initiative. Also, key to the success of a reform is having M&E activities. 29  That reform can be achieved not only by amending or abolish some 'unnecessary' regulation but also by accommodating some small steps into one step procedure.  The workshop provided input for crafting strategies.  A better understanding of what areas to focus on for reform when trying to reduce the amount of time it takes to register a business.  The workshop promoted Taiwan's government to establish the One-Stop Website. The One-Stop Website (http://onestop.nat.gov.tw) for Online Applications to Start a Business formally came into operation on May 30, 2011. The relevant procedures for checking the uniqueness of a new company’s name, approving the name, registering the company, obtaining labor and national health insurance coverage, and filing work rules can all be conducted on this same website, making the whole process of starting a company much easier and more convenient for members of the public. Policy Round Table on Low Level Presence of Products of Agricultural Biotechnology in Food, Sapporo, Japan, 27-28 May 2010  Involvement of farmers in the decision making improvement of the National Reglamentatio on GMOs.  The workshop helped me understand the LLP issues that can impact on trade, how LLP should be handled and managed and the policies that should be formulated to address the LLP concerns. It also helped in drafting policies and rules on LLP for the Philippines.  GMO development and biosafety. Developing Food Safety Plans for the Supply Chain Module, Beijing, 5-7 November 2010 Sixth Conference on Good Regulatory Practice, Washington, 1-2 March 2011 (e-mail is prepared)  There was good opportunity to exchange experience and learn success with member countries.  As part of the team of the Peruvian WTO-TBT Focal Point, this workshop has helped me increase my knowledge in transparency issues, in impact evaluation, and about the tools that technical cooperation can provide for developing economies.  Engaging regulators and other stakeholders in standards development and use.  Conformity assessment and MRA.  Good regulatory practice and the implementation experience in the other economies.  Good Regulatory Practice (GRP). Using Regulatory Impact Analysis to Improve Transparency and Effectiveness in the Rulemaking Process, Washington, 3-4 March 2011  I could clearly understand the benefits to develop Regulatory Impact Analysis, increasing the effectiveness and efficiency of regulatory outcomes.  Regulatory reforms.  The best practices in implementation of RIA in different economies & the role of Government in championing RIA.  Better understanding on how to implement RIA in various circumstances. 30  That RIA does not have to be expensive step to develop regulation. RIA is also a tool which can be half implemented or fully implemented.  Structural reform process and experience of other countries that already implemented the policy reform. Workshop on Microfinance Best Practices Hanoi, Viet Nam, 7-8 April 2011  Benchmark our Microfinance practices in the Philippines with other APEC economies, esp. the use of latest technology in Micro credit.  Share best practices and policies aimed at diversifying products and services for the poor and to rural areas (Savings, Micro-insurance...)  Different skills practice use at different countries, the concepts and technology use in microfinance level. As being first timer and developing country discussions gained help a lot in implementing in my country.  In Peruvian economy there is not enough financial instrument to support early stage business, in these sense the ministry of production is implementing an esqueme of factoring and also it has elaborated a guaranty program project.  Understanding on micro financing practice at other countries. Conference on the Framework and Options for Public and Private Financing of Infrastructure, Washington, DC, 22-23 June 2011  I supported involvement by Australia in the workshop. Much of it was focused on a speech given by our senior representative outlining our experiences in the area. Our participation in the workshop highlighted that our experiences and challenges are similar to many other countries - including developing countries. It confirmed that there are no 'silver bullets' to solve the infrastructure challenges in many countries around the world. It also highlighted that developing countries should be wary of signing up to many years of payments to private sector providers that may not represent good value for their taxpayers. It confirmed most of our ideas. We are due to shortly discuss these matters with Indonesian colleagues in the Fiscal Policy Office.  How to improve investment environment.  It was a great opportunity to meet other countries' points of contact on infrastructure, and the get private investors together. APEC Workshop on Private Sector Emergency Preparedness, Sendai, Japan, 1-3 August 2011  The promotion of smarter policies on public investment, to prevent disaster risks, and further progress in the development, adoption and implementation of more and better practices for an optimal handling of risks.  It helped to raise consciousness about business continuity planning, not only in a personal level but also in my area of the company (technical).  It helped me to work as a team player with concerned stakeholders in our economy in the business of Geohazards Management.  Gained detailed insight on private sector and public sector emergency preparedness and the importance to have an emergency preparedness plan for each company, organization and economy. 31  The workshop presented me possibility to learn about possible solutions of problems in emergency preparedness, which are already solved in other countries. Workshop on Low Emissions Development Strategies, San Francisco, 11-12 September 2011  Pushing forward smart and micro grid paradigms in Russia and APEC, promoting renewable energy options, promoting statistics and accounting methodologies to measure green growth progress. Green Buildings and Green Growth: Approaches to Encouraging a Positive Green Building Climate, Singapore, 12-13 September 2011  Trade facilitations, Supporting SMEs, Standards and Conformance Education, Standards and Conformance for GreenHarmonization, Regulatory Convergence and Cooperation.  After we returned to Papua New Guinea there is not much dialogue with the Private Sector Women Entrepreneurs like me who attended the APEC Summit, we are a key stakeholder who should be part of discussions and should be participating in partnership with the Govt to bring about much needed reforms etc. Workshop on Approaches to Assessing Progress on Structural Reform, San Francisco, 19-20 September 2011  Prioritization of objectives and developing effective indicators for both evaluating programs and communicating results to citizens.  Engagements with public and stakeholders on the reforms. Benchmarked based on the World Bank Doing Business Report.  How to measure the progress using qualitative and/or quantitative indicators. How to combine external and internal survey which has different methodology to measure progress.  Proper choices of quantitative and qualitative indicators. Drug Safety and Detection Workshop, Beijing, China, 27-28 September 2011  This workshop facilitated to understand in building the ability to plan strategy to reform and also to gain experience and knowledge.  My first time to attend an APEC sponsored workshop and found it very useful especially the need for internal co-operation within developing member states and among others within the A/P region. PNG still lacks the capacity to acquire the technology required to combat the growing presence of counterfeit medicine within the country due to the demand for cheap/affordable medicine in remote areas and this problem is prevalent in the other Pacific island states in the region. Regional cooperation in information sharing and capacity building is the way forward at this point in time.  It was a very important contribution to identify the benefits and limitations of detection technologies in assuring genuine drug products in my country. Basically, I acquired technical skills with my participation and I applied in my professional activity. 32  Update global movement against counterfeit medicines. Workshop to Assess and Improve Agricultural Data Collection and Dissemination by APEC Member Economies, Manila, Phils., 27-28 October 2011  As the statistician, the workshop was helpful. Nevertheless, it was at higher level and should be done in smaller group.  The organizers be would develop a work plan consistent with the global estartegia but not received. In Mexico recently took a meeting with representatives of FAO, which will resume the process.  I (Roque Nochebuena) and Gustavo Tenorio had a participation in the Workshop to Assess and Improve Agricultural Data Collection and Dissemination by APEC Member Economies in October 2011. We have had some extra activities related with it and it´s good for all participants.  I am very interested in the workshop, it helps me to improve my knowledge in how to collecting good data, how to organize data and information, especially I understand the important of data in providing decisions.  Agricultural data collection and data dissemination.  Food security issue. Industrial Science and Technology Working Group Strategic Planning Workshop, Kazan, Russia, 28 May 2012  Basically the ways to encourage stakeholders to participate in the tasks to improve the mechanisms and activities. Aligning Energy Efficiency Regulations for ICT Products: Developing a Strategic Approach, Seoul, Korea, 18 July 2012  Establishment of minimum standard for test methodology establishment of common data set.  Application of national standards in economics, monitoring of the implementation of national standards, examples of cooperation with partner countries.  Knowledge of the future standard that it will applied to the IT products.  I learned about the experience of other countries in regulation of energy efficiency in products IT, this knowledge I am transferring to people that work on the issue of my country.  I became aware of the development of energy efficiency programs, and I hope this issue can be more intense in the future.  I report this information to the management. And plan to continue for the standard, testing lab and CB for future of ICT product in Thailand.  I got very useful information about energy efficiency regulations. But I hope we will discuss more regulations and technical problem in the next meeting. APEC’s New Strategy for Structural Reform (ANSSR) Regional Project Development Training Workshop, Lima, Peru, 1-3 August 2012  Technical agreements for cooperative learning process. 33  Developing a comprehensive framework to analyze structural reforms. However, due to its complexities, we are still working on the project.  To cooperate in one stop shop government model in public service delivery.  Standardization international one stop shop business models. Workshop on Climate Change Adaptation in the Asia-Pacific: Observations and Modeling Tools for Better Planning, Singapore, 16-17 August 2012  More focus on high-resolution global model and down-scaling issue.  It was important to better understand the relevance for many economies (Asia), that sea level rise and storms have on them, in comparison to the importance of crops adaptation in Chile. About remote sensing, was also interesting to know some models to apply to environmental monitoring in order to generate important information for crops adaptation. It could be important to share more information about how to model with remote sensing in agriculture (science basis, confiability of data, images availability).  I feel I have a better understanding of the role of APEC and some of its technical facilities that my country will now make better use of. Technical Assistance Seminar – The Establishment and Use of Accountability Agents in the APEC Cross Border Privacy Rules System, Sendai, Japan, 15 September 2010  Enhancing the technical skill to establish the Accountability Agents in the APEC Cross Border Privacy Rules System.  In depth info about the CBPR system and personal data protection.  Without political will in each economy, all the skill and reform efforts won't be able to be implemented.  It helps to improve my awareness and capacity building of implementing the APEC CBRS, from this point, it contributes to our e-commerce policy making. Workshop Not Specified  The importance of the distributed generation in developed countries. In my economy (developing economy) it is needed more zonal generation reserve. Due my economy is promoted the renewable energy use, it was very useful to see how the renewable energy can be implemented through a mix with other generating sources in order to achieve continuity of service and economy.  Interaction with India's regulators was helpful both in understanding how they will address domestic anti-counterfeiting concerns and how they view their responsibility in the global supply chain.  Share best practices and policies aimed at diversifying products and services for the poor and to rural areas( savings, micro-Insurance ...) -Risk management and portfolio qualify.  Standards and Conformance.  This seems helpful for government only? Not sure how a company should make efforts.  This workshop assisted in the progress toward selecting focus areas for the topic of sustainable construction. It also identified the potential for collaboration with ASEAN nations. 34 SUMMARY OF FIELD VISITS AND INTERVIEWS The objective of the field portion of the APEC TATF contractor performance review was to validate and supplement the available documentary project information and initial interviews of main stakeholders by obtaining feedback from a broad array of key respondents in member economies on their perceptions of the degree to which the project’s strategic objective and intermediate results have been achieved. Based on guidance from initial contacts during the evaluation’s earlier phases, APEC Secretariat staff in Singapore and selected other observers from member economies were targeted as sources of assessments of APEC TATF performance in so-called “Category 1” professional capacity-building activities as described in the PMP intermediate results. Meetings were held with more than forty persons in the APEC Secretariat alone, using differentiated tailored slates of questions. The information obtained in these discussions was complemented with input obtained from selected other respondents designed to elicit comments on their experiences working with TATF on various administrative and institutional strengthening activities. In order to measure performance in PMP “Category 2” activities aimed at advancing APEC’s Bogor Goals and REI through support for APEC’s “Three Pillars,” former and current participants and sponsors of TATF-supported events from member economies were identified to provide assessments as individual respondents and in small groups. See Appendix 10 for a full listing of persons contacted with respect to PMP Categories 1 and 2 activities during this Phase of the evaluation. Also, see Appendix 3 for samples of the prepared agendas of questions that were used to structure the discussions with different communities of respondents. In addition to the in-person interviews, a survey of participants in TATF-assisted events was designed and distributed in parallel with the field work and country visits. All told, the survey was sent to 1327 participants that attended 26 randomly selected seminars/workshops/conferences of the 43 “major” events that TATF facilitated to some degree during the life-of-project through the period in October 2012 when the questionnaire was distributed. The response rate on the survey was approximately sixteen percent. The survey instrument was designed to elicit opinions on the quality of the services and support provided by TATF, and to attempt to assess the utility of the workshop speakers, information, and discussions provided at these events in so far as they informed subsequent measures taken by participants in their home economies to implement or accelerate reforms and actions consistent with the achievement of Bogor Goals. See Appendix 9 for a copy of the survey questionnaire (distributed as a Survey Monkey link embedded in a personally addressed e-mail) and Appendix 8 for the list of seminars/workshops/conferences that were canvassed. Appendix 9 also contains summary quality and content data derived from the 210 partially and fully completed questionnaire returns. Member Economies Visited and Surveyed United States. The Evaluation Strategy and Action Plan approved in September 2012 (See Appendix 2) specified the five APEC member economies that were to be included in this contractor performance review. The United States was surveyed in the first Phase, primarily through telephone interviews with key participants in the inter-agency working group for APEC, chaired by the U.S. Department of State. In addition to introductory discussions with State/EAP/EP coordinators of the APEC relationship, interviews were held with a sampling of 35 officials from the community of USG agencies that most actively participate in APEC project activities. Included were USTR, USDA, U.S. Department of Commerce, and USAID. Through the APEC TATF project, all such involved USG agencies may access TATF to support them to some degree in implementing projects and initiatives carried out in cooperation with other APEC member economies involving APEC Committees, working groups and other technical fora, with which these USG entities cooperate in order to achieve their information-sharing or policy development/reform agendas. These discussions with USG stakeholders yielded a number of broadly shared opinions concerning the responsiveness and effectiveness of the TATF in supporting USG agencies’ projects:  All judged the TATF to be a critical adjunct in achieving their objectives, be it through administrative, operational, or technical support services.  All confirmed that TATF is accountable and performs their assistance duties as agreed.  For many USG agencies, limitations in their international budgets mean that without TATF’s support they risk being unable to participate in important APEC technical fora or unable to drive initiatives in APEC with high value to the USG.  TATF has worked well combining their own staff and externally recruited experts with those provided by USG agencies.  Some USG agency participants admitted frustration with the work planning and approval process, which often leaves decision-making on resource allocation to the eleventh hour.  All found utility in the fact that TATF is embedded at APEC headquarters and has good communications and action reach throughout the Secretariat.  All commended the technical and cooperation skills of the core TATF staff and of the extended roster of consultants that the contractor makes available at all project stages, including for execution of assessments, policy studies, preparing concept papers, and participating in meetings as speakers, experts and facilitators. Singapore – Discussions Inside APEC. Following a period of in-briefing and planning for country visits carried out at RDMA/Bangkok in late September/early October, six days (October 8 - 13) were devoted to visits in Singapore with TATF staff, APEC Secretariat staff, and Government of Singapore officials currently or formerly connected to APEC. See the final Singapore agenda located at Appendix 5. At this point, the evaluation team consisted of the two external consultants, the Deputy COR for the APEC TATF Project from RDMA, the USAID Regional Development Counselor from the U.S. Embassy in Beijing, and two Development Cooperation staff members from the People’s Republic of China’s Ministry of Commerce, Chinese Academy of International Trade and Economic Cooperation (CAITEC), who joined the team in Singapore as observers. Although extensive telephone interviews had been carried out by the evaluation team with DAI/Nathan headquarters and APEC Secretariat-based TATF staff prior to the team’s visit to Singapore, data collection began with a half-day of meetings with APEC TATF staff to clarify historic project and organizational details, to review the preliminary agendas for the country visits, and more generally to familiarize with TATF staff and with TATF structure and linkages inside the APEC Secretariat. It may be noted here that later in the week of Singapore consultations the team also met with the original TATF CoP, who now is working for a private commercial firm in Singapore. He had provided leadership to the Facility from its inception in 2008, until March 2011, and had invested it with many of its operational principles and style. 36 The team found that meeting to be extraordinarily enlightening, especially regarding such organizational precepts as voluntarism in APEC, the non-binding nature of APEC’s “agreements”, the commitment to low operating budgets and limiting the powers of the Secretariat, and perhaps most of all the objective of reducing the development gap amongst the widely varying member economies. Subsequent to the orientation meeting with current TATF staff, the team engaged in a series of structured meetings with individuals and groups in the APEC Secretariat that regularly interact with TATF, including:  the APEC Executive Director, Ambassador Muhamad Noor  leadership and staff from the Program Management Unit (PMU)  approximately half of the member economies’ Project Directors (PDs) then seconded to the Secretariat  leadership and staff of the Policy Support Unit (PSU)  leadership and staff of the Information Technology (IT) Unit  a sample of Program Executives All of these persons and functions have direct links with TATF as counterparts, cooperators and clients under PMP “Category 1” capacity-building activities (with the exception of the Program Executives, who function mostly as administrators of services supporting “Category 2” project activities). All are in a position to judge the quality of the services that TATF provides to the Secretariat – through the work of resident staff and through TATF’s extended network of experts and consultants. Many were able to offer keen insights into the functionality and effectiveness of the TATF embedded platform structure itself. Some highlights of these discussions follow. APEC’s Executive Director (ED), currently completing his third/final year in that capacity, shared with the team his observations and judgments concerning the many ways he turned to TATF for assistance in realizing his vision of upgrading the professionalism of the Secretariat. During his tenure, the ED called upon TATF to support his initiatives on staff evaluation, strategic planning, team-building, developing a comprehensive Secretariat training program, implementing quality project management training for attendees at SOMs, and to upgrade the quality of the project development process. He invited TATF to develop a training-of-trainers program to help the developing member economies put together higher quality project proposals, and he demonstrated his confidence in the resident TATF team by inviting the CoP to participate in his weekly meetings with Secretariat senior staff. The ED’s bottom line on the full-time presence and accessibility of TATF, on the quality and quantity of its contributions, and as a visible manifestation of support by the USG for the goals and objectives of APEC, was to declare TATF, at once, a win-win for both APEC and for the U.S. Government. TATF’s contributions to building capacity across the operational units of the Secretariat are similarly diverse and deep. For the PMU, TATF supported the initial formulation of the APEC Project Guidebook and currently is working to digitalize the Guidebook and project concept note and proposal development form as an interactive on-line tool. TATF has helped in the development of training for members of APEC Committees and working groups to improve the quality of project design and management. TATF cooperated with the Project Management Unit in resourcing a PMU-designed SOW for a searchable project data-base, which, when completed, will be available to all APEC stakeholders, and will integrate operational, financial and project output information, facilitating implementation of multi-year programs and more effective 37 monitoring and evaluation. And for the remaining period of the project, TATF will continue its cooperation with Australia’s AusAID in designing a modern and effective post-project impact evaluation process. In the IT domain, TATF managed the procurement for consultants based in Singapore to upgrade the APEC website content management system, assisted with accounting system data￾sharing that contributes to support of multi-year projects, upgraded e-mail security and capabilities, and trained all Secretariat staff for transition to M/S Office 2007. TATF expeditiously upgraded the Secretariat’s networking capabilities with member economies by quickly and at low cost designing and installing a digital video communications system at APEC headquarters. All told, during TATF’s time at APEC, the Facility has contributed approximately USD 0.6 million to support IT equipment and software upgrades. But IT managers will be quick to point out that TATF’s real impact has been as much through the quality of consultants it has provided, the expedited decision-making that has enabled quick takeoffs on important initiatives that likely would have languished while waiting for institutional funding (through the APEC Budget Management Committee), and through TATF-provided IT training and team-building that have contributed to strengthening relationships both inside the IT establishment and between IT and other Secretariat users. The PSU’s main responsibility is to carry out evidence-based empirical research and analysis to support APEC project initiatives, and to advance and measure member economies’ progress in achieving critical milestones toward Bogor Goals. There is a symbiotic relationship between PSU and TATF. Where TATF has developed special expertise or is deeply involved in APEC initiatives such as Ease of Doing Business (EoDB), structural reform and environmental goods and services, TATF is called upon to contribute from their comparative advantage – and harmoniously with the PSU – to such regular publications as the Economic Committee’s “APEC Annual Economic Policy Report.” Similarly, where TATF is involved, for example, in extensive training of member economy officials in implementation of structural reform, PSU offers support in conceptualizing the training programs and in vetting effectiveness of different training approaches. Another measure of the partnership between PSU and TATF has been their cooperation in leadership on APEC’s behalf of the sectoral “mapping exercise” and other dialog objectives established between APEC and ASEAN to identify areas of complementarity and redundancy in the two organizations’ programs. Although of only limited success thus far in affecting APEC￾ASEAN program harmonization, several candidate sectors including SME, health management with emphasis on pandemics and the harmonization of Customs procedures, have been targeted for possible action, as have more strategic questions such as linkage between the ASEAN Economic Community and the nascent Trans-Pacific Partnership, and the role of the “ASEAN Caucus” inside APEC. APEC TATF, in coordination with ASEAN TATF, have comprised an important circuit for such communications, albeit thus far informally. The PDs, as a community, are responsible for supporting the APEC Committees, technical working groups and other technical fora that take the lead in organizing the myriad workshops, seminars, case studies, diagnostics, and training initiatives of APEC, which TATF is called upon to support as their primary PMP “Category 2” programmatic activities. As has been noted, the PDs are all secondees of their respective member economy governments. There is expected to be one PD for each member economy, though at any given moment there may be as many as two or three (in the case of a small number of more developed member economies such as Australia, Japan and Korea), or possibly none, which may occur due to the high perceived cost 38 to less developed member economies of detailing an officer, or simply due to turnover and transition. The PDs credit TATF with highly useful training for themselves and for the leaders of the various fora they support in areas such as strategic planning and team-building. Overall improvements in success rates in the project approval process – especially for developing member economies – are attributed by PDs to TATF training of project proponents and of TATF-designed proposal process improvements. PDs that the team interviewed all supplied anecdotes of the various ways that TATF had added value to, or facilitated, APEC’s efforts, through deployment of experts performing at the state-of-the-art level and generally functioning as an in-house consultancy. Notably less successful was TATF’s attempt to serve as alchemists to design an APEC personnel appraisal system for PDs under circumstances where PDs’ chain of command runs back to their home economy, not to executive structures in the APEC Secretariat. Singapore – Discussions With GoS Entities. With its movement to engage sources external to APEC headquarters, the team was able to begin triangulating its own formative views on TATF performance and feedback from cooperating APEC Secretariat staff with opinions of officials in the Ministry of Foreign Affairs and the Ministry of Trade and Investments, the two key Government of Singapore (GoS) ministries that engage with APEC. The team benefited from the fact that, as it happened, several of the GoS interviewees had experienced TATF from multiple perspectives, including working in the APEC Secretariat, participating in negotiations held under APEC auspices with TATF facilitation, and as APEC Committee and working group leaders/members representing Singapore who had received project support from TATF.5 Respondents offered unique (and appreciative) insights into how TATF is perceived inside the Secretariat as offering critical human and fiscal resource supplementation for certain functions that a better financed organization might carry out as core functions. On the projects side, as the team was to confirm in later conversations with government counterparts in Indonesia and Vietnam, the GoS representatives were able to report in detail mostly anecdotally on their observations of TATF in a number of workshop encounters in which Singapore had a sponsorship role, including the hands-on facilitation of preparation by developing member economies of structural reform plans. For this series of meetings, featuring intensive work in small groups held in Bali and San Francisco, TATF was given high marks for their coordination of agenda preparation, facilitation of co-sponsorship amongst Singapore, Australia and the United States, and for their implementation of a quality process that married the wisdom of review by experienced peers to inputs from outside expert consultants. Indonesia. The team held a series of meetings with Government of Indonesia (GoI) officials from the Ministry of Foreign Affairs and the Coordination Ministry for Economic Affairs. The team’s Jakarta agenda can be found in Appendix 6. Since TATF’s inception, as power users and intensive participants in APEC activities, Indonesian officials have had a wide range of contacts with TATF – i.e., through being a lead subject of assessments and diagnostics, being a pathfinder economy in new areas, engaging in multi-event sequenced initiatives – and in a wide variety of technical areas, including ease of doing business (EoDB), structural and regulatory reform, public participation in the rule-making process, public sector governance, corporate governance, food 5 The team reports with regret that they were unable to obtain an interview with the Singaporean diplomat, Ambassador Michael Tay, who served as Executive Director of APEC during the start-up of the TATF, and who was a strong advocate for TATF involvement to support Secretariat operations and APEC’s project portfolio. Unfortunately, Ambassador Tay was not in Singapore during the period of the team’s visit. 39 security, and other areas under the SOM Steering Committee on Economic and Technical Cooperation (SCE), which Indonesia chaired in 2012. As Indonesia is slated to accede as APEC host in 2013, they were able to get a head-start influencing directions for their year through such Committee chairmanships as the SCE. Our respondents cited a number of ways in which TATF had assisted them, and in which they intended to call upon TATF for more support during the 2013 Indonesia year. One of the most promising areas mentioned was Indonesia’s intention to broaden the focus of smaller APEC working groups to integrate more directly and at a more strategic level with other fora responsible for complementary issues and thereby to more comprehensively address complex issues less narrowly. Examples included hoped-for TATF support for supply chain connectivity, food security, and disaster resilience and response, all areas where Indonesia has signaled its intention to focus during 2013. As with many other respondents, Indonesian officials noted when describing their aspirations and priorities how much easier and expeditious it is likely to be to gain traction in these new areas using TATF resources than would be the case were they to rely uniquely on the already stretched APEC Secretariat institutional resources. The team also learned of some of the frustrations experienced in the course of enjoying TATF support, when an initiative would come to a close, leaving behind a good product and/or some strong member economy supporters prepared to take on a reform initiative, but where implementation of change could not be made to materialize. In the case of Indonesia, the team was informed of some material advances in understanding of the rule-making process and on starting a business that TATF was instrumental in developing. In the specific case of Indonesia, with so much fragmentation and competition among ministries with overlapping jurisdictions particularly on economic matters, TATF-client champions of reform often found it difficult not only to win the support of other ministries with a stake in a particular sector, but equally achieved limited success in building consensus up the line to their own ministries’ executive levels. Conversations with long-time observers of Indonesia in the U.S. Embassy confirmed that even with further support from TATF and experts from other member economies such as Australia, New Zealand and Korea (such additional support was provided in a few cases), institutional and political forces affecting ministerial cooperation could be limiting factors on reform outcomes, no matter the quality of the support from these supplementary outside forces. Notably, however, the team formed an impression that interventions to support such reform activities could be positively influenced by systematic interventions of high-level USG officials in USAID/Indonesia and the Embassy, or through targeted project support from a well-regarded and flexible research and development support entity such as the USAID/Indonesia SEADI project working in GoI institutions but carrying out their dialogues through parallel informal channels. The team’s discussions of such matters with USAID/Indonesia and U.S. Embassy staff, and with SEADI leadership, while not confirming of this prospect, did not rule it out as a viable tactic for accelerating downstream follow-up to TATF-assisted initiatives that have serious supporters in the member economy. This theme arose similarly in the team’s visit to Hanoi, as will be described below. Vietnam. In a series of meetings with officials from the Ministry of Planning and Investment and the Ministry of Industry and Trade, and with their cooperators from research organizations the Central Institute for Economic Management (CIEM), and Economica, the team encountered a 40 group of cooperators involved in an array of APEC Economic Committee projects. See Appendix 7 for the team’s Hanoi agenda. Particularly in cases where Vietnam was a lead economy in the project, such as the 2011 workshop held in Hanoi to review micro-finance case studies that examined Vietnam, Philippines, Indonesia and Mexico, and the Jakarta public consultation case studies workshop, the regulatory impact assessment workshop in Washington, and a number of structural reform initiatives, respondents made it clear how much the successes of these initiatives resulted from TATF expert technical assistance, training, and facilitation. Importantly, these “successes” covered the whole gamut of outcomes that TATF is held accountable for as objectives under the PMP. TATF experts worked with CIEM on the Vietnam case study, reportedly tapping comprehensively into the large Vietnam micro-finance policy and practitioner network to obtain diverse inputs on all forms of small-scale lending including to women entrepreneurs; assisted in drafting the case study; and coordinated the workshop agenda and speaker presentations. As a result of having key decision-makers in attendance at the workshop, a presentation by proponents from the GoVN of the recommendations from the workshop to the Prime Minister, and the resultant issuance of a government decree, a number of new directives and laws were promulgated regulating the micro-finance environment and the roles of all key players, including the Ministry of Finance, the State Bank, key NGOs and other finance intermediaries. While direct attribution for these outcomes clearly could not be ascribed to TATF’s involvement, interviewees expressed confidence that the quality of inputs, and the workshop itself, were seminal factors leading to the chain of reforms that eventually were implemented. GoVN Ministry of Industry and Trade representatives expressed great enthusiasm and appreciation for the many forms of assistance they had received from TATF, not least the training in quality proposal writing that TATF had offered in conjunction with SOMs and Economic Committee meetings, which has led to greater success for Vietnam in its concept paper-writing and project applications to APEC’s Project Management Unit, and Budget and Management Committee. Occasionally, TATF would contribute to the financing of lower-level Vietnamese officials to attend these meetings in order to receive, inter al., this training. Substantively, these officials credited TATF assistance in organizing Vietnam’s case study for the public consultation process workshop with encouraging a number of formerly non-participating ministries to establish space on their websites to publish consultative drafts ahead of promulgating new laws, regulations and directives. The Ministry of Industry and Trade, itself, adopted procedures for widespread consultations with other government entities, businesses and academics to formulate more transparent and better-informed positions on a number of trade agreements. Similarly, the team’s interlocutors saw the 2011 Assessing Progress on Structural Reform action planning workshop in San Francisco as usefully outcome-oriented. Vietnam attendees credited the small working group and peer review processes facilitated by TATF as helping greatly in devising performance indicators useful for measuring real results rather than mere statistical data gathering. They also claimed to have gained a greater appreciation for how best practices as might be agreed at such meetings often come into conflict with preferences of domestic policymakers in home governmental structures. In the case of Vietnam, it was interesting to learn how such conflicts – particularly concerning trade – could be influenced by interventions of experts under such USAID bilateral programs as Support for Accelerated Trade Reforms 41 (STAR), and Vietnam Competitiveness Initiative (VNCI), which on occasion historically had taken place. The team’s meeting with Embassy Hanoi Econ leadership, in which we discussed the above TATF involvement in Vietnam, along with other engagements on EoDB, corporate governance, the Environmental Goods and Services case study, and the APEC supply chain connectivity “de minimus” exemption initiative for tariff payments, led to exploration of a mechanism for TATF to circularize to Embassies (and USAIDs) more information on their various activities. Both Embassy and USAID contacts confirmed that in many cases they were only just learning that TATF was working on areas on which US diplomatic and development concerns, respectively, similarly were focused. The team found this request for better and more current information on TATF activities to be a major take-away from these consultations. It will be addressed below in the Recommendations chapter of this Report. Thailand. While carrying out its consultations at RDMA in Bangkok, the team opportunistically reached out to interview a sample of Thailand APEC-connected individuals. These included the unit in the Department of International Economic Affairs of the Ministry of Foreign Affairs responsible for APEC oversight, a contact in the Ministry of Agriculture and Cooperatives, and the APEC Industrial Science and Technology Working Group (ISTWG) Chair, affiliated with the Thailand Ministry of Science and Technology Center of Excellence for Life Sciences. The meeting with the MFA group proved to be of little value, as the individuals contacted had only limited association with TATF in any form, and consequently disqualified themselves from offering any assessments of TATF performance. Similarly, the person from the Ministry of Agriculture and Cooperatives, though a participant in APEC meetings on food security and related topics, claimed to be informed only on the contributions of the Thailand Program Director in the APEC Secretariat, who had been her principal point-of-contact on the occasions she attended meetings. On the other hand, the “lead shepherd” (Chair) of the ISTWG advised that, although he had been involved in the APEC working group for nearly a decade, he had contact with TATF only since the 2011 U.S. host year. That association was in the context of TATF technical support to the working group in formulating a roadmap and strategic plan for the group’s activities. As it happened, the Chair’s preference had been to source this assistance from a Thai organization. However, the Thailand floods interrupted coming to agreement with that group in 2011. Subsequently, in 2012 he organized to receive this assistance from TATF. The first phase of TATF’s support for this strategic planning exercise took the form of preparing an agenda and identifying trainers for the strategic planning exercise. This formative phase went well. However, according to the Chair, the quality of the TATF-provided facilitator was deemed inadequate, and his approach too “conventional”. As a result, the implementation of the strategic planning process itself did not go well, allegedly because of the combination of the above-mentioned defects in the consultant, because of the presence of new private sector first-time participants requiring too great a focus on first principles of the working group, and because of a preference promoted by 2012 host Russia for incorporating a new “Policy Partnership in Public Participation” dimension in the working group. These forces collectively caused the effort to be put on a slower track, with TATF dropping out of the picture. This was judged to be the least successful outcome for a TATF intervention that the team encountered in its interviews, but evidently could not have been all that bad, as we were given to understand that the Chair now was proposing more TATF assistance, this time 42 for assistance in team-building to better integrate the more diverse membership now present in the working group. 43 INTERVIEWEES FOR TATF EVALUATION U.S.-BASED CONTACTS INTERVIEWED Name Position Organization Ryan MacFarlane APEC Coordinator State/EAP/EP Deanne De Lima APEC Deputy Coordinator State/EAP/EP Arrow Augerot USTR Eric Holloway USTR Jai Motwane USTR Jeff Gren US Dept of Commerce Dorsey Luchock USDA/FAS Andrew Rude USDA/FAS Amy Searight Senior Advisor USAID/AA/ANE Ann Katsiak TATF Program Officer (HQ) Nathan Associates/DNG 44 CONTACTSINTERVIEWED IN ASIA-PACIFIC REGION, Sept-Oct 2012 Last Name First Name Economy/ Organization Organization/Agency Name Position/Title Noor Muhamad Malaysia APEC Secretariat Executive Director/Ambassador Nii Natalie United States APEC Secretariat Program Director Mailewa Nadira Australia APEC Secretariat Program Director Hunt Adam Australia Program Monitoring Unit Director Honda Yumiko Japan APEC Secretariat Program Director Belevan Diego Peru APEC Secretariat Program Director Sirikul Thanawat Thailand APEC Secretariat Program Director Tuan TuAhn Vietnam APEC Secretariat Program Director Zhiwei Lu People's Republic of China APEC Secretariat Program Director Myung-hee Yoo Korea APEC Secretariat Program Director Gopalakrishnan Gopika Singapore APEC Secretariat Information Manager Chew Daphney Singapore APEC Secretariat Program Executive Mohd Ali Noritabte Singapore APEC Secretariat Program Executive Jeong Jaehoon Korea APEC Secretariat Director Information Technology Go Ronald Singapore APEC Secretariat Information Technology Manager Barron Jose Philippines APEC Secretariat IT Portal Manager Leng Teo Lee Singapore APEC Secretariat IT Administrator Hew Denis Malaysia APEC Secretariat Director, Policy Support Unit Kuriyama Carlos Peru APEC Secretariat Senior Analyst, Policy Sup[port Unit Xiaojing Mao People's Republic of China Department of Development Aid, Chinese Academy of International Trade and Economic Cooperation Deputy Director, Evaluation Observer 45 Shuai Yao People's Republic of China Department of Development Aid, Chinese Academy of International Trade and Economic Cooperation Evaluation Observer Waite Victoria United States APEC Technical Assistance and Training Facility (TATF) Chief of Party Grell Heather United States APEC Technical Assistance and Training Facility (TATF) Deputy Chief of Party Katz David United States Visa Worldwide Pte. Limited Head of Government Relations, North Asia Tan TengTeng Jolene Singapore Ministry of Foreign Affairs Deputy Director, Americas Directorate Lim Craig Singapore Ministry of Foreign Affairs Deputy Director Chua Darrel Singapore Ministry of Foreign Affairs Desk Officer Wee Jun Wen Singapore Ministry of Foreign Affairs Desk Officer Eng Joy Singapore Ministry of Trade and Industry Assistant Director, International Trade Cluster Lim Li Wen Evon Singapore Ministry of Trade and Industry Assistant Director, International Trade Cluster Bahweres Huda Indonesia Coordinating Ministry for Economic Affairs Assistant to the Deputy for Regional Economic Cooperation Permadi Irfan Indonesia Coordinating Ministry for Economic Affairs Assistant to the Deputy for Regional Economic Cooperation Prio Hitono Indonesia Coordinating Ministry for Economic Affairs Assistant to the Deputy for Regional Economic Cooperation Suryodipuro Arto Indonesia Ministry of Foreign Affairs Director, Intra-Regional Cooperation in Asia, Pacific, and Africa ParnohadiningratW ibawa LintangParamitasari Indonesia Ministry of Foreign Affairs Director General, Intra-Regional Cooperation in Asia, Pacific, and Africa Isnomo Kamapradipta Indonesia Ministry of Foreign Affairs Deputy Director for APEC 46 Ringo Saud Indonesia Ministry of Foreign Affairs Director General, Intra-Regional Cooperation in Asia, Pacific, and Africa Azhari Noordin Malaysia ASEAN Secretariat Deputy Chief of Party - Program, ASEAN-U.S. Technical Assistance and Training Facility Buehrer Timothy United States Ministry of Trade Chief of Party, USAID Contractor Uerpairojkit Rutchanee Thailand International Economic Policy Division, Department of International Economic Affairs, Ministry of Foreign Affairs First Secretary Sahussarungsi Sansanee Thailand International Economic Policy Division, Department of International Economic Affairs, Ministry of Foreign Affairs Director Sinprasert Preechaya Thailand Department of International Economic Affairs, Ministry of Foreign Affairs Attache Damrongchai Nares Thailand Thailand Center of Excellence for Life Sciences, Ministry of Science and technology Director Binh Le Duy Vietnam Economics, a Private Consultancy Firm Economist/Policy Analyst ThiHanh Nguyen Vietnam Ministry of Industry and Trade, Multilateral Trade Policy Department APEC-ASEM Division Binh Minh Tran Vietnam Ministry of Planning and Investment, Central Institute for Economic Management Director 47 Ahn Duong Nguyen Vietnam Ministry of Planning and Investment, Central Institute for Economic Management Deputy Director Quynh Mai Pham Vietnam Ministry of Industry and Trade, Multilateral Trade Policy Department Deputy Director General Nehrbass Michael USAID/Indonesia Economic Growth Office Director Fenley Brandon USAID/Indonesia Economic Growth Office COR Jung Adam USAID/Indonesia Program Office Director Carouso James U.S. Embassy/Indonesia Economic Section Counselor for Economic Affairs Wohlauer Benjamin U.S. Embassy/Indonesia Economic Section Deputy Economic Counselor Hamner Todd USAID/Vietnam Economic Growth Office Director Nguyen Thuy USAID/Vietnam Economic Growth Office Program Management Specialist Walker Lisa USAID/Vietnam Economic Growth Office Private Enterprise Officer Stone Laura U.S. Embassy/Vietnam Economic Section Economic Counselor Turner Marybeth U.S. Embassy/Vietnam Economic Section Deputy Chief of the Economic Section Rendon-Labadan Maria U.S. Embassy/Beijing Donor Coordination Regional Coordinator 48 EVALUATION STATEMENT OF WORK Midterm Scope of Work for an External, Participatory Performance Evaluation of the Asia￾Pacific Economic Cooperation Technical Assistance and Training Facility I. BACKGROUND INFORMATION A) Identifying Information 1. Program: Asia-Pacific Economic Cooperation Technical Assistance and Training Facility (APEC￾TATF) 2. Contract Number: EEM-I-00-07-00009-00 3. Order Number: EEM-I-03-07-00009-00 4. Award Dates: September 22, 2008 - March 1, 2013 5: Implementing Organization: USAID/Regional Development Mission Asia (RDMA)/General Development Office (GDO) 6: Cognizant Officer’s Representative (COR): Michael Satin Asia-Pacific Economic Cooperation (APEC) Technical Assistance and Training Facility (TATF) is implemented by Nathan Associates, Inc. On September 22, 2008, USAID awarded a competitive four and a half year contract to Nathan Associates, Inc. for the APEC TATF activity ending March 1, 2013, with a ceiling of $17,657,720. An external participatory evaluation of APEC TATF activities will be conducted from July 2012- November 2012. B) Development Context 1. Program Background and Development Hypothesis USAID designed and implemented the APEC TATF project to improve overall support to the APEC Secretariat and select APEC member economies and economy sub-regional clusters. The intention of this project is to operate a “facility,” which assists APEC and the Secretariat in furthering Regional Economic Integration (REI) to achieve the Bogor Goals1 , including the three broad areas below collectively known as the “three pillars of APEC,” to promote free and open trade and investment in the Asia-Pacific region; by 2010 for developed economies and 2020 for developing economies.  Trade and Investment Liberalization  Business Facilitation  Economic and Technical Cooperation (ECOTECH) This project development hypothesis: << A technical facility housed in a regional body, furthers international trade and investment and supports regional economic integration by providing technical assistance to U.S. foreign-assistance-eligible member economies in a variety of sectors, while improving the Secretariat’s overall ability to appropriately administrate and implement its business plan>> 49 The facility serves as an institutional catalyst for change within the regional body to encourage a movement from the best intensions of management to achieved intended outcomes through actionable work plans. The facility provides a platform to support regional and negotiated commitments to allow APEC to affectively work towards mutually beneficial goals of member economies. Housed in the APEC Secretariat in Singapore, TATF delivers assistance in a variety of forms, ranging from policy studies and assessments to capacity building, training, and expert advice on integration, trade liberalization, and institutional management. The TATF works to improve the Secretariat’s internal structures and processes as well as its information technology systems, improve the efficiency and effectiveness of the Project Management Unit (PMU) in supporting APEC member economies, and improve communications between the Secretariat and member economies. TATF also provides training and technical assistance to both U.S. foreign-assistance eligible member economies and the Secretariat in relation to APEC’s three pillars. Project activities include, but are not limited to, cross-border services trade, corporate governance, public consultation in the rule-making process and micro-credit for women entrepreneurs. The activities under this program enhance Regional Economic Integration (REI) and cooperation within the APEC region, contribute to APEC’s effort to become a stronger and more strategically managed regional institution, and assist APEC and the Secretariat in addressing the issues of the three pillars while working towards a Free Trade Area of the Asia-Pacific. The APEC TATF focuses on sectors of particular importance to the Secretariat, member economies, sub-regional economy clusters and APEC working groups. 2. Target Areas and Groups Strategically facilitated progress in the three areas by the APEC Technical Assistance and Training Facility will enable APEC assistance eligible member economies in the region to further strengthen their economies, pool resources and achieve greater efficiencies. Tangible benefits will also be delivered to targeted citizens in the APEC region through increased training and employment opportunities, greater choices in the marketplace, cheaper goods and services and improved access to international markets. The target areas of this project are both the APEC Secretariat, in terms of capacity building, as well as, support to the APEC assistance eligible member economies. Workshops and trainings support the Secretariat staff as well as senior government officials for APEC U.S. foreign assistance eligible member economies. C) Approach and Implementation & Intended Results The APEC Technical Assistance and Training Facility consists of long- and short-term technical assistance to develop and implement technical capacity building initiatives, including policy studies, assessments, training, and advisory services while taking into account U.S. legal and policy restrictions. Activities funded under this program are intended to directly support APEC priorities, as well as compliment ongoing work by APEC and other donors. D) Existing Data 50 Evaluators will be privy to project work plans, performance management plans (PMP), and the original monitoring and evaluation (M&E) plan. Evaluators will also have access to quarterly and annual reports on the progress of the TATF and deliverables such as project reports, training materials, and additional resources through the COR. II. EVALUATION RATIONALE USAID/RDMA/GDO is conducting a mid-term project evaluation (contractor performance review (CPR)) of the APEC TATF to determine project impact thus far; evaluate contractor performance with respect to compliance with the award by meeting agreed upon project objectives and intermediate results; as well as developing a lessons-learned package of information around which the current program can be shaped and future projects can be designed. Following the completion of the current contract, support to APEC will continue to be important for meeting both U.S. foreign policy initiatives and ongoing development objectives. The mid-term evaluation will identify TATF’s techniques, programs, and workshops that have been successful and should be duplicated or continued in future program design. The evaluation will also identify key weaknesses that USAID should address with future project designs. It is important to note that this is not an evaluation of the Secretariat itself but is instead, an evaluation of the contractor’s implementation of the facility and the direct support it provides to the Secretariat and APEC assistance-eligible member economies. This evaluation will inform additional projects with the APEC Secretariat and will act as a “lessons learned” on USAID engagement with regional bodies. Therefore, the primary user of this evaluation will be the USAID/RDMA/GDO. However, secondary users could include the U.S. Department of State and other USAID regional or bilateral missions that engage directly with international and regional trading bodies. A) Evaluation Purpose This exercise intends to evaluate ongoing programs, improve effectiveness, and inform decisions about the APEC TATF’s current and future activities and the broader context of U.S. assistance to APEC. There are two key elements of this exercise, including a focus on USAID foreign assistance delivery to various APEC stakeholders and how those stakeholders utilize that assistance. The primary objective of this analysis will be to evaluate the engagement and outcomes of the current TATF staff and the contractor unit supporting APEC. The analysis will provide USAID with insight to the efficiency and productivity of the contractor. Also, the evaluation will allow stakeholders to gain insights and reach conclusions about the effectiveness and efficiency of specific activities conducted by the TATF for the Secretariat and participating APEC member economies in order to determine their value-added, both qualitatively and quantitatively. The evaluation will also allow stakeholders to determine the validity of development hypotheses (described above), utility of performance monitoring efforts, factors in the development context that may have an impact on achieving results, and the types of actions that need to be taken to improve performance in the designing and providing of assistance. B) Audience and Intended Uses Stakeholder Information Needs 51 USAID Program/mechanism effectiveness, efficiency, and recommended changes Gauge level of achievement of stated development objectives and identify constraints to achievement. Identify what the program delivered and how does this compares with the program design and objectives/milestones set. Project management and oversight needs. Implementing partner performance. Recommended focus areas for impact in the future. Cooperation or complement to other donor programs. U.S. Department of State Impact achievements of U.S. Foreign Policy goals within U.S.- APEC Dialogue process Recommended focus areas for impact in the future APEC Secretariat Impact on regional integration process and community building C) Evaluation Questions The evaluation team is tasked with addressing three overarching questions, including the following. Additional Questions can be found in Appendix A: 1. How has each APEC TATF program, as well as the implementing partner, changed APEC as an institution? 2. To what extent did the services provided by the facility further international trade and investment and assist regional integration? 3. What are the most important areas of focus for U.S. assistance in the future to achieve the greatest impact given limited budgets and APEC capacity? The evaluation team is expected to include the differential impacts of the projects on male and female beneficiaries. III. EVALUATION DESIGN AND METHODOLOGY A) Evaluation Design An evaluation team (hereafter, the “Evaluation Team” or the “Team”) will examine the current and past performance of the APEC program from inception to evaluation period. While the exercise should evaluate past performance, USAID/RDMA is also interested in forward-looking recommendations on possible strategies for improving the effectiveness of the activities under the APEC program over the remaining period 52 in its current life cycle, as well as broad recommendations for how U.S. assistance to APEC can build upon successes and lessons learned thereafter. The evaluation will address and provide six key areas of performance: 1. Validity and Effectiveness of the Approach. Evaluate the validity of the programmatic approach, including each program’s results framework to achieve its goals. 2. Confirmation of Results. The Team will review the performance and status documentation for each project and additional documentation of results achieved, and comment on the validity of results and the expectations for achievement of indicator targets. 3. Challenges to Implementation. The Team will identify the major challenges to effective implementation that impact program performance and analyze potential mitigation strategies. Challenges should include logistical and management issues, as well as the political context in which the program operates. 4. Lessons Learned. The Team will identify lessons learned and best practices from the initial program period and distill them into clear guidance for the remaining period of implementation. Lessons learned and best practices may include both technical and implementation/management guidance. 5. The Team will provide forward-looking comments and recommendations on opportunities to replicate practices deemed successful under the APEC/TATF program to date to design future U.S. assistance programs for APEC. The Team should also include areas and activities that did not go well￾what sectors were underutilized and in which APEC economies. Finally, the team should explore potential areas or services to develop and expand upon, and where these services are vital. The Evaluation Team will be coordinated by an external, independent consultant (the “Team Leader”). In addition to the Team Leader, the Evaluation Team will consist of: (i) USAID/RDMA staff (from General Development Office and Program Development Office); (ii) representatives from State EAP, depending on availability; (iii) APEC Secretariat staff depending on availability; (iv) and an observation team of project evaluators from the People’s Republic of China (PROC), Ministry of Commerce (MOFCOM), subject to their availability and technical area of interest. The participation on the part of the PROC is an action designed to exchange ideas and best practices on evaluation, engage with Chinese colleagues on project design and evaluation; better understand mutual areas of evaluation interest and deepen U.S. – China relations in the area of foreign assistance and project implementation. All work will be conducted in close coordination with the COR, who will supervise the Team Leader’s work. The COR will provide strategic direction and guidance throughout the evaluation process, including the development of the evaluation report outline, approach, and content. Working through the RDMA COR, the Team will hold consultations and coordinate closely with relevant USAID bilateral missions, U.S. Embassy personnel, and other USG counterparts, as well as with key bilateral and regional development partners. B) Data Collection and Analysis Methods The majority of this effort will be performed through interviews, document reviews, surveys, rapid appraisals, and site visits with the APEC TATF regional and national beneficiaries and partners in the APEC region. The data collection should include identification of the program challenges. 53 The evaluation should gather evidence and highlight the association (if any) between TATF’s reported achievements and outcomes observed. Since the APEC program is complex and involves a large target population and multiple activities, the Evaluation Team will consider a contribution analysis. The evaluation team will attempt to prove that alternative explanations for outcomes have been ruled out or have limited influence. The team leader will not be responsible for the day-to-day direction of U.S. Government team members. Rather, s/he will oversee the overall drafting of the evaluation framework, including methodology determinations; organization of calendar/travel/meetings; coordinating the desk study, interview, survey and other data collection; and analyzing the data with input from team members and USAID/RDMA to draft an evaluation report. C) Observers from APEC Member Economies APEC Member economy observers’ involvement will be a unique element to this evaluation. As part of the new USAID initiative to provide local capacity to government officials, the evaluation of the APEC Secretariat will include constituencies from APEC Member economies sitting in on a subset of focus groups, interviews, and evaluations. The COR will coordinate this effort, and the participating APEC member economies will fund their own participation in the evaluation. This evaluation will act as a training initiative for evaluators in the Secretariat. This should not alter the level of effort outlined below. This evaluation element will be closely coordinated with the APEC Small Working Group on Monitoring and Evaluation (SWGME). IV. EVALUATION PRODUCTS A) Deliverables (See Logistics and Scheduling for a list of activities coupled with deliverables) B) Reporting Guidelines In order to ensure the highest quality reporting, the final report will follow the guidelines below:  The evaluation report should represent a thoughtful, well-presented, well-researched, and well organized effort to objectively evaluate what worked in this project, what did not work, and why.  The evaluation report is expected to be a high quality technical report, in a professional writing style, which can be subjected to peer review and is publishable.  The evaluation report will meet the criteria outlined in USAID’s Evaluation Policy, Appendix B.  Evaluation reports shall address all evaluation questions included in this scope of work.  The evaluation report should include all the key sections: cover sheet, table of contents and acronym list/glossary of terms, executive summary, background, evaluation objectives, main evaluation questions, methods, findings, conclusions and lessons learned, recommendations, and any other sections requested.  The evaluation report should include the scope of work as an appendix. All modifications to the scope of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology, or timeline need to be agreed upon in writing by the technical office.  The evaluation report should include an introduction that adequately describes the project, explains where it is implemented, includes contextual information, and includes the “theory of change” or development hypotheses that underlie the project. 54  Evaluation methodology shall be explained in detail and all tools used in conducting the evaluation such as questionnaires, check lists, and discussion guides will be included in an appendix in the final report.  Evaluation findings will assess outcomes and impacts on males and females.  Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.).  Evaluation findings should be presented as analyzed facts, evidence, and data and should not be based on anecdotes, hearsay, or a compilation of opinions. Findings should be specific, concise, and supported by strong quantitative or qualitative evidence.  Sources of information need to be properly identified and listed in an appendix.  The report must clearly distinguish between conclusions, findings, and recommendations.  Recommendations need to be supported by a specific set of findings.  Recommendations should be action-oriented, practical, and specific with defined responsibility for the action. V. TEAM COMPOSITION Two, independent consultants (also referred to as “Team Leader” and “Analyst”) will coordinate an evaluation team composed of U.S. Government employees, APEC Secretariat employees, APEC Economy representatives, and evaluation observers as identified. The Team Leader must be an Evaluation Methods Specialist. The specialist should ideally have at least 15 years of strong and substantial experience in evaluating6 trade and economic development programming. S/He must have significant professional experience coordinating highly complex evaluations, and leading evaluation teams composed of multiple stakeholders. The specialist must have substantial experience producing and presenting evaluation results and recommendations in a collaborative setting. Candidates must have exceptional organizational, analytical, writing, and presentation skills. S/He must be fluent in English and have a master’s level degree in a relevant analytical field, although doctorate level credentials are preferred. The candidate must have experience evaluating high profile international entities. The Analyst must have the same skills identified for the Team Leader. It would be highly desirable for both to have demonstrated experience working with and/or having intimate knowledge of international bodies, such as APEC, ASEAN, and other Asian institutions, including the East Asia Summit (EAS) and the ASEAN Regional Forum (ARF). VI. EVALUATION MANAGEMENT A) Period of Performance This evaluation will be performed from September-November 2012. The Evaluation Team is expected to visit USAID/RDMA (one trip at the inception and the other to present the final draft report), the APEC Secretariat and selected sites as needed in APEC member economies. 6 USAID Automated Directive System 203.3.6: “A systematic analysis to gain insights and reach conclusions about the effectiveness of specific activities, validity of development hypotheses, utility of performance monitoring efforts, factors in the development context that may have an impact on the achievement of results, and the types of actions that need to be taken to improve performance in the design and provision of assistance.” 55 Approximate Level of Effort for the Team Leader is 59 days, estimated as follows:  Preparation, Review of Documents, In-briefing in Bangkok and Singapore 15 person days  Fieldwork within APEC Secretariat and APEC member economies 20 person days  Draft Report Preparation and Presentation to RDMA 10  Final Report Preparation 8  Final Evaluation Report Presentation to RDMA 5  Washington (DOS and USAID) presentation of Final Evaluation 1 TOTAL 59 person days B) Logistics and Scheduling ACTIVITIES DELIVERABLES Kick-off and Desk Research/Preparation Period 1a) Commencing 5-10 working days after award at USAID/RDMA’s discretion (the “Kick-off”):  Distribution of Kick-off materials to Team Leader: o Key documents of each of three task orders, including but not limited to: (i) the scope of work for the contract; (ii) any monitoring and evaluation provisions contained in the contract; (iii) work plans for the four years of activities; (iv) the semi-annual and/or annual work plans; (v) the periodic progress reports; and (vi) activity reports, success stories, etc. o Interview guide 1b) Post-Kick-off commence drafting of preliminary evaluation strategy:  Following the Kick-off, draft a preliminary evaluation7 strategy (including but not limited to: methodology, evaluation schedule, anticipated contacts, stakeholders to be engaged, stakeholder relevant issues and issues to be covered by stakeholder (Appendix A includes an illustrative list of topics to be covered.) This deliverable will act as a work plan that will need COR approval. 7 USAID Automated Directive System 203.3.6: “A systematic analysis to gain insights and reach conclusions about the effectiveness of specific activities, validity of development hypotheses, utility of performance monitoring efforts, factors in the development context that may have an impact on the achievement of results, and the types of actions that need to be taken to improve performance in the design and provision of assistance.” 56 ACTIVITIES DELIVERABLES o Key APEC documents  Background briefings shall be provided by the RDMA APEC CORs (Background briefings will be completed prior to the in￾briefing at USAID/RDMA) * The Kick-off commences a period of ‘desk research and consultation’. The Team Leader is not expected to be at USAID/RDMA but the Team Leader is expected to be available for consultations, conference calls and otherwise available by electronic means to facilitate preparation for the evaluation. The Team Leader will be expected to initiate engagement and begin coordination with the Team over this period too. ** Approximately, 15 working days are budgeted for the Kick-off and desk research period. Any questions/clarification on timing should be directed to USAID/RDMA. 2) Approximately 8 working days post-Kick-off (timing to be confirmed with USAID/RDMA at Kick-off) forward preliminary evaluation strategy to USAID/RDMA:  Forward the preliminary evaluation strategy to USAID/RDMA approximately 5-6 working days after Kick-off (timing to be confirmed with USAID/RDMA at Kick￾off) and prior to arrival at USAID/RDMA. o RDMA will review the preliminary strategy in consultation with the State Department, prior to the Team’s arrival at RDMA, and provide input into the development of the strategy and facilitate coordination of Team logistics. * The evaluation strategy must be approved by USAID/RDMA before the Team commences with the actual evaluation or represents that the strategy has been approved or is final (the Team will be mindful to avoid omission of 57 ACTIVITIES DELIVERABLES references such as ‘draft’, ‘preliminary’, ‘pending approval, and the like in all discussions of the evaluation strategy). In-briefing Period 3a) Approximately 5-10 working days post￾Kick-off at USAID/RDMA’s discretion (the “In￾briefing”):  Attend an in-briefing at USAID/RDMA, conducted approximately 5-10 working days after the Kick-off *At this point, the Team is expected to assemble at USAID/RDMA in preparation for the evaluation. The timing for the In-briefing will be confirmed by USAID/RDMA at the Kick-off but is subject to change for factors outside of USAID/RDMA’s control. 3b) Present draft evaluation strategy at USAID/RDMA In-briefing:  Review the evaluation strategy (including re-submission of schedule to permit confirmation with RDMA, for travel clearances and appointments with APEC contacts, USAID Mission and U.S. Embassies).  At the In-briefing the latest draft evaluation strategy will be presented to USAID/RDMA. This briefing should include, but not limited to, a PowerPoint presentation highlighting findings and detailed evaluation strategy to date. Feedback on the draft evaluation strategy will be provided by USAID/ RDMA. At the In-briefing, USAID/RDMA may make requests to fill gaps in information, suggest consultations, etc. Evaluation Initiation Period 5) Initiate Evaluation (time to be determined by USAID/RDMA): Conduct the evaluation, which may include, but not be limited to, interviews, consultations, focus group discussions, document reviews, surveys, site visits, etc., as deemed necessary. Mid-Point Evaluation Briefing Period 6) Mid-Point Evaluation Briefing: Provide a mid-point evaluation briefer (either written (not to exceed 5 single spaced pages) and/or tele-or video-conference; method to be selected by USAID/RDMA) to USAID/RDMA to apprise all parties of the Team’s progress. 58 ACTIVITIES DELIVERABLES The anticipated Mid-Point Evaluation Briefing date will be included in the preliminary evaluation strategy presented to USAID/RDMA prior to the In-briefing. The date will be confirmed no less than 10 working days prior to the anticipated date for the briefing. Post-Evaluation Preliminary Draft Review Period 7) Preliminary draft evaluation report:  Present preliminary draft evaluation results (key findings and recommendations in report and presentation form) to the USAID/RDMA GDO Director, Program COR, and other USAID/RDMA staff after field visits for review and comments. USAID/RDMA may circulate the draft for review to the State Department and other relevant APEC stakeholders seeking comment, if the documents are deemed to be sufficiently developed for productive stakeholder dialogue.  USAID/RDMA may share the draft report, in whole or part, with its own experts and, at USAID/RDMA’s discretion, with other regional experts for comment. Comments will be discussed with the Team Leader for incorporation, in the final version, as appropriate. 8) Update preliminary draft evaluation report per USAID/RDMA guidance:  USAID/RDMA feedback will be incorporated into an updated draft evaluation report 1-2 business days following the presentation of the draft evaluation results and circulated to USAID/RDMA. Final Evaluation Report and Out-Briefing Period 59 ACTIVITIES DELIVERABLES 10) Final Evaluation Report developed at USAID/RDMA (the “Final Report”):  The Team will finish drafting the final evaluation report at USAID/RDMA, in consultation with USAID/RDMA, incorporating, or responding to, comments from relevant stakeholders (comments and responses may be appended, as necessary), with overall findings/conclusions and recommendations to be presented at an out-briefing at USAID/RDMA.  The Final Report should be no longer than fifty (50) pages excluding annexes. The Final Report will be provided to USAID/RDMA in an electronic version in Microsoft Word format and an Executive Summary in presentation form (PowerPoint).  The final evaluation report will at the minimum include the following contents: a) Executive Summary; b) Scope and Methodology Used; c) Important Findings (empirical facts collected by evaluators); d) Conclusions (evaluators’ interpretations and judgments based on the findings); e) Recommendations (proposed actions for management based on the conclusions); and f) Lessons Learned (implications for future designs). 11) Final Evaluation Report presented (the “Out-briefing”):  The Final Report will be presented to USAID/RDMA (attendees and timing will be arranged by USAID/RDMA in coordination with the Team Leader).  Present final evaluation report to USAID/Washington and to the U.S. Department of State in Washington, D.C. 60 ACTIVITIES DELIVERABLES (1-2 weeks following the out-briefing at USAID/RDMA; timing to be discussed with USAID/RDMA).  Following the out-briefings in Washington, D.C., the contract is considered complete upon relinquishment to USAID/RDMA of all notes, files, reports, impressions and all other materials (whether hard or soft copy) used in the preparation and development of the evaluation report. The Team Leader will not retain any files related to this evaluation once the evaluation is deemed complete by USAID. A flash drive will be submitted with all soft data information including all instruments and data in formats suitable for reanalysis. (One additional day of LOE)  The report will be uploaded by USAID/RDMA/PDO to the public area of the Development Experience Clearinghouse within three months of being finalized. Appendix A (of Statement of Work): Additional Questions to Consider Sub questions to question 1): Were identified changes expected and did they exceed expectations? Did these changes help meet development objectives set forth by the Secretariat? Did the TATF demonstrate a comparative advantage in certain sectors? Are there sectors that the Secretariat and assistance eligible member economies would have liked to see more or less engagement? What are the overall comparative advantages for the U.S. government in terms of development assistance to the APEC Secretariat and assistance eligible member economies? What are the development gaps among APEC member economies? 61 Are program interventions reaching/benefiting all sectors of the target population and their identified needs? What is the perception of the program to individuals in and outside target population? Do we need to look at any cross cutting areas – i.e. how activities impact environmental, health issues?