FINAL EVALUATION REPORT MID-TERM PERFORMANCE EVALUATION OF THE PRIVATE SECTOR DEVELOPMENT ACTIVITY August 10, 2018 This publication was produced at the request of the United States Agency for International Development. USAID Contract No. AID-486-I-14-00001 Order No. 72048218F00002 TABLE OF CONTENTS Abstract................................................................................................................................................................................. i Executive Summary ........................................................................................................................................................... ii Activity Background...................................................................................................................................................... ii Evaluation Purpose and Questions........................................................................................................................... ii Evaluation Methods and Limitations......................................................................................................................... ii Findings and Conclusions........................................................................................................................................... iii Recommendations ....................................................................................................................................................... vi Activity Background..........................................................................................................................................................1 Evaluation Purpose and Questions................................................................................................................................3 Evaluation Purpose........................................................................................................................................................3 Evaluation Questions....................................................................................................................................................3 Evaluation Methods and Limitations..............................................................................................................................4 Data Collection Methods............................................................................................................................................4 Data Collection Sites and Respondents ..................................................................................................................5 Sampling...........................................................................................................................................................................6 Analysis............................................................................................................................................................................7 Quality Assurance.........................................................................................................................................................8 Limitations.......................................................................................................................................................................8 Dissemination and Utilization Plan............................................................................................................................9 Findings and Conclusions...............................................................................................................................................10 Evaluation Question 1................................................................................................................................................10 Evaluation Question 1. A...........................................................................................................................................15 Evaluation Question 2................................................................................................................................................18 Evaluation Question 3................................................................................................................................................21 Evaluation Question 4................................................................................................................................................24 Recommendations...........................................................................................................................................................30 Strategic.........................................................................................................................................................................30 Evaluation Question 1 and 1a ..................................................................................................................................31 Evaluation Question 2................................................................................................................................................32 Evaluation Question 3................................................................................................................................................32 Evaluation Question 4................................................................................................................................................32 Annexes.............................................................................................................................................................................34 Annex 1: Evaluation Statement of Work ..............................................................................................................34 Annex II: Data Collection Tools..............................................................................................................................41 Annex III: Disclosure of Conflict of Interest Forms...........................................................................................59 Annex IV: Evaluation Team Member Profiles.......................................................................................................63 Annex V: Bibliography................................................................................................................................................66 Annex VI: List of Respondents.................................................................................................................................67 Annex VII: Monitoring and Evaluation Additional Findings and Recommendations....................................77 Annex VIII: Summary of Findings, Conclusions, and Recommendations.......................................................79 Annex IX: Additional Gender Findings and Conclusions..................................................................................80 TABLES AND FIGURES Table 1: Summary of Respondents by Category........................................................................................................6 Table 2: Catalyzing Economic Reform Indicators....................................................................................................10 Table 3: Effectiveness of Catalyzing Economic Reform Approaches...................................................................15 Table 4: Dropped or Re-Aligned Issue Areas...........................................................................................................16 Table 5: Added Issue Areas...........................................................................................................................................16 Table 6: Capacity Building Indicators..........................................................................................................................18 Table 7: Capacity Building Effectiveness by Approach and Issue Area ...............................................................21 Table 8: Entrepreneurship Indicators .........................................................................................................................21 Table 9: Effectiveness of Grants by Objective and Issue Area .............................................................................24 Table 10: Access to Finance and Mobilization of Savings Indicator....................................................................25 Table 11: Effectiveness of Facilitating Access to Finance Approaches................................................................29 Table 12: Evaluation Team Roles and Responsibilities............................................................................................64 Table 13: Cross-Cutting Considerations Indicators................................................................................................80 Figure 1: Development Hypothesis of Private Sector Development Activity.....................................................2 ACRONYMS A-Bank Ayeyarwaddy Farmers Development Bank ACTI ASEAN Connectivity through Trade and Investment ADB Asian Development Bank ADS Automated Directive System ASEAN Association of Southeast Asian Nations A2F Access to Finance CBM Central Bank of Myanmar COP Chief of Party CSO Civil Society Organization DCA Development Credit Authority (USAID) DDG Deputy Director General DEAR Development for Environmentally Friendly Agriculture and Rural Life DEC Development Experience Clearinghouse DG Director General DOA Department of Agriculture DOCA Department of Consumer Affairs DOT Department of Trade DRI Department of Research and Innovation DRI Development Resources International EdM Entrepreneurs du Monde EG Economic Growth EQ Evaluation Question EQUI Evaluation Quality, Use, and Impact ET Evaluation Team IFI International Finance Institutions IPR Intellectual Property Rights FDA Food and Drug Administration FGD Focus Group Discussion FOSTA Food Science and Technology Association FRD Financial Regulatory Department FTF Feed the Future FY Fiscal Year GIZ German Agency for International Cooperation GOM Government of Myanmar HQ Headquarters IFI International Finance Institutions IP Implementing Partners IPPAM Intellectual Property Proprietor’s Association of Myanmar IPR Intellectual Property Rights IT Information Technology KII Key Informant Interviews LIFT Livelihoods and Food Security Trust Fund LOLC Lanka Orix Leasing Company MAB Myanma Apex Bank MBA Myanmar Business Answers MCU Myanmar Consumers Union MDI Myanmar Development Institute MDP Myanmar Development Partners M&E Monitoring and Evaluation MES Myanmar Engineering Society MFI Microfinance Institution MEG Myanmar Gem Enterprise MMK Myanmar kyat MMFA Myanmar Microfinance Association MOC Ministry of Commerce MOE Ministry of Education MOPF Ministry of Planning and Finance MONREC Ministry of Natural Resource and Environment Conservation MSME Micro, Small and Medium Enterprises MYEA Myanmar Young Entrepreneurs Association NGO Non-Governmental Organization NLD National League for Democracy PE Performance Evaluation PGMF PACT Global Microfinance Fund PSDA Private Sector Development Activity PSSAG Pyu Sin Saydana Action Group SI Social Impact, Inc. SME Small and Medium-sized Enterprise SPS Sanitary and Phytosanitary Standards TA Technical Assistance TBT Technical Barriers to Trade TL Team Leader UMFCCI Union of Myanmar Federation of Chambers of Commerce and Industry UNOPS United Nations Office for Project Services US United States USAID United States Agency for International Development USG United States Government WTO World Trade Organization YWCA Young Women’s Christian Association i | USAID PSDA MID-TERM EVALUATION USAID.GOV ABSTRACT This report is a mid-term performance evaluation (PE) of the United States Agency for International Development (USAID)/Burma Private Sector Development Activity (PSDA), a $23 million five-year project designed to 1) create more inclusive and transparent economic governance rules and processes and 2) increase access to finance for emerging economic actors. PSDA began operations in mid-2015 and is being implemented by Nathan Associates. Beneficiaries of the activity include civil society organizations (CSOs), microfinance institutions (MFI), associations, entrepreneurs, private sector actors, and the Government and people of Myanmar. USAID designed PSDA to be an economic policy reform and institution-building operation with an emphasis on promoting evidence-based, public-private policy dialogue. At the same time, the Activity delivers specific, rapid-response technical assistance (TA) to foster important and evolving policy and legislative reforms. This report answers the following evaluation questions: 1. Which approaches to catalyze economic reforms have been most effective and least effective, based on progress achieved toward intended results? 1a: Which factors influenced the achievement or non-achievement of intended results? 2. To what extent have the approaches to capacity building of the public/private sector made progress toward intended results? 3. To what extent have the grants contributed to increasing broad-based economic opportunities? 4. Which approaches to achieve the objective of improving financial services have been most effective and least effective, based on progress achieved toward intended results? The methods used in this qualitative PE consist of document review, key informant interviews (KII), and focus group discussions (FGDs). Overall, PSDA is a well-managed Activity operating in a complex environment. After two and a half years of intensive work in a slow-moving and opaque policy reform context, PSDA has achieved concrete policy reform success in access to finance, which has led to an increase in broad based economic opportunities, primarily for women. PSDA has also succeeded in improving the capacity of a variety of CSOs to engage in policy dialogue with relevant Government of Myanmar (GOM) ministries. It is recommended that PSDA capitalize on current momentum in a focused number of issue areas for the remainder of the Activity by leveraging key relationships with committed and capable individuals within the GOM to further economic policy reform and access to finance. USAID.GOV USAID PSDA MID-TERM EVALUATION | ii EXECUTIVE SUMMARY Social Impact, Inc. (SI) is pleased to present this mid-term performance evaluation (PE) of the United States Agency for International Development (USAID)/Burma Private Sector Development Activity (PSDA). The primary audience of this PE is USAID/Burma, particularly the Economic Growth (EG) Office, which seeks to use the evaluation findings, conclusions, and recommendations to inform strategic and programmatic decisions as well as dissemination to the Implementing Partner (IP) and other stakeholders. ACTIVITY BACKGROUND Given the challenges and the associated opportunities to support economic policy reform in Myanmar, the EG Office at USAID/Burma established PSDA with a five-year budget of $23 million. Launched in 2015 and implemented by Nathan Associates, PSDA focuses on 1) creating more inclusive and transparent economic governance rules and processes and 2) increasing access to finance for emerging economic actors. Beneficiaries of the activity include civil society organizations (CSOs), microfinance institutions (MFI), associations, Government and people of Myanmar, entrepreneurs, and private sector actors. USAID designed PSDA to be a rapid-response economic reform and institution-building operation with an emphasis on promoting evidence-based, public-private policy dialogue. At the same time, the Activity delivers specific technical assistance (TA) to foster important policy and legislative reforms. EVALUATION PURPOSE AND QUESTIONS The PE is intended to help determine what Activity components are or are not working well and why, and to make modifications and mid‐course corrections, if necessary. This report answers the following questions posed by USAID/Burma: 1. Which approaches to catalyze economic reforms have been most effective and least effective, based on progress achieved toward intended results? 1a: Which factors influenced the achievement or non-achievement of intended results? 2. To what extent have the approaches to capacity building of the public/private sector made progress toward intended results? 3. To what extent have the grants contributed to increasing broad-based economic opportunities? 4. Which approaches to achieve the objective of improving financial services have been most effective and least effective, based on progress achieved toward intended results? EVALUATION METHODS AND LIMITATIONS The evaluation team (ET) utilized a mixed-methods approach including multiple qualitative data collection methods and analysis of quantitative Activity data. Field visits occurred in Yangon and Nay Pyi Taw. Data collection methods included: • Documentation review of Activity documents as well as non-Activity related secondary sources. • Key informant interviews (KII) with USAID/Burma and PSDA personnel, PSDA grantees, representatives of several PSDA governmental and partner non-governmental organizations (NGOs), several donor representatives, and several MFI and Banks. • Focus group discussions (FGD) with PSDA trainees at the Ministry of Planning and Finance (MOPF), Ministry of Agriculture, Livestock and Irrigation, Banks/MFI, Myanmar Engineering Society (MES), media personnel, and entrepreneurs/tech experts. Interviewees and site visits were selected in consultation with both USAID/Burma and PSDA. Methodological and logistical limitations to the evaluation design included possible response bias, recall bias, selection bias due to non-random sampling, and the brevity of the data collection period (less than iii | USAID PSDA MID-TERM EVALUATION USAID.GOV three weeks); however, a substantial number of interviews were conducted according to a rigorous schedule (totaling 157 participants: 87 men and 70 women).1 The ET made attempts whenever possible to ensure gender balance among interviewees, and to ask interview questions that specifically addressed gender issues as per USAID policy. Information and perspectives obtained were triangulated through additional interviews and data sources. FINDINGS AND CONCLUSIONS EVALUATION QUESTION 1 AND 1A PSDA utilized multiple approaches to ‘catalyze economic reform’ including engagement with Government Champions2, engagement with Non-Government Champions3, development of technical analysis and reporting, issuance of grants, and capacity building. These approaches, at the mid-term, were found to influence Activity results in varying degrees. The most effective approach PSDA used to achieve intended results has been the identification, development, and support given to Government Champions. This approach most directly resulted in policy gains ranging from improved policy inputs to draft legislation. Another approach that a majority of respondents noted was effective, but one PSDA has not utilized, is embedded advisors. Multiple respondents from the IP, USAID, and International Finance Institutions (IFIs) noted that this approach has been useful in the past and can be useful in the Myanmar context (e.g. a respondent from the Financial Regulatory Department (FRD) explained that the FRD had requested support for a long-term Information Technology (IT) advisor for their department). The leadership of the Ministry of Commerce (MOC) also informally expressed interest in a long-term advisor). Respondents noted that this approach is especially impactful when the specific government ministries in question directly request assistance and have a say in the selection and approval of the individual(s) intended to become the embedded advisor. Other donors [e.g. Asian Development Bank (ADB), German Agency for International Cooperation (GIZ) (Access to Finance), and the World Bank] have also used embedded and/or committed long-term advisors with positive results in terms of providing sustained capacity building support as well as advice in economic policy reform (such as increased tax collection) and banking regulations. While short-term capacity building and grants to CSOs were effective at improving institutional strength and individual capacity and knowledge, at the mid-term these efforts had not yet directly resulted in economic policy reform (noting that the former is their mandate and not specifically the latter). The results from the use of Non-Government Champions were limited primarily due to the Champions’ lack of power and inability to link directly and be heard by a counterpart ministry official or a government champion. This was achieved when a Non-Government Champion from the Myanmar Microfinance Association (MMFA) was able to gain an audience with the government champion at the FRD of MOPF to make the wishes and concerns of the MMFA member MFIs heard and incorporated into legislation that eventually became law. The development of technical analysis and reports were overall found ineffective at directly promoting policy reform; however, when these outputs were developed in response to a specific request by a stakeholder, they were regarded more highly by Government of Myanmar (GOM) respondents regardless of the quality of information and potential value the report may have (e.g. Entrepreneur White Paper and the Agriculture White Paper). 1 Several respondents noted above participated in both KIIs and FGDs (14 in total). Additional details are included in the report about evaluation respondents. 2 Counterparts in GOM that are particularly committed to the Activity effort and eager to promote policy reform. 3 Non-Governmental stakeholders that are particularly committed to the Activity effort and eager to promote policy reform. USAID.GOV USAID PSDA MID-TERM EVALUATION | iv In addition to PSDA approaches, three factors were found to influence intended results related to catalyzing economic reform. First, PSDA’s flexible and adaptive activity design, and related monitoring and evaluation (M&E) plan, led to a broad focus on multiple issue areas, which may have led to slower progress made in promising initiatives given that issue areas added later in the activity received less time to achieve results. The Activity was intentionally designed this way at the outset given the situation in Myanmar and the lack of USAID institutional experience in the country at the time. At the Activity inception stage, there was no way to accurately gauge the true situation in each issue area. Over the Activity’s period of implementation up to this midterm, however, knowledge was generated that now provides more evidence regarding potential issue areas for the focus of reform efforts. Second, while capacity of stakeholders and activity staff in addition to re-organization of Ministries (Ministry of Science and Technology with the Ministry of Education – Standardization and IP) delayed achievement of results, IP and USAID respondents noted positive collaboration and successful coordination with each other. Third, factors in the enabling environment such as the slow and opaque policy-making process, and multiple donors in the same sector have particularly complicated PSDA implementation and led to inefficiencies. In some cases, it may have been possible to better coordinate with other donors to achieve direct buy-in from the relevant GOM ministry (e.g. United Nations Office for Project Services/Livelihoods and Food Security Trust Fund (UNOPS/LIFT) – agriculture sector). EVALUATION QUESTION 2 Trainings, mentoring, and interactive consultation have been the primary form of PSDA capacity building efforts since Activity inception. These efforts were designed with two objectives: 1) improved knowledge/awareness/practices and 2) policy reform. As detailed in Evaluation Question (EQ)1, capacity building efforts were found to be only moderately effective at promoting policy reform (objective 2) at the mid-term. Notable exceptions include PSDA’s work in the following issue areas: the advancement of microfinance sector/mobilization of credit, food safety, trade, Intellectual Property Rights (IPR), and role of media. Additional efforts that have advanced the long-term goal of influencing policy reform were found in the issue areas of ‘quality standards’ and potentially at ‘Myanmar Development Institute (MDI) development’ (which is in its earliest stage). Capacity building efforts have not positively influenced policy reform at the Union of Myanmar Federation of Chambers of Commerce and Industry (UMFCCI) and the agricultural sector, although consumer rights systems have improved and capacity building efforts in consumer protection have resulted in a draft law in parliament and new SOPs in use. Effectiveness on the ground will hopefully improve with the introduction of a new revised law on consumer protection that PSDA is supporting. Efforts were found to be less successful due to the lack of a connection between the CSO and the relevant government body as well as an overall lack of understanding concerning the subject matter at both the GOM and CSO level. Capacity building was found to be effective/successful at objective 1 or ‘improving knowledge and awareness’ particularly in the microfinance sector/issue area. Respondents/trainees explained that trainings were targeted to address the needs of their organizations and were tailored to the specific capacities of the attendees in a language they could understand, using appropriate media. Importantly, trainings with the FRD of MOPF first included the introduction of a comprehensive computer and internet package, followed by targeted trainings on identified needs/issue areas. Capacity building was found to be ‘in progress’ in trade (customs, trade portal) and Small and Medium-sized Enterprises (SME) financing sector related to objective 1. The ET could not fully assess training effectiveness of ongoing trainings (i.e. how the knowledge gained is being applied and how this contributes to higher objectives), but it is clear many advances have been made thus far in terms of increased knowledge and awareness, especially considering Myanmar’s low starting point. For trainings to have a lasting effect, however, they must be protracted and there must be follow-up to ensure trainees are using what they have learned correctly. Awareness has been achieved in food safety, IP, and consumer rights, to name a few. Trainings on quality standards have been successful at transferring new knowledge and skills to trainees. PSDA has v | USAID PSDA MID-TERM EVALUATION USAID.GOV provided four years of support to build the capacity of the Department of Research and Innovation (DRI) for back standardization and metrology resulting in improved staff capacity. Despite this progress, challenges remain. Basic computer literacy does not exist at many departments in key ministries. Additionally, application of training knowledge to day-to-day work can only serve to enhance the capacity of ministry personnel. At the midway point in the project, it is unclear whether increased staff knowledge and capacity will directly translate to economic policy reform, but it is certainly a positive development. When the targeted CSOs are linked to and accepted by counterpart ministries in the GOM, the likelihood of achieving policy reform and impact has proven to increase markedly (e.g. MMFA). EVALUATION QUESTION 3 To date, PSDA has awarded 22 grants to CSOs and MFIs for a total amount of Myanmar kyat (MMK) 2,301,944,697, approximately $1,730,785. Most grants were directed at the provision of capacity building and training efforts. Grants at the mid-term were found to be 1) ineffective in the promotion of economic policy reform, 2) somewhat effective in the building of capacity of NGOs to engage in policy dialogue, 3) largely effective in providing NGO members with better knowledge for them to take advantage of economic reforms, and 4) ineffective in increasing broad-based economic opportunities. The most effective grants in terms of contributing to increasing broad based economic opportunities are in the access to finance issue area. Respondents noted that economic benefits resulted from grants to a business incubator (Phandeeyar) and operators in the agricultural sector [Development for Environmentally Friendly Agriculture and Rural Life Myanmar (DEAR Myanmar), Development Resources International (DRI)] have reached the grantees themselves (and their direct beneficiaries, who have received small-farming income-generating opportunities) but have not yet extended beyond those directly receiving the grants. These agriculture sector grants also notably created economic opportunities for women to engage in farming. Regarding the management of the grants component, respondents from the IP noted that it has been difficult and time-consuming to find viable entities that qualify to receive PSDA grant funds in Myanmar. IP staff further explained that the vast majority of PSDA grantees lack a serious plan to generate sufficient funds to remain operational without continued donor funding. In this context, while progress has been noted above, it may not be sustainable if organizations do not generate sufficient cash flow to cover their operational expenses. Given the situation in Myanmar, it is worthwhile to make the initial steps to support early stage CSOs despite the risk of dependency on donor funds. EVALUATION QUESTION 4 PSDA utilized multiple approaches to ‘improve access to finance’ since its inception including MFI sector policy advocacy; deployment of the Development Credit Authority (DCA); capacity building at MFIs and Banks, the FRD, and MMFA; and to a lesser degree support of Angel Investing. These approaches, at the mid-term, were found to influence the results in varying degrees. At the mid-term, PSDA has achieved intended results related to ‘improving financial services and access to finance’ revealing adequate progress in increasing access to credit but little to no progress yet realized toward the goal of increased access to savings (insurance). Progress in increasing access to credit reveals both a high demand for credit in the country and the faster-than￾expected uptake of credit by PSDA supported MFI borrowers and the market in general, while the lack of progress in increasing access to savings was largely attributed to the slow speed at which the GOM has moved to reform the insurance sector in Myanmar. To its credit, PSDA has provided TA to advance the USAID.GOV USAID PSDA MID-TERM EVALUATION | vi liberalization of the insurance sector and has provided support to reform the insurance law. They have also made efforts to build capacity to conduct risk-based insurance supervision and endeavored to support regulations to allow for the increase in savings deposits by MFIs, which would be the foundation for increased savings among resource-poor families. It is hoped these efforts will yield results over the second half of the project. The most effective PSDA approaches to achieve these gains in increasing access to credit were policy advocacy in the microfinance sector and capacity building at MMFA. PSDA’s microfinance sector policy advocacy work has been a resounding success resulting in four major new regulations leading to increased access to finance for those in need. This issue area has vastly outperformed other PSDA efforts in terms of achieving intended results and contributing to policy reform at the mid￾term. Capacity building at FRD was also noted by respondents as effective toward this goal. These approaches most directly resulted in relevant access to finance gains ranging from improved policy inputs to draft legislation as well as enhanced policy advocacy and dialogue capacity. While the deployment of the DCA loan guarantee scheme was viewed by some respondents as having a positive impact on improving financial services, the impact was not widespread at the mid-term relative to the whole microfinance sector in Myanmar given the relatively small amounts guaranteed. While short￾term capacity building to MFIs and Banks were effective at improving institutional strength and individual capacity and knowledge, at the mid-term these efforts had not been effective in improving access to finance given the fact that PSDA’s efforts have not yet directly resulted in the deployment of new loan products. PSDA supported a few small workshops on Angel Investing and the private sector leadership did not follow this investment methodology. Despite the success of Phandeeyar as a business incubator and angel investor (although PSDA did not support them with the intention of their providing financial support to entrepreneurs), the support was ineffective at promoting entrepreneurship beyond the relatively few individuals who received direct support. RECOMMENDATIONS All recommendations are targeted at the IP, Nathan Associates, and USAID/Burma PSDA staff. STRATEGIC • Focus on building capacity in targeted sectors first through digitalization of GOM offices and optimally at ministries that have identified Government Champions of reform in place, using embedded advisors as and when it is feasible and has been requested by the ministry in question. • Continue the effort to focus on fewer issue areas and interventions with the goal of having a deeper impact on the general population. • Be conservative and eschew early stage and start up initiatives which have not proven to be viable (e.g. MDI – consider small scale support at the initial stage and only continue if tangible results are seen in the near term as a direct result of the support.) • Continue approaches which serve to help in accessing and building relationships with Government Champions e.g. through requested TA, capacity building, and training given for targeted ministries; requested technical analysis and reports; and through requested and approved embedded advisors. Should a person not exist that fits the description of a government champion, PSDA should work to link with and develop those who may be considered to have the highest potential to advance policy reform. • Focus top priority near-term efforts on following up on in-progress policy work concerning insurance reform, IPR law passage, and gemstone policy legislation passage into law. vii | USAID PSDA MID-TERM EVALUATION USAID.GOV EVALUATION QUESTION 1 AND 1A • Continue to develop strategic relationships with committed Government Champions in economic reform likely affecting the broadest population who work in targeted ministries where important positive policy change is most likely to occur, preferably in the near-term. • Consider placement of embedded advisors in ministries where there is a demand to build capacity, improve subject matter expertise, establish trust, and form relationships with individuals (Government Champions) who work in targeted departments where important positive policy change is most likely to occur. • Continue to develop strategic relationships with committed Non-Government Champions for economic reform (individuals) in NGOs/CSOs working in targeted issue areas where important positive policy change is most likely to occur as a secondary priority strategy. • Continue to engage in short-term capacity building efforts only where it has proven to work in the past, and primarily where it will have the highest probability of leading to economic policy reform, namely at government offices working in targeted issue areas where important positive policy change is most likely to occur. This should not be a main focal point in the strategy, and when done, should be done in a strategic way (see EQ2). • Continue to engage in grants to CSOs only where it has proven to work in the past, and primarily where it will have the highest probability of leading to economic policy reform, namely at the few CSOs/NGOs where grants have achieved intended results, e.g. MMFA. This should not be a main focal point in the strategy, and when done, should be done in a strategic way (see EQ3). • Produce technical analysis studies and reports only when it has been specifically requested by a prominent person in GOM and where it will have the highest probability of leading to economic policy reform. On occasion, it may be a worthwhile investment to conduct a brief study in a high priority issue area (e.g. economic benefits to improved access to finance for SMEs) that has not been considered yet by GOM. Any reports should elucidate key subject matter and be succinct, with reasonable and attainable recommendations including steps to enhance the likelihood of achieving results. EVALUATION QUESTION 2 • Design capacity building/training plans primarily for departments of ministries that have been successful in other trainings (e.g. Trade Portal, IT, Consumer Protection, Intellectual Property Rights). • Focus initial capacity building efforts at targeted GOM ministries where there is a demonstrated commitment to use and maintain computers and the internet, replicating what was done at the FRD. This more focused effort on building capacity by digitizing GOM offices, particularly where Champions for reform are based, will help link the objective of increasing capacity to influencing policy change, which should be followed up on. EVALUATION QUESTION 3 • Approve grants to CSOs only when the CSO has the political support of the relevant GOM ministry and that a linkage has been formed with a person or people at the ministry who are committed to economic policy reform. At the time of writing, PSDA has adjusted its grant policy in this direction. • Provide grant support to those organizations that have successfully utilized funds in the past and optimally have detailed and realistic plans for becoming self-sufficient. USAID.GOV USAID PSDA MID-TERM EVALUATION | viii EVALUATION QUESTION 4 • Continue microfinance sector policy advocacy work and capacity building at FRD and MMFA to achieve additional positive results which will lead to increased access to finance for those in need. • Expand the DCA program and work to streamline procedures within MFIs to increase efficiency in the deployment of the scheme and to reach more under-served targeted areas, especially in the agriculture sector. • Discontinue the use of Angel Investing considering it revealed minimal results at the mid￾term and has a low probability of success by the end of the Activity (PSDA has acknowledged this). • Continue to work with financial institutions to build capacity and promote cash flow-based SME lending (access to credit), and more individual loans of larger size to satisfy the cash-starved middle market. • Continue to work with FRD to bolster previous efforts to catalyze insurance sector policy reform and liberalize the sector, paving the way for foreign investment. • Consider the use of an embedded advisor should the need arise, the case warrant, and the budget be sufficient. This method should only be considered if the targeted ministry expressed a clear desire to receive a long-term advisor and the investment is deemed to be worthwhile in terms of likely impact. A table summarizing key findings, conclusions, and recommendations is included in Annex VIII. 1 | USAID PSDA MID-TERM EVALUATION USAID.GOV ACTIVITY BACKGROUND Political reforms since 2011 have led to critical transitions and political and economic developments in Myanmar. The past years have witnessed the election of Aung San Suu Kyi and other members of the National League for Democracy (NLD) party to parliament, the release of political prisoners, a government more responsive to public opinion and environmental concerns, and the signing of ceasefire agreements with some armed ethnic minority groups. For the first time in 50 years, Myanmar’s military has eased its total control of the state, and its quasi-civilian successor government is not only opening to the United States Government (USG) and other foreign powers, but is inviting United States (US) investment, assistance, and partnership in reform. These reforms and administrative changes have improved the business environment for the private sector, though benefits have not been equitable amongst private sector actors. For instance, micro and small businesses have seen little change in economic governance. While larger businesses have seen improvements in tax collection and administration, tax payments for small businesses are relegated to negotiations with township-level officials.4 Micro, Small and Medium-sized Enterprises (SMEs) face notable barriers in access to finance, a problem often compounded for women and young entrepreneurs.5 Women and young entrepreneurs often cannot meet the collateral requirements for formal loans, thus inhibiting access to Myanmar’s financial sector.6 While some civil society engagement on such challenges exists, civil society organizations (CSOs) are limited in their ability to champion independent policy positions on economic issues.7 While Myanmar’s economy has rapidly liberalized, become more market-focused, and grown considerably in recent years—with economic growth projected to reach 6.4 percent in 2017/188—it continues to be constrained by access to finance, land, electricity, and skilled labor. Agriculture constitutes the primary economic activity in Myanmar, however, secondary and tertiary sectors remain underdeveloped with manufacturing businesses averaging only 15 employees, for instance.9 Myanmar’s business enabling environment is generally ranked below that of most regional and international competitors, with governance issues often playing a constraining role.10 Private associations also have a mixed record of representing the interests of their members and are often strongly linked with the government. 11 Given these challenges and the associated opportunities to support economic policy reform in Myanmar, the Economic Growth (EG) Office at the United States Agency for International Development (USAID)/Burma established the Private Sector Development Activity (PSDA) with a five-year budget of $23 million. Launched in 2015 and implemented by Nathan Associates, PSDA focuses on 1) creating more 4 Bissinger, Jared. 2016. Local Economic Governance in Myanmar. Yangon: The Asia Foundation. 5 UNESCAP/OECD. 2014. Myanmar Business Survey 2014. 6 UNIDO. 2015. Executive Summary: Access to finance for youth and women entrepreneurs in Myanmar. 7 Bissinger, Jared. 2016. The PrivAcronyms ate Sector and the Governance of Business in Myanmar. Yangon: USAID. 8 World Bank Group. October 2017. Capitalizing on Investment Opportunities. Myanmar Economic Monitor. 9 Employment statistics differ among sources, particularly for agriculture, as definitions of what is included in ‘agriculture sector’ vary. Census data cited by the GOM’s Ministry of Information says 40% are employed in agriculture (http://www.moi.gov.mm/moi:eng/?q=news/22/06/2017/id-10930), while the Food and Agriculture Organization (FAO) estimates the percentage could be as high as 70% (http://www.fao.org/myanmar/fao-in￾myanmar/myanmar/en/). 10 World Bank Group. 2015. Myanmar Investment Climate Assessment. Report No. 93848-MM. Washington, January. 11 Bissinger, Jared. 2016. The Private Sector and the Governance of Business in Myanmar. Yangon: USAID. USAID.GOV USAID PSDA MID-TERM EVALUATION | 2 inclusive and transparent economic governance rules and processes and 2) increasing access to finance for emerging economic actors as shown in Figure 1. Figure 1: Development Hypothesis of Private Sector Development Activity Through activity interventions in 12 issue areas, PSDA seeks to foster an effective policy foundation for an inclusive market economy, support entrepreneurship, and expand access to finance and insurance. Beneficiaries of the activity include CSOs, microfinance institutions (MFI), associations, Government and the people of Myanmar, entrepreneurs, and private sector actors. Based on previous diagnostic works and technical, sectoral analyses of Myanmar and a long history of economic theory, USAID/Burma believes that more inclusive and efficient economic governance and increased access to finance will stimulate employment and income growth across regions and among different segments of society. This, in turn, will strengthen the economic reform process, leading to a virtuous cycle of economic success and reform. Ultimately, PSDA intends to contribute to stimulating employment and income growth across Myanmar’s 14 states and regions. See Annex I for the complete Statement of Work for this mid-term evaluation. 3 | USAID PSDA MID-TERM EVALUATION USAID.GOV EVALUATION PURPOSE AND QUESTIONS EVALUATION PURPOSE USAID/Burma (or ‘the Mission’) requested Social Impact, Inc (SI) to conduct a mid-term performance evaluation (PE) to identify the extent to which PSDA has achieved intended results and specifically to identify the approaches to catalyzing economic policy reform, capacity building, and improving access to finance as well as the contribution of grants toward increasing broad-based economic opportunities. The Mission also seeks to inform it’s and PSDA’s strategic decision-making on programming decisions for the remainder of PSDA, and any subsequent economic policy reform interventions that they might seek to support. The primary audience for this evaluation is USAID/Burma, particularly the EG Office and Mission management. The EG team will be interested in the findings, conclusions, and recommendations to inform strategic and programmatic decisions. The secondary audience for this PE are Government of Myanmar (GOM) counterparts and the implementer, Nathan Associates, the latter of which may consider the ET’s recommendations when designing and implementing activities through 2020. EVALUATION QUESTIONS This evaluation addresses the following evaluation questions (EQ): EQ1. Which approaches to catalyze economic reforms have been most effective and least effective, based on progress achieved toward intended results? Approaches to examine include, but are not limited to: • Government champion • Non-government champion • Technical analysis and report • Capacity Building (short-term) • Grants to CSOs • Embedded Advisor • Others EQ 1A. Which factors influenced the achievement or non-achievement of intended results? EQ 2. To what extent have the approaches to capacity building of the public/private sector made progress toward intended results? EQ 3. To what extent have the grants contributed to increasing broad-based economic opportunities? EQ 4. Which approaches to achieve the objective of improving financial services have been most effective and least effective, based on progress achieved toward intended results? Approaches to consider include but are not limited to: • MFI sector policy advocacy • MFI Development Credit Authority • Angel Investing (and perhaps promotion of entrepreneurship) • Capacity building at commercial banks and MFI’s • Capacity building at Financial Regulatory Department (FRD) • Capacity building at Myanmar Microfinance Association (MMFA) USAID.GOV USAID PSDA MID-TERM EVALUATION | 4 EVALUATION METHODS AND LIMITATIONS This section describes in detail the methodological approach the evaluation team (ET) used for data collection and analysis. The ET has also presented the approach to site and informant selection, quality assurance, and dissemination and utilization. DATA COLLECTION METHODS The ET utilized a mixed-method approach that included several qualitative data collection methods and an analysis of quantitative data on various aspects of the work accomplished by PSDA thus far. Following the nature of the Activity (technical assistance (TA), reform and capacity building activity with many trainings), the majority of the PE data is qualitative. The evaluation methodology, inclusive of data collection methods, instrumentation, sampling, and analytical approaches was designed to answer the four EQs stated above. Quantitative: The ET collected and reviewed activity quantitative monitoring data and quantified structured elements of interviews, where deemed necessary. Quantified data from qualitative interviews include training quality, relevance, access, as well as user-friendliness. See the section on data analysis below for details on the recording and analyzing of such data. Qualitative: • Document Review: The ET began the evaluation by examining relevant background documents acquired through USAID, the Implementing Partner (IP), and other sources.12 These included activity design documents, quarterly/annual reports, monitoring data, relevant assessments and evaluations, appropriate contextual data, and other information from government sources, program implementers, and researchers. The ET organized and analyzed this data as a first step toward answering the EQs. The ET developed and applied a document review guide to focus the desk review. A complete list of reviewed documents is included in Annex V. In total, the ET reviewed and analyzed 27 documents including all relevant Activity, TA, capacity building, and grant design documents and manuals, key sectoral analyses, and progress reports. • Key Informant Interviews (KIIs) and Focus Group Discussions (FGDs): The bulk of the evaluation's data has come from in-person KIIs and FGDs with a range of stakeholder types. Informants were drawn from the categories or groups shown in the first column of Table 1. Guidelines and questionnaires for KIIs and FGDs are included in Annex II. These tools were explained prior to beginning field work to ensure that respondents understood the questions, that the questions elicited valuable and necessary data, and that the interview protocols (e.g., duration, informed consent, note-taking) were well adapted to the local context. All KIIs and FGDs were led by members of the ET, with another member serving as a note taker. The ET strived to a) ensure interviewees were interviewed by an interviewer of the same gender and b) assemble FGDs with participants of the same gender and conduct FGDs with an interviewer of the same gender as the respondents (when feasible). In the very few cases where the ET found that, for reasons of gender or status, some participants did not speak freely during the FGD, a brief private follow-up discussion was held. Data collection tools for key informants and FGD participants included questions designed to determine if men and women experience the activity 12 The ET inquired about a gender analysis, but Nathan and USAID confirmed that none was completed before the activity launched. To ensure the ET’s evaluation design is gender sensitive to the Myanmar context, the ET discussed with Nathan’s team during the first days of fieldwork. It was determined that separate FGDs for men and women was not necessary. 5 | USAID PSDA MID-TERM EVALUATION USAID.GOV and its impacts differently, which in practice has not occurred (with the exception of some beneficiary groups being represented by women mainly). The ET consulted with the IP and respondents to ensure that KIIs and FGDs were scheduled at convenient times and in safe, accessible locations to facilitate participation by each type of respondent. Additionally, data collection tools elicited information on perceptions of women and men regarding factors that facilitated or hindered change and any positive or negative consequences of the changes brought about by the activity (noting in the very few cases where these perceptions differed by gender). For total KIIs and FGDs conducted (disaggregated by men and women), see below. Gender Oriented Approach: USAID’s Automated Directive System (ADS) 203 requires integration of gender into evaluation design and implementation. SI took this guidance several steps further in this PE by assuring representation of men and women on the ET (see Annex IV for team member qualifications and roles); requiring all ET members to complete USAID’s Gender 101: Gender Equality Training; considering gender in sampling strategies; developing gender-sensitive data collection tools (as detailed above); collecting sex-disaggregated data; and considering gender and social dimensions in data analysis and reporting, where relevant. DATA COLLECTION SITES AND RESPONDENTS DATA COLLECTION SITES The two focus sites selected for the evaluation were Yangon and Nay Pyi Taw. These focus areas were selected because they a) were identified as priority areas by USAID and the IP, and b) include multiple (more than one) activities/approaches that have been addressed in the EQs. During startup, the ET reached out to the IP Chief of Party (COP) to clarify the implementers and beneficiaries within each location. The IP provided a list of beneficiaries that were targeted during the activity to date. These inputs guided further site and respondent selection, as detailed below. This final sample reflects the breadth of types of PSDA activities. To maximize resources and accelerate data collection, the ET divided into two sub-teams for data collection in both locations. Each sub-team consisted of an international team member, a Local Specialist, and an Interpreter. The teams conducted data collection in Yangon in weeks 1 and 2, and Nay Pyi Taw in week 3. At the end of the third week in country, the ET reconvened in Yangon to conduct preliminary data analysis, prepare for, and conduct a successful exit briefing at USAID. RESPONDENTS Table 1 presents a summary of respondents by category. The ET conducted 51 KIIs with USAID and IP staff, GOM officials, Microfinance Institutions (MFIs), International Financial Institutions (IFI), grantees, and other donors and stakeholders; and 11 FGDs with PSDA training beneficiaries and MFI borrowers (totaling approximately 157 individuals – 70 women and 87 men).13 Annex VI provides a complete list of respondents by category and organization. 13 Adding the total number of KII participants and FGD participants = 122+49 = 171. Since there is some overlap in counting participants among different groups under each of the EQs (e.g. some organizations needed to be counted under both grantees and MFIs, due to the overlap in EQ respondents), the exact number of participants/interviewees is 157 individuals. The number of overlapping participants is 4 males and 10 females, 171-14 = 157 individuals. USAID.GOV USAID PSDA MID-TERM EVALUATION | 6 Table 1: Summary of Respondents by Category RESPONDENT CATEGORY ORGANIZATION QUANTITY AND GENDER USG USAID 2M IP Nathan Associates, Internews 6M, 6F PSDA Grantees 15 total grantees (with KIIs and FGDs including both staff and beneficiaries of grantee programming) 24M, 26F Banks/MFIs 14 Banks/MFIs (with KIIs and FGDs including both staff and beneficiaries of Bank/MFI programming) 22M, 17F IFIs/other donors 3 IFIs and other donors 3M, 2F GOM 11 total ministries/departments/divisions 31M, 29F Business Associations Myanmar Young Entrepreneurs Association (MYEA) 3M The ET worked with IP staff to facilitate introductions with respondents, where necessary. The IP also assisted the ET with scheduling/coordinating FGDs in the evaluation sites. Prior to meeting with GOM officials, the ET ensured a letter of introduction was provided by USAID/IP to the appropriate office/official, when required. All interviewees and FGD participants were informed about the objectives of the evaluation and given the opportunity to verbally consent to be interviewed and promised confidentiality. Respondents are identified throughout this report via generic reference to the respondent category (e.g. “Trainee,” “CSO grantee representative,” “IP staff,” etc.). See Annex II for copies of the consent form and data collection tools. SAMPLING Key informants were individuals who are knowledgeable about PSDA in their areas of involvement (component). Considering the majority of PSDA efforts have been directed at improving access to finance followed by policy development and support for entrepreneurship, the mix of interviewees reflects this focus. In addition, the sampling was based on and took into account considerations stipulated in the Mission’s statement of work for this evaluation, information gleaned from the ET’s document review, potential interviewees’ depth of knowledge of PSDA, time constraints posed, and the manageable interest of the respondents. Sampling of KII respondents was done through a four-stage sampling process. The ET: 1. Asked USAID and the IP to make recommendations for individuals and offices to interview; 2. Reviewed activity documents to identify additional informants not included on the USAID and the IP lists; 3. Combined the lists and purposively sample respondents with an aim of maximizing representativeness across multiple key dimensions including activity, gender, and length of time involved in the activity; and lastly, 4. Over the course of conducting interviews scheduled from this more focused list, identified additional respondents through snowball sampling; i.e., soliciting recommendations from informants. 7 | USAID PSDA MID-TERM EVALUATION USAID.GOV Sampling of FGD respondents was done according to the following process. The ET: 1. Developed a sampling frame (for trainees, the IP provided list of trainees from each Fiscal Year (FY) and training topic/theme/type; for Development Credit Authority (DCA) borrowers, the IP provided a list of borrowers from 2017); 2. Organized and cleaned sampling frame (for trainees, the ET has cleaned the data provided by Nathan as needed and ensure information on gender, training type, contact information and date of training is included; and 3. Selected FGD respondents ensuring representation across gender. Furthermore, for FGDs with trainees, the ET included trainings with men and women separately where possible. The ET also composed FGDs with individuals of similar trainings (e.g., an FGD with trainees from a Policy Reform training). Furthermore, respondents were selected randomly from the sampling frame developed by the ET. ANALYSIS The ET used a variety of data analysis techniques to support the development of evaluation findings, conclusions, and recommendations. Both at the mid-way point and upon completion of fieldwork, the Team Leader (TL) led internal working sessions to discuss emerging findings or hypotheses. The TL captured these preliminary findings and conclusions in a matrix that categorizes findings, conclusions, and recommendations by EQ. This helped ensure that the ET a) developed a systematic and thorough response to each EQ, b) assured that preliminary analysis accounted for gender and social dimensions, c) identified any gaps where additional clarification or analysis were necessary, and d) developed a rigorous basis for developing this evaluation report. Data Triangulation: Overall, the ET used data triangulation to answer EQs. To the extent possible, similar questions were included/asked across various methods and informants to facilitate triangulation. Data from these varied streams have been compared and contrasted against one another to determine whether findings are divergent or convergent. The extent to which multiple informational streams provided consistent findings has informed the certitude and internal validity of evaluation conclusions presented in this report. Quantitative: The ET used quantitative performance monitoring data to inform their evaluation of PSDA’s progress toward intended results. Performance monitoring data includes aggregated achievements from relevant PSDA indicators and any additional data tracked by the activity. Through review of activity and progress reports completed through quarter 2 of FY 2018 (March 2018), the ET a) identified indicator achievement against targets, b) assessed relevance of indicators to activity interventions and goals, and c) noted any gaps in progress that may affect the ultimate achievement of activity goals/objectives. In the case of intrinsically quantitative data (e.g. activity monitoring, structured elements of interviews), data was analyzed in excel. The ET cleaned the data and generated some descriptive statistics and preserved and documented all raw data to facilitate reporting. To the extent that it assisted analysis, interpretation, or reporting of evaluation findings, the ET produced supportive tables. Qualitative: The ET developed a findings matrix summarizing the results of early data analysis and updated it bi-weekly throughout the data collection. Team members prepared electronic summaries of each KII, FGD, and observation. They identified response themes and key points, meeting frequently to resolve questions about interpretation, meaning, context, and analysis. The ET used qualitative analytical techniques to collate and interpret data captured through notes collected from KIIs, FGDs, and direct observations. The ET used relational content analysis to identify USAID.GOV USAID PSDA MID-TERM EVALUATION | 8 response categories and patterns as well as elucidate emergent themes, contextual factors, and trends. The ET disaggregated interview and discussion group data by sex and length of involvement in the activity to capture differential outcomes among groups, where relevant. The ET employed a data analysis strategy in which qualitative data was first analyzed independently, then in parallel (for example, KII data was analyzed first and then analyzed against FGD data). Findings from each data collection method informed and substantiated findings from other data collection methods using triangulation (as detailed above). The ET also used different forms of frequency, trend, theme, and pattern analysis to compare results across respondent groups. This allowed findings that identify who has been benefiting from what types of activities, which elements of an activity were contributing to the achievement of outputs and outcomes, which elements are not, and most importantly, why. Gender and Social Analysis: The ET worked across all activity objectives to capture and compare the results as they specifically benefit (or do not benefit) women, men, or particular social groups, where relevant. The ET ensured a generally equal number of males and females during the planning and execution of KIIs and FGDs, which is reflected in the list of interviewees. Members of ethnic minority groups were interviewed as per PSDA engagements with them (e.g. Pyu Sin Saydana Action Group (PSSAG)). All data collected through was disaggregated by gender and analyzed for effects on both male and female beneficiaries to show any significant gender differences, where relevant. The ET also assessed if women vs men faced different challenges in participating in the activity (or evaluation) due to gender. In addition, SI’s Gender Specialist advised the ET in planning, implementation, analysis, and report writing phases of the evaluation to ensure that gender and social dimensions were properly mainstreamed in each stage of the evaluation process. It must be pointed out that this is mainly a policy reform program, impacting on the entire population of Myanmar, both men and women, and any differences in accessibility by groups or individuals, based on sex or ethnicity, beyond the equal participation to program activities, was possible to assess at the time of the mid-term PE. QUALITY ASSURANCE SI has pioneered an innovative, best-in-industry approach, Evaluation Quality, Use, and Impact (EQUI®) to achieve high quality, influential evaluations that enhance development effectiveness. Developed over ten years through 300 evaluations in 85 countries, EQUI® has proven effective in driving exceptional learning, accountability and performance improvement. At all stages of the evaluation, SI applied its EQUI® approach, which entails five Quality Assurance checkpoints and collaborating closely with USAID and its IPs to promote utilization. Through EQUI®, SI offered the TL and team members designated trainings and resources that cover problem-solving tactics, activity management tools, and tips for effective communication. Lastly, to ensure technical quality, the SI Project Director employed five rigorous Quality Assurance checkpoints while integrating gender and social analysis at each stage. LIMITATIONS Below is a list of some noteworthy limitations and biases of the evaluation, as well as mitigation tactics that the ET employed to ensure limitations did not affect data quality. Recall bias: Recall bias occurs when respondents incorrectly attribute impacts or changes to an activity. They may have attributed results incorrectly to the PSDA activity, confounding them with results from other interventions they experienced. Informants answering questions about activities that were completed in the past (for example, some of the PSDA grants) had difficulty accurately recalling changes, improvements, or sources of assistance. The ET minimized this bias by ensuring that the respondent sample considered the scope and nature of the specific activity component/intervention and how it was designed to contribute to each of the activity objectives, directly or indirectly, based on the Theory of 9 | USAID PSDA MID-TERM EVALUATION USAID.GOV Change, as well as which other donor or Government programs could possibly have had an influence on any such planned outcome and how. Response bias: Response bias occurs when informants form their responses based on a personal motivation rather than the most accurate information. For example, informants may give the ET positive remarks about the activity because s/he would like to receive more assistance in the future. In some cases, informants may understand that a negative evaluation could mean the end of activity opportunities and respond accordingly. The ET has minimized this bias by including language in the informed consent that there are no inherent benefits to them from participating in the evaluation and that there will be no retaliation or direct consequences against them for their responses. Selection bias: Selection bias can occur because the IP may be more likely to guide the ET to those people who had positive experiences with the activity. The ET has minimized this bias by identifying a range of informants at any one site with differing degrees of involvement with the activity. Other forms of selection bias may occur when respondents available for interviews are not necessarily representative of the population because they have more free time, higher social status, or are better connected. The most effective approach to combating the forms of bias and limitations noted here, i.e. the recall bias, the response bias, and the selection bias, was the use of multiple sources of data to triangulate on an EQ. By combining information found in documents or interviews from multiple sources, the ET ensured that any one piece of biased data did not skew the analysis. Another approach that pertained specifically to interviews was the inclusion of key informants from organizations that do not directly benefit from the evaluated activity, like other donors and International Finance Institutions (IFIs). DISSEMINATION AND UTILIZATION PLAN Following USAID’s aim for uptake of recommendations and use of findings provided through this evaluation, the ET recommends that USAID 1) share the Mid-Term Evaluation Report with the IP, and after review and approval, 2) publish it on the Development Experience Clearinghouse (DEC) for wide dissemination of all Findings, Conclusions, and Recommendations, 3) discuss each of the relevant recommendations with the associated GOM counterparts, and finally, 4) use the Mission’s internal results tracking and learning system to ensure utilization and follow-up of recommendations. USAID.GOV USAID PSDA MID-TERM EVALUATION | 10 FINDINGS AND CONCLUSIONS EVALUATION QUESTION 1 FINDINGS Quantitative: In this section, the ET explores progress against intended results through quantitative indicators that relate to ‘catalyzing economic reform’ in intervention area 1. Indicators are taken from the activity M&E plan. The ET analyzed progress through March 2018 (quarter 2 of FY2018) against the cumulative target for FY2018 (intended to be met by September 2018). Any unmet targets are therefore not necessarily reflective of unsuccessful programming as two quarters remain in the FY for activity progress. Notably, the activity has already achieved four of the five relevant indicator targets by quarter 2 of FY2018. See Table 2 below for a summary of the result indicators and progress to date. For Indicator 1.1.1, in FY2017, a total of 423 stories were published by journalists who had completed trainings by PSDA partner Internews. Stories were published through all types of media, with print media (286 stories out of 423) and websites (74 stories) dominating. Television (37) and radio (26) coverage appears to be increasing from activity inception, however, Indicator 1.1.2’s target was reached with achievements like the successful implementation of the Trade Portal Website. Government agencies are increasingly aware of the Trade Portal’s benefits and are publishing laws more systematically on the Trade Portal. PSDA has assisted GOM agencies to publish on the trade portal. The number of visitors to the website is measured – sources include website analytical reports from Technical Barriers to Trade (TBT) Enquiry Point, Sanitary and Phytosanitary Standards (SPS) Enquiry Point, and the Internews web portal. Indicator 1.1.3’s target was unmet at the time of the mid-term and this is largely because of an unrealistic cumulative target and remaining publicizing work to be done. The Activity saw increased traffic on the site after the distribution of the Trade Portal brochure. Based on these positive results, PSDA should continue promoting and advertising the site to reach the target. In 2017, counting all steps in the legislative process, 17 policies or laws went through a total of 42 steps in the process of analysis, public consultation, drafting or revision, or full and effective utilization with PSDA support (Indicator 1.3.1). The activity successfully achieved this indicator’s cumulative target by the mid-term. The activity also achieved the target for the indicator related to non-agricultural enabling environments policies (Indicator 1.3.2) by the mid-term. Table 2: Catalyzing Economic Reform Indicators RESULT INDICATORS PROGRESS TO DATE (MID-TERM) Intended Result 1.1. Transparency of GOM policies increased Indicator 1.1.1: # media stories on the economy published by trainee journalists -Target MET (achievement is significantly higher than targeted); 300 (targeted) vs. 1891 (achieved) Indicator 1.1.2: Cumulative # economic laws, regulations and policies published -Target NOT MET (achievement is significantly higher than targeted); 120 (targeted) vs. 100 (achieved). Indicator 1.1.3: # visitors to websites Which approaches to catalyze economic reform have been most effective and least effective based on progress achieved toward intended results? 11 | USAID PSDA MID-TERM EVALUATION USAID.GOV -Target NOT MET (achievement is significantly lower than targeted); 2,700 (targeted) vs. 660 (achieved)14 Intended outcome 1.3. Economic governance aligned with international obligations and best practices Indicator 1.3.1: # agricultural and nutritional enabling environmental policies analyzed, consulted on, drafted or revised, approved and implemented (Feed the Future (FTF) indicator: EG 3.1-12) -Target MET (achievement is slightly higher than targeted); 50 (targeted) vs. 73 (achieved) Indicator 1.3.2: # non-agricultural enabling environmental policies analyzed, consulted on, drafted or revised, approved and implemented -Target MET (achievement is slightly higher than targeted); 10 (targeted) vs. 16 (achieved) Qualitative: Through KIIs and FGDs, the ET explored what played the most significant role in driving the achievements noted above. PSDA utilized multiple approaches to ‘catalyze economic reform’ since its inception, including engagement with Government Champions, engagement with Non-Government Champions, development of technical analysis and reporting, issuance of grants, and capacity building. These approaches, at the mid-term, were found to influence results in varying degrees.15 PSDA used Government Champions, or counterparts in GOM that were particularly committed to the activity effort and eager to promote policy reform, as an approach to influencing change in economic policy and legislation. The majority of respondents (including respondents from the IP, USAID, and IFIs) stated that this approach was particularly effective in mobilizing credit and in advancing the microfinance sector. Respondents referenced Champions (mainly female) who have been able to advocate effectively for reform including the former General Director of FRD, a current staff member at FRD, and a staff member in the Ministry of Education’s (MOE’s), IP Dept. The respondents did not mention any differences between men and women concerning challenges Government Champions faced in advocating for nation￾wide policy change. Like the use of Government Champions, PSDA also experimented with supporting Non-Government Champions to advance specific policy reform and legislative efforts. At the mid-term, respondents noted that PSDA successfully identified and worked with several capable and committed Burmese Non￾Government Champions, from both non-governmental organizations (NGOs) and CSOs. These champions, in some instances, were able to advocate for policy reform in effective ways. For example, Ms. M was able to bridge a gap between PSDA and FRD regarding micro finance sector reform. She was able to do this because of her strong and relevant personal connections/relationships in GOM and also her access to relevant sector data and information. Another example of a successful Non￾Government Champion is the former head of the Myanmar Consumers Union (MCU). He was elected to Parliament and PSDA was able to leverage the relationship to arrange workshops with Parliamentarians. One of the other successes was Mr. O who led the development and dialogue around initial changes in agricultural policy with PSDA’s White Paper on Agricultural Development – From Rice Bowl to Food Basket. In other instances, however, this approach did not prove as effective. 14 While the actual achieved noted here is under the target, PSDA informed the ET in June 2018 (post-fieldwork) that the MOC and PSDA seriously underreported the number of visitors visiting the Trade Portal. As of June 2018, and per the analytics report from June 19, 2018, the number of visitors to the website is over 44,000. While these achievements have not yet been reported to USAID and therefore cannot be reflected in the table, the ET has revised conclusions about the website based on this information. 15 While the ET explores approaches singularly in this section to determine the effectiveness of each one (as requested in the EQ), it is important to note that it is difficult to directly attribute success to one approach for observed achievements used by PSDA. In many cases, multiple approaches in combination have resulted in the progress visible at the mid-term. USAID.GOV USAID PSDA MID-TERM EVALUATION | 12 Ineffectiveness occurred when champions were not able to lobby relevant government ministries regarding desired policy change. An example of ineffectiveness can be seen in the case of IPPAM where the champion proceeded with little to no linkage with the CSO management. PSDA engaged in a significant amount of short-term capacity building work at several NGOs/CSOs, MFIs, Banks, and Government offices in an effort to build capacity that was intended to lead to economic policy reform, increased capacity for policy dialogue, and improved access to finance. At the mid-term, respondents noted that capacity building has been helpful in building institutional strength and individuals’ capacity; has contributed to improved policy inputs; and has fostered multi-stakeholder coordination where it previously did not exist. Each of these positive results is explored below and in EQ2: a. FGD respondents noted that an IT16 and insurance-related training conducted by PSDA with FRD was particularly effective and resulted in positive benefits for trainees. Trainees noted that their institutions were stronger due to the training, and additionally their individual capacities increased as a result of the training. Respondents also reported that the SME lending trainings at banks were effective. However, at PSDA’s mid-term it is not clear if these initial gains will lead to actual improved, useful, and relevant policy inputs and/or ultimate policy reform. While capacity building has been impactful in increasing organizational and individual capacity, respondents did not yet observe outcomes related to policy reform. This will require time and GOM willingness to change. b. One set of hands-on PSDA capacity building efforts was found to be useful for the development of a new draft law concerning a national gemstone policy. Respondents from the Myanmar Gemstone Enterprise explained that without the assistance received from PSDA, they would have been unable to draft a policy paper. Among other objectives, the policy paper is intended to help shape a divergent Gemstone Law to be drafted by Parliament, when the Gemstone Policy is eventually submitted to Parliament. They found the capacity building was, in this way, targeted to their needs and relevant to their ongoing work. The Activity also provided key TA to help MMFA leadership develop their White Paper around their policy recommendations, with support for a number of workshops to gain consensus among their members (which also notably brought the foreign and local for profit and non-profit MFIs together for the first time to collaborate on a common goal). Additionally, the Activity supported the first ever MFI General Assembly where the recommendations were to the DG, with relevant FRD representatives on interpretation of the new regulations helped catalyze the ideas and advocacy of the MMFA and the receptiveness of the FRD leadership to reform. c. With guidance and coordination from the US Embassy, PSDA is currently supporting the Myanmar Gemstone Sector Supporting Committee to a) hold a series of multi-stakeholder consultation workshops in areas where jade and gemstones are mined and traded, along with b) TA to facilitate the development of a Myanmar National Gemstone Policy. This process of holding multi￾stakeholder workshops aims for the first time to bring together previously conflicting interests and marginalized groups to reach a broad stakeholder consensus. PSDA made sure to include representatives from ethnic groups in regions where the gems are mined to work together with a broad range of stakeholders to develop a national Gemstone Policy to advance inclusive development and resolution of conflict. A draft Gemstone Policy is expected to be finalized in April 2018. PSDA provided grants to 22 CSOs and MFIs by the mid-term in an effort to build capacity that would likely lead to promoting economic policy reform (though policy reform was not specifically each grants’ mandate). Respondents representing 12 of the 16 grant recipients (NGO and MFI grantees) interviewed explained that they were not able to achieve a specific policy reform during the grant period, or as a result of the grant. Those that were able to contribute to policy reform noted that they had access to open￾16 IT training around using digitalizing regulatory processes. 13 | USAID PSDA MID-TERM EVALUATION USAID.GOV minded government ministries (e.g. MMFA and FRD). While 75 percent of grants did not directly contribute to policy change, grant respondents explained that they improved their knowledge and capacity. For example, the grant to Intellectual Property Proprietor’s Association of Myanmar (IPPAM) helped members improve their knowledge of intellectual property rights and the importance of creating and passing laws to protect intellectual property. Additionally, 12 of the 12 grants to the NGOs interviewed were effective in building the capacity of NGOs, and to specifically engage in policy dialogue, but did not (nor were they always designed to) lead to policy reform. Improved policy dialogue represents additional voices in the policy-making arena and additional policy inputs from those outside of the GOM, therefore these represent positive mid-term results from grants to CSOs. PSDA funded several technical analysis studies and reports in an effort to enhance knowledge and build capacity that would likely lead to promoting economic policy reform (e.g. one agricultural study, six economic and finance reports, and three insurance reports). Use of reports and studies have had mixed results. In some cases, it was difficult for the ET to measure the reports’ impact on progress toward policy reform. For example, some reports were not aimed directly at policy reform initiatives but were rather intended to help USAID better understand aspects of the country’s economy. While PSDA-produced reports that were intended to influence policy have educated key GOM personnel and improved awareness on key issues in some cases (e.g. Agricultural White Paper, Entrepreneur White Paper), they have not singularly led to economic policy reform according to evaluation respondents. The Agricultural White Paper, for example, called strongly to free up the ability for farmers to grow crops in line with market demand, which was done in large part soon after the White Paper came out and the new MOALI took power. However, there has been no confirmation of a written policy directive that implemented this important policy reform. This approach was found to be most effective at contributing to reforms when the targeted institution or individual specifically requested the analysis or report (e.g. Association of Southeast Asian Nations (ASEAN) Single-Window Assessment). When the analysis was specifically requested, it helped improve the understanding of the recipient regarding the discussed issues and policies. When technical analysis and reports were not specifically requested by GOM or other target recipients, however, the ET found it more difficult to identify influence of the outputs on the ultimate goal of policy reform. For example, two Banking and Finance in Myanmar reports by Sean Turnell were commissioned and paid for by USAID through Nathan Associates as part of the PSDA activity. The ET was not able to draw any direct link between their production and dissemination, however, and achievement of project objectives, notwithstanding the high quality ofthe reports and value to the international donor community. These reports did, however, help contribute to increased dialogue around Banking and Finance. PSDA has not used embedded advisors, however, multiple respondents from the IP, USAID, and IFIs noted that this approach has been useful in the past and can be useful in the Myanmar context (e.g. a respondent from FRD requested PSDA support for a long-term IT advisor for their department.17 The leadership of the Ministry of Commerce (MOC) also informally expressed interest in a long-term advisor). Respondents noted that this approach is especially impactful when the specific government ministries in question directly request assistance and have a say in the selection and approval of the individual(s) intended to become the embedded advisor. Other donors [e.g. ADB, GIZ, and the World Bank] have also used embedded advisors and/or committed long-term advisors who intermittently spend time at the project sites, with positive results in terms of providing long-term sustained capacity building support as well as advice in economic policy reform (such as increased tax collection) and banking regulations. Respondents noted, however, that a reason this approach is not often used in policy advocacy is because of the inordinate cost of the advisor. Furthermore, it can be challenging to identify sufficiently experienced advisors, particularly in Myanmar. While there is certainly no guarantee of success, the placement of a qualified and determined consultant at the right time in the right ministry, who has been personally 17 The FRD respondent explained that the IT consultant in place at the time was at the end of his contract. This respondent may have confused PSDA support with that given by LIFT. USAID.GOV USAID PSDA MID-TERM EVALUATION | 14 selected by the GOM counterpart, enhances the likelihood of success. Given the glacial pace of policy reform in Myanmar which would undoubtedly lead to much downtime for a long-term consultant, the cost-benefit efficacy of deploying an embedded advisor must be considered. A possible solution could be a long-term dedicated advisor based in Naypyidaw tasked with advising the selected most prospective GOM ministries on a roving basis. All of this needs to take into account the remaining limited PSDA timeframe and budget. CONCLUSIONS At the mid-term, PSDA has already achieved four of the five indicators related to ‘catalyzing economic reform.’ Qualitative data reveals that the most effective approach PSDA used to achieve these gains was identification, development, and support of Government Champions. This approach most directly resulted in policy gains ranging from improved policy inputs to draft legislation. While short￾term capacity building and grants to CSOs were effective at improving institutional strength and individual capacity and knowledge, at the mid-term these efforts had not yet directly resulted in economic policy reform. Specifically, these approaches resulted in enhanced policy advocacy and dialogue capacity. The result of the use of Non-Government Champions had a limited effect primarily due to champions’ lack of power and inability to link directly and be heard by a counterpart ministry official or optimally a Government Champion. This was achieved when a Non-Government Champion from MMFA was able to gain an audience with a Government Champion at FRD to make the wishes and concerns of the MMFA member MFIs heard and incorporated into legislation which eventually became law. The development of technical analysis and reports were largely ineffective at directly promoting policy reform; however, when these outputs were developed in response to a specific request by a stakeholder, they were regarded more highly by GOM respondents. Another approach that a majority of respondents noted was effective, but one PSDA has not utilized, is embedded advisors. While this approach comes with a cost, and often requires careful recruitment, it is a tried and tested approach to working with stakeholders at a national level, although the ET is not aware of any evidence that an embedded advisor has directly led to policy change in Myanmar. Table 3 below organizes these conclusions by level of effectiveness. “Effective” can be defined as exhibiting concrete and visible policy reform results and/or significant movement toward influencing economic reform with high future promise. “Moderately Effective” can be defined as mixed results: visible movement and/or some noticeable movement toward influencing economic reform, and some sign of future promise. “Limited Effectiveness” can be defined as little if any known direct causal influence on economic policy reform and little if any future promise. Additionally, issue areas where the approach resulted in the most positive results are included in column three. These issue areas will be explored further in EQs 2 – 4. 15 | USAID PSDA MID-TERM EVALUATION USAID.GOV Table 3: Effectiveness of Catalyzing Economic Reform Approaches NO. APPROACHES LEVEL OF EFFECTIVENESS MOST IMPACTED ISSUE AREAS 1. Government champions Effective in catalyzing economic reform. MF sector, Trade, Food safety, Intellectual Property Rights (IPR) Embedded advisors Effective (although not used by PSDA) MF sector 2. Non-government Champions Moderately effective in catalyzing economic reform MF sector / mobilize credits 3. Short term capacity building Moderately effective in catalyzing economic reform MF sector, Food safety, Trade, IPR, Role of media 4. Grants to CSOs Moderately effective in catalyzing economic reform MF sector / mobilize credit 5. Technical analysis and reports Limited effectiveness (ineffective) in catalyzing economic reforms MF sector / mobilize credit, Insurance, Agriculture, Entrepreneurship, EVALUATION QUESTION 1. A FINDINGS As noted in EQ1 above, achievement of intended results was influenced by the type of approach utilized by PSDA. Where Government and Non-Government Champions were used, greater results were seen in policy reform. Conversely, technical analysis and report writing was not found to have positively influenced policy results at the mid-term. In EQ1a, the ET explores what other factors influenced the achievement or non-achievement of intended results related to catalyzing economic reform. The ET explored this question with all respondents from USAID, the IP, and IFIs through KIIs and FGDs. First, respondents noted M&E and activity design as factors that influenced achievement of intended results. In order to monitor the progress of the activity towards intended results, the M&E Plan tracks result indicators [including four Feed the Future (FTF) indicators and 16 custom indicators]. As seen in EQ1, and in each of the following EQs, the majority of indicators have achieved results over target at the mid-term, primarily due to the quick mobilization of staff, needs-based activities, and strong implementation capacity by the IP, as well as a high (higher than expected) turnout at events, including at the regional level. This reflects the effectiveness of working through local partners who have been able to greatly expand the reach of the activity by connecting with remote areas and target groups. Given the fluid and opaque environment in Myanmar and limited information on which several assumptions for target setting were based, a reconsideration of targets should be expected and planned. Several assumptions, for example about the reach of local partners, do not hold at the mid-term. Regarding activity design, the IP and USAID faced challenges in setting the activity scope because of a a) transitioning and changing GOM and b) largely unknown (and untested) country context. With the political transition, IP respondents noted that it was sometimes difficult to determine policy priorities with GOM counterparts. IP respondents also noted that there are weak coordination processes in Myanmar, with counterparts often asking more than one donor to support the same initiative. In addition to challenges with understanding the GOM, the IP also had to explore a relatively unknown country and untested sector. PSDA is USAID’s first foray in attempting to influence policy reform in Myanmar. USAID had no institutional knowledge in this discipline nor did any members of the international donor community. As such, USAID and PSDA learned and adapted over time through a flexible, wide-ranging activity design. The Which factors influenced the achievement or non-achievement of intended results? USAID.GOV USAID PSDA MID-TERM EVALUATION | 16 IP used multiple approaches to see what worked and did not work well (as explored above), resulting in a focus on many issue areas over the four activity components. As depicted in Table 4 and Table 5 below, five issue areas were dropped from the initial activity design and nine were added by the mid-term. A flexible design, resulting in changing issue areas and use of multiple approaches, allowed the activity to quickly learn and adapt and drop and add issue areas as the activity progressed over time. This broad focus over multiple issue areas, while logical given the context, may have led to slower progress in promising initiatives given that issue areas added later in the activity received less time to achieve results. Additionally, several informants reported that these adaptations and added issue areas required additional administrative efforts, which may have stressed limited project budget and resources. It is worth noting that USAID and/or the US Ambassador have tasked PSDA with a significant number of interventions in addition to those originally in the PSDA workplan. PSDA has proven capable in this rapid-response role in high profile initiatives that include Agriculture Policy White Paper, Gemstone Sector Policy, Myanmar Development Institute (MDI) development, Hundi System, project bank and PPP notification, CSO policy participation workshop, and the assessment of potential support to the Central Bank of Myanmar (CBM), etc. These important activities and this flexible approach have resulted in a diversion of time and resources away from activities in the original mandate, resulting in less focus for periods of time, which has led to delays in progress toward project goals (greater breadth over depth). GOM’s low prioritization for some PSDA initiatives may have also contributed to delays. Table 4: Dropped or Re-Aligned Issue Areas ISSUE AREA RATIONALE Commercial law Changing priorities by agreement between USAID and Nathan Associates Civil society advocacy Changing design (grants) Procurement Changing priorities by agreement between USAID and Nathan Associates Women As stand-alone issue area Legal draft and commercial law (2018) Provision of comments and support to draft legislation for ministries Table 5: Added Issue Areas ISSUE AREA RATIONALE Trade and investment High Potential Sub-national Regional coverage Post-conflict Inclusive growth Gender / ethnic minority Inclusive growth, a cross cutting issue Agriculture National priority and promising export potential National gemstone policy Request by ASSK MDI Promising research & advisory function UMFCCI For research Grants to associations For support to GOM reform Second, respondents reported that factors related to activity stakeholders and staff influenced achievement of results, largely in a negative way. Since activity inception, the GOM has seen high levels of turnover (and a re-organization of ministries). This has led to delays as the activity has had to change counterparts, champions, and key stakeholders that they had “invested in” and were engaged with. GOM counterparts also reportedly have limited understanding of the subject matter, for example concerning consumer protection, intellectual property, or insurance. Within the IP, respondents noted that PSDA 17 | USAID PSDA MID-TERM EVALUATION USAID.GOV struggled to recruit for several positions. Furthermore, staff that were hired had limited experience and exposure to USAID processes and international best practice (but were noted as having good insights on the domestic environment and context). This caused delays in day-to-day operations and also influenced activity success. On a positive note, strong personal relationships between PSDA, USAID staff, and capable Myanmar nationals reportedly promoted activity coordination and helped ensure multiple adaptations and changes to the activity did not de-rail PSDA efforts in the long-run. Third, the enabling environment has influenced PSDA’s ability to achieve results since activity inception. The slow-moving policy and government environment in Myanmar requires patience, as in many developing countries. It takes years to realize policy gains for multiple reasons. The policy environment is reportedly opaque, making it difficult for donor projects or civil society in Myanmar to identify and focus on issue areas with a high probability for improvement and change. Furthermore, the government continues to have little direct communication and interaction internally – between offices and departments – and also with external stakeholders like those in the private sector or civil society. Respondents noted that this centralization of power and information challenges the level of impact projects like PSDA can have, particularly in relation to policy change. More broadly, respondents explained a general lack of motivation (or lack of incentives) for change in the sector as well as powerful private sector vested interests. The Myanmar economic policy reform environment will likely continue to be slow and difficult. Another important enabling environment factor that has complicated PSDA achievement is other development assistance efforts. As noted in Annex VII, there are many donors in Myanmar working in the same sectors. They are often working in siloes, not coordinating and collaborating. This makes it difficult for any one donor to take sole credit for successful economic policy reform without utilization of an impact evaluation approach to program measurement (with a clearly defined counterfactual). This overlap in efforts can also cause inefficiencies, though none were particularly noted to the ET during the mid-term data collection. In some cases, it may have been possible to better coordinate with other donors to achieve direct buy-in from the relevant GOM ministry (e.g. United Nations Office for Project Services/Livelihoods and Food Security Trust Fund (UNOPS/LIFT) – agriculture sector). CONCLUSIONS In addition to PSDA approaches, three factors were found to influence intended results related to catalyzing economic reform. PSDA’s flexible and adaptive activity design, together with their responsiveness to additional requests from USAID and the Ambassador, resulted in an Activity focused on many areas. This broad focus over multiple issue areas, while logical given the context, may have slowed early progress in promising initiatives given the multiple changes in policy issue areas. While the overall inexperience and low capacity of activity stakeholders and staff as well as the turnover resulting from a re-organization of ministries and resulting personnel changes likely delayed the achievement of results, certain respondents (USAID and IP) noted positive collaboration and successful coordination between the IP and USAID. Lastly, enabling environment factors such as the slow-moving policy-making process in the GOM and multiple (sometimes uncoordinated) donors in the same sector have complicated and often hampered PSDA’s successful implementation. In some cases, it may have been possible to better coordinate with other donors to achieve direct buy-in from the relevant Myanmar ministry. USAID.GOV USAID PSDA MID-TERM EVALUATION | 18 EVALUATION QUESTION 2 FINDINGS Quantitative: There are two quantitative indicators related to ‘capacity building’ efforts (see Table 6). As noted in EQ1, the ET analyzed progress through March 2018 (quarter 2 of FY2018) against the cumulative target for FY2018 (intended to be met by September 2018). Notably, the activity has already achieved both relevant indicator targets at the mid-term, one by a substantial margin and one by a slight margin (see Table 6). Table 6: Capacity Building Indicators RESULT INDICATORS PROGRESS TO DATE (MID-TERM) Intended Result 1.2. Private sector and civil society input into policy making increased Indicator 1.2.1: # public-private dialogues and public awareness events held -Target MET (achievement is slightly higher than targeted); 147 (targeted) vs. 165 (achieved) Intended Result 1.3. Economic governance aligned with international obligations and best practices Indicator 1.3.3: # capacity building workshops conducted with Activity Assistance -Target MET (achievement is significantly higher than targeted); 160 (targeted) vs. 480 (achieved) Qualitative: Based on interviews and direct observation, the ET noted that PSDA CSO intervention partner, government institution, and MFIs/Bank staff’s capacity and competence in their relevant subject/sector matter is low. In general, Burmese staff lack subject matter expertise, computer skills, and English language capability. This situation makes capacity building support particularly relevant for the Myanmar context. To address some of these gaps, PSDA utilized multiple approaches to ‘build capacity’ since its inception, but the primary form of capacity building has been short-term trainings, mentoring, and iterative consultations. Additional forms of capacity building support include assistance with comments on draft legislation, financial and organizational support for workshops and conferences, financial support for office supplies, support to develop websites and hardware equipment, digitalization support18, and a study tour. Examples of PSDA work thus far include computer IT training, cash-flow based lending/individual SME loan training, and specific subject-matter training (e.g. risk-based supervision training to FRD insurance examiners, food safety awareness, basic nutrition awareness, IPR, and consumer protection). These capacity building efforts were designed with two objectives: 1) improved knowledge, awareness, and practices, and 2) policy reform. As detailed in EQ1, capacity building efforts were found to be only moderately effective at promoting policy reform at the mid-term. Capacity building, however, was found to be effective/successful at objective 1 or ‘improving knowledge and awareness’ particularly in the microfinance sector/issue area. PSDA engaged in a significant amount of short-term capacity building trainings at Banks and MFIs in an effort to build capacity that was intended to ultimately lead to improved financial services and access to finance. PSDA also engaged in capacity building at FRD in an effort to strengthen staff capacity specifically in the microfinance sector. A majority of respondents noted that 18 PSDA is using digitalization as a major strategic initiative to improve the effectiveness of regulatory processes and reduce their unnecessary burden on users. To what extent have the approaches to capacity building of the public/private sector made progress toward intended results? 19 | USAID PSDA MID-TERM EVALUATION USAID.GOV capacity building efforts have educated them concerning the importance of key issue areas and acknowledged PSDA’s efforts to advocate for effective policy reform by linking CSO’s with relevant government ministries through workshops and conferences designed to promote policy dialogue and draft and/or introduce legislation for consideration by parliament. The majority of respondents also noted that a standout example of successful employment of this tactic can be seen in the linkage of MMFA and FRD to successfully bring about economic policy reform which increased access to finance. Respondents/trainees explained that trainings were targeted to address the needs of their organizations and were tailored to the specific capacities of the attendees in a language they could understand, using appropriate media. Trainees were able to explain to the ET how they used information from the training they attended in their day-to-day work, with the exception of banks and MFIs who have yet to base loan assessments on cashflow only, despite having received PSDA training in this discipline. Moving to a cashflow-based lending policy from a collateral-based policy in the SME sector will require a significant cultural change, and policy reform to support this has not yet occurred in Myanmar. Changes of this magnitude will require time. Importantly, trainings with the FRD of the Ministry of Planning and Finance (MOPF) first included the introduction of a comprehensive computer and internet package, followed by targeted trainings on identified needs/issue areas. Capacity building was found to be ‘in progress’ in trade (customs, trade portal) and SME financing sector related to objective 1. Because these training efforts are on-going, the ET could not fully assess effectiveness at achieving the objective of increased knowledge and awareness. Trainings on quality standards were found to be less successful at transferring new knowledge and skills to trainees, relative to other trainings. One training in this issue area included nine individuals designed to increase knowledge concerning the need for the development of a national quality infrastructure and legislation to support it. These trainees are eventually to become internationally recognized auditors, after an apprenticeship period, for the Myanmar Engineering Society’s Myanmar Quality Management Center (QMC), and form a group of trained experts in the country to serve as the basis for the QMC to deliver fee-based consulting services on ISO quality management standards, where there is considerable demand and growing competition in Myanmar. This effort is considered a work in progress and it is planned that additional trainings will occur to supplement the nine individuals who were trained in quality standards to date. PSDA has also provided much support to build the capacity of the Department of Research and Innovation (DRI) for standardization and metrology resulting in improved staff capacity. Respondents explained why some capacity building efforts by PSDA had limited success. In some situations, the trainings were too short and conducted only once (with no follow on except for Insurance, IPR, and Standards). Although much PSDA training did not focus on IT, some respondents noted that without basic computer literacy, training content was difficult to absorb and learn. In particular, the lack of internet connection has hampered ministries’ ability to enable staff members to conduct independent research. As noted in EQ1, capacity building efforts have had moderate success at promoting policy reforms (objective 2) at the mid-term. Notable examples include PSDA’s work with the FRD and MMFA (microfinance) and with the Gemstone Enterprise. In the issue area of IPR, PSDA has a relatively high chance for success/policy reform because the legislation is already before Parliament. Capacity building efforts were found to be less successful in influencing policy in other issue areas due to the lack of a connection between the NGO and the relevant government body as well as overall lack of understanding concerning the subject matter, both at the government and CSO levels (e.g. Union of Myanmar Federation of Chambers of Commerce and Industry (UMFCCI) development). These CSOs noted that PSDA capacity building efforts were not linked with the goal of economic policy reform. PSDA has contributed to the development of a new Consumer Protection law that will require much support for implementation and awareness raising. PSDA’s grant to the Myanmar Consumer Union has USAID.GOV USAID PSDA MID-TERM EVALUATION | 20 led to relatively successful policy reform engagement, including a meeting with Parliament officials on the revised consumer protection law. Another example of capacity building that has been positively received by respondents include PSDA’s support to MDI. Respondents reported that they participated in an international study tour as well as tailored trainings for MDI staff designed to better prepare them to respond to Ministry requests for evidence-based policy reform recommendations. Additionally, at the mid-term the ET found that capacity building and trainings were accessible to men and women equally. Both PSDA trainers and trainees confirmed equal participation (equal access) of men and women in all capacity building activities, with many trainings exceeding targets (minimum 50 percent) for participation. Many recipient institutions are majority female, for example, Deputy Director General (DDG) and Director General (DG) levels. CONCLUSIONS At the mid-term, PSDA has already achieved the quantitative indicators related to ‘capacity building’. Qualitative data reveals that the most effective instances of PSDA capacity building support were in the microfinance sector where the Activity’s efforts at MMFA and FRD most directly resulted in policy gains (objective 2) ranging from improved policy inputs to draft legislation. Noting these positive expectations, capacity building has more impact in increasing individual and organizational skill, knowledge, and capacity (objective 1) also notably in the microfinance sector. More common benefits noted by respondents were building institutional strength and individuals’ capacity; improving policy inputs; and fostering multi-stakeholder coordination where it previously did not exist. In addition to the advancement of microfinance sector/mobilization of credit, capacity building efforts have met the objectives effectively in these specific issue areas: food safety, trade, IPR, and role of media. PSDA has been more effective at increasing trainee knowledge and awareness than utilizing capacity building efforts to directly promote policy reform. PSDA was found to effectively identify and target needs of organizations and trainees. Challenges, however, remain and there are areas for improvement. As explained by several respondents, some trainings were too short and were conducted only once. In other cases, respondents noted that without basic computer literacy, training content was difficult to absorb and learn. Additionally, application of training knowledge to day-to-day work can only achieve policy reform and impact if the targeted institutions are linked to counterpart ministries in the GOM in some way. When the targeted CSOs are linked to and accepted by counterpart ministries in the GOM, the likelihood of achieving policy reform and impact has proven to increase markedly (e.g. MMFA). Table 7 below organizes these conclusions by capacity building objective and level of effectiveness by issue area. 21 | USAID PSDA MID-TERM EVALUATION USAID.GOV Table 7: Capacity Building Effectiveness by Approach and Issue Area NO. CAPACITY BUILDING OBJECTIVE ISSUE AREA EFFECTIVE IN PROGRESS (MODERATELY EFFECTIVE) LIMITED EFFECTIVENESS (INEFFECTIVE) 1. Capacity building towards TA objectives “improved knowledge/awareness/ practices” Microfinance sector /mobilization of credit Trade (customs, trade portal), SME financing sector, Quality Standards 2. Capacity building towards “policy reform” Microfinance sector /mobilization of credit, , trade, IPR and role of media Insurance sector, gemstone policy, quality standards, food safety, consumer rights, MDI development (early stage) UMFCCI development, agricultural sector EVALUATION QUESTION 3 FINDINGS Quantitative: Two indicators are used to track progress against intended results that relate to the contribution of grants toward increasing broad-based economic opportunities. The indicators that most closely relate to this topic are under the IRs for entrepreneurship. As noted in EQ1, the ET analyzed progress through March 2018 (quarter 2 of FY2018) against the cumulative target for FY2018 (intended to be met by September 2018). Notably, the Activity has already achieved the two relevant indicator targets at the mid-term, both by a wide margin. Table 8 below provides a summary of indicator progress. Indicator 2.1.1 was overachieved largely because of a greater than expected delivery of business services through Activity sub-grantees. For example, PSDA grantees held a large number of trainings particularly in the area of food safety from activity inception to the mid-term. Indicator 2.1.2 was also overachieved by a wide margin; however, reaching gender parity is a challenge. For the entrepreneurs training, approximately 35 percent were women. Considering there are fewer females participating in the IT and high-tech sectors in Myanmar, PSDA will continue to face a challenge in achieving equal participation of women in their activities related to entrepreneurship. Other Cross-Cutting Findings and Conclusions related to Gender are in Annex IX. Table 8: Entrepreneurship Indicators RESULT INDICATORS PROGRESS TO DATE (MID-TERM) Intended Result 2.1. Entrepreneurship Opportunities and Capacity Increased Indicator 2.1.1: # for-profit private enterprises, producers’ organizations, water users’ associations, women's groups, trade and business associations, and CBOs receiving USG food security related organizational development Assistance (FTF indicator: EG 3.2-4) -Target MET (achievement is significantly higher than targeted); 40 (targeted) vs. 272 (achieved) Indicator 2.1.2: # entrepreneurs (M/F) receiving Activity assistance -Target MET (achievement is significantly higher than targeted); 285 (targeted) vs. 1292 (achieved) To what extent have the grants contributed to increasing broad-based economic opportunities? USAID.GOV USAID PSDA MID-TERM EVALUATION | 22 Qualitative: PSDA provided grants to CSOs and MFIs since its inception. To date, PSDA has awarded 22 grants for a total amount of MMK 2,301,944,697, approximately $1,730,785. Grant funds were for delivery of capacity building trainings, payment of CSO staff salaries, subsidization of transportation and lodging for workshop/conference participants, computer support and office supplies/equipment, office rent payment, and payment for consultants. Most grants were directed at the provision of capacity building and training efforts. As noted in the activity documents (Scope of Work), the grant instrument is intended to be used for multiple objectives including: 1) complement and leverage TA with key government policy makers to promote economic policy reforms, 2) build the capacity of non-government groups to participate effectively in economic reform policy dialogue, and 3) provide NGO members with better services and understandings for them to take most advantage of the economic reforms. In addition to these stated objectives, the ET also explored grants achievement of the objective of increasing broad-based economic opportunities. It is important to note, however, that none of the grants were given the explicit goal of directly increasing/contributing to the increase of broad-based economic opportunities. PSDA’s use of grants to achieve objectives at the mid-term was found to have varying degrees of success by interest area. Toward the goal of promoting economic policy reform, a majority of grant recipients (NGOs and MFIs) interviewed (12 out of 16) were ineffective at using grant funds to lead to specific policy reform. As explained in EQ1 and EQ2, PSDA activities have not yet led to multiple instances of policy reform (which is notably outside their mandate). Only MMFA was found to be effective at promoting reform by the mid￾term, while grants related to Food Science and Technology Association (FoSTA), and IPR are still in progress. PSDA’s grants to the Myanmar Consumer Union facilitate their participation in policy dialogue. Toward the goal of strengthening capacity of CSOs to engage in policy dialogue, a majority of grantees interviewed (12 out of 16) were moderately effective. Again, MMFA was the only grant found to be successful with this objective at the mid-term, while grants to MFIs were found to be largely ineffective toward this objective. A total of 10 out of the 16 grant recipients (NGOs and MFIs) interviewed were able to use grant funds to achieve improved knowledge allowing for opportunities to take advantage of economic reforms. This is explored in detail in EQ2. Grants to MFIs, however, were only moderately successful in this objective area at the mid-term. Though little influence on broad-based economic opportunities was noted at the mid-term, grants to MFIs (Entrepreneurs du Monde (EdM), GRET Delta, and MMFA) effectively led to increased access to finance due to improved MFI capacity to better service the microfinance marketplace. Grants to business incubators were found moderately successful at contributing to broad-based economic opportunities at the mid-term. Respondents noted that economic benefits resulting from grants to a business incubator (Phandeeyar) and operators in the agricultural sector [Development for Environmentally Friendly Agriculture and Rural Life Myanmar (DEAR Myanmar), Development Resources International (DRI)) have reached the grantees themselves (and their direct beneficiaries, who have received small-farming income￾generating opportunities] but have not yet extended beyond those directly receiving the grants. These agriculture sector grants also notably created economic opportunities for more women to engage in farming. The following grants were found ineffective in this objective area: PSSAG, MES, MCU, IPPAM, Young Women Christian Association (YWCA). Due to the situation in Myanmar, there is a significant lack of subject matter expertise and general knowledge concerning civil society activities. This is also the case in ministries and parliament. It will take time for the relevant decision-makers and organizations to understand and ‘buy into’ many of the civil society initiatives. 23 | USAID PSDA MID-TERM EVALUATION USAID.GOV Regarding the management of the grants component, respondents from the IP noted that it has been difficult and time-consuming to find viable entities that qualify to receive PSDA grant funds in Myanmar. Many groups are informal with limited membership bases. IP staff further explained that the vast majority of PSDA grantees lack a serious plan to generate sufficient funds to remain operational without continued donor funding. In this context, while progress has been noted above, it may not be sustainable if organizations do not have strong infrastructure and business plans. Given the state of affairs in Myanmar (in which CSOs and NGOs are immature, but critical), however, it is reasonable to provide financial support to promising nascent CSOs in key issue areas without the strict expectation that they be financially self-reliant in the early stages. Lastly, PSDA has tracked participation of women in several grants in particular. PSSAG and Internews promulgated information concerning women’s’ participation in the peace process. The following grants targeted participation by women by more than 50 percent specifically: YWCA, MES, MFIs, DEAR Myanmar and DRI. There has been a confirmed equal or significant participation and access for women to grant activities, though as noted in EQ2, it is too early to measure access to equal benefit. CONCLUSIONS At the mid-term, PSDA has already achieved both indicators related to ‘grant effectiveness in contributing to broad-based economic opportunities.’ Qualitative data reveals that grant performance to date can be seen as: 1) ineffective in the promotion of economic policy reform, 2) somewhat effective in the building of capacity of CSOs to engage in policy dialogue, and 3) largely effective in providing CSO members with better knowledge for them to take most advantage of economic reforms. These conclusions are summarized in Table 9 below. While the grants led to little, if any, direct effect on increasing broad-based economic opportunities, they did have varying levels of influence on access to finance at the mid-term. Grants had little, if any, direct effect on increasing broad-based economic opportunities first and foremost because this was not a clear, stated objective of the grants. Secondly, this objective has not been realized because it is expected to take longer than one short grant cycle to achieve. Grants and support to grantee institutions require time to realize Activity objectives and each case is different. There is no direct cause and effect relationship between giving a grant and increasing broad-based economic activities. A grant may support a CSO to build its capacity and ability to advocate for economic policy reform which will lead to broad-based economic benefits for the masses, but all or most conditions must be in evidence for this to occur, and time is needed. Thirdly, grants may be challenged in their ultimate ability to meet these objectives considering they do not have infrastructure that supports sustainability of their current activities. Related to gender, grants are a mechanism through which PSDA has and can continue require, track, and achieve equal participation targets. Additionally, while agriculture grants have notably provided opportunities for women farmers, PSDA may continue to face a challenge in achieving equal participation of women in their activities related to entrepreneurship. USAID.GOV USAID PSDA MID-TERM EVALUATION | 24 Table 9: Effectiveness of Grants by Objective and Issue Area NO. GRANT OBJECTIVE PERCEIVED PERFORMANCE OF GRANTEE IN RELATION TO GIVEN GRANT OBJECTIVE EFFECTIVE MODERATELY EFFECTIVE LIMITED EFFECTIVENESS (INEFFECTIVE) 1. Promotion of economic policy reform* MMFA In-Progress: FoSTA, IPR 12 out of 16 interviewed grantees 2. Strengthening capacity of CSOs to engage in policy dialogue MMFA 12 out of 16 interviewed grantees MFIs 3 Providing CSO members with better knowledge (for them to take most advantage of economic reforms if/when they occur) MMFA, 10 out of 16 interviewed grantees MFIs - 4 Contribution towards broad-based economic opportunities* Microfinance (EdM, GRET Delta and MMFA) Business incubators (Phandeeyar), Agriculture (DEAR Myanmar, DRI) PSSAG, MES, MCU, IPPAM, YWCA, etc. *Note: Grantees were not given this explicit goal EVALUATION QUESTION 4 FINDINGS Quantitative: In this last EQ, the ET explores progress related to ‘Improved Access to Finance and Mobilization of Savings.’ The Activity has achieved four of the six relevant quantitative Activity indicator targets at the mid-term, all by a significant margin (see Table 10). Overachievement of indicators 3.1.1, 3.1.2, 3.1.3, and 3.1.4 reveals both a high demand of credit in the country and the faster-than-expected uptake of new lending techniques by PSDA partners. Of particular note, close to 40 percent of the loans have been made directly to producers, and the majority of these loans are made to men. Furthermore, larger loans have been taken out by men relative to women. The vast majority of microfinance loans and those guaranteed under the DCA are taken by women. The fifth and sixth indicators deal with access to savings (insurance), and targets have not been set or realized due to the early stage of the intervention and the slow pace of reform found in the insurance sector in Myanmar. Of the DCA MFIs, only PACT Global Microfinance Fund (PGMF) has a license to take savings other than 5 percent forced and 5 percent voluntary. Despite reforms allowing MFIs to take deposits, implementation of deposit taking policies by MFIs has been moderate. Insurance is designed to mobilize long-term savings. EQ4: Which approaches to achieve the objective of improving financial services have been most effective and least effective? 25 | USAID PSDA MID-TERM EVALUATION USAID.GOV Table 10: Access to Finance and Mobilization of Savings Indicator RESULT INDICATORS PROGRESS TO DATE (MID-TERM) Intended Result 3.1. Access to Credit Increased Indicator 3.1.1: # micro, small and medium enterprises (MSMEs) including farmers, receiving agricultural related credit (FTF indicator: EG.3.2-3) -Target MET (achievement is significantly higher than targeted); 2,850 (targeted) vs. 5,227 (achieved) Indicator 3.1.2: # micro, small and medium enterprises (MSMEs) receiving non-agricultural related credit -Target MET (achievement is significantly higher than targeted); 1,760 (targeted) vs. 5,457 (achieved) Indicator 3.1.3: Value of agricultural and rural loans (FTF indicator: EG3.2-6) -Target MET (achievement is significantly higher than targeted); $7,087,500 (targeted) vs. $16,627,508 (achieved) Indicator 3.1.4: Value of non-agricultural and rural loans disbursed -Target MET (achievement is significantly higher than targeted); $2,862,500 (targeted) vs. $33,399,773 (achieved) Intended Result 3.2. Access to Saving Products Increased Indicator 3.2.1: # people with a savings account or insurance policy indicator 3.2.2: "Value of new savings account or insurance policy" - No data Qualitative: PSDA utilized multiple approaches to ‘improve access to finance’ since its inception to achieve the results noted in the indicators above, including MFI sector policy advocacy; deployment of the DCA; capacity building at MFIs and Banks, the FRD, and MMFA; and to a lesser degree support of Angel Investing. At the mid-term, these approaches were found to influence the results in varying degrees. PSDA engaged in MFI sector policy advocacy as an approach to influencing change in economic policy and legislation with the goal of improving access to finance. The majority of respondents (including respondents from the IP, USAID, GOM, and IFIs) stated that this approach was particularly effective in mobilizing credit and in advancing the microfinance sector. Respondents referenced a specific example of PSDA support for advocacy: PSDA helped collect MFI views through the MMFA, set up consultative meetings with MFI members throughout the country, and present collective requests to the FRD. This resulted in four major new regulations (August 2016) that liberalized access to funds and savings as well as reduced other regulations. PSDA then conducted a ‘question/explanation’ workshop for MFIs with FRD a month after the regulations were issued so that MFIs would be in a better position to take full advantage of the positive policy changes. PSDA also made use of the DCA, a 50 percent loan guarantee scheme, to improve access to finance. DCA participation requires that guaranteed loans be used for agriculture and agribusiness. MFIs were also encouraged to make larger loans on an individual basis rather than solely extending group loans, which is the standard form of MFI lending in Myanmar. PSDA’s DCA Facility for portfolio lending by MFIs was put in place in 2017 and made available to five MFIs: PGMF, ASA, Lanka Orix Leasing Company (LOLC), Proximity Finance, and Myanmar Development Partners (MDP). As of the end of March 2018, the five participating MFIs had disbursed and placed under the guarantee a total of 4,100 loans amounting to approximately $5.26 million (not including the loans made by Proximity Finance). In terms of Commercial Banks, a DCA facility is under development for AYA Bank which will share the risk (50 percent guarantee) in lending to MFIs for on-lending and for SME lending. PSDA will provide TA to both banks on-lending to and monitoring of MFI performance. As of this evaluation, these guarantees are in the process of being deployed. USAID.GOV USAID PSDA MID-TERM EVALUATION | 26 The DCA guarantee has allowed half of MFI respondents to increase loan sizes and move faster to take on more risk in individual lending to agribusinesses in higher risk areas (moderately effective). The other half of MFI respondents report that they would be doing the same type of lending in the same way based on their established loan policies, regardless of the DCA guarantee. While DCA has been well received by financial institutions, four of the five MFIs interviewed mentioned that the process of gaining approval was slow and the documentary and due diligence requirements were substantial. Most MFIs and banks also stated that the amounts guaranteed under the DCA are not significant relative to the overall portfolio sizes of their financial institutions. The DCA is intended to only work on the margin of an institution’s lending, e.g. the high-risk areas that are new and outside of their original loan policy. DCA’s are fundamentally an opportunity to lenders to learn about new markets, achieve success, and then adjust their loan policies accordingly. PSDA engaged in a significant amount of short-term capacity building at Banks and MFIs in an effort to build capacity that was intended to ultimately lead to improved financial services and access to finance. These trainings were made available to both men and women equally and took the form of short-term (two to four day) trainings on cash-flow based loan assessments for individual loans in the SME category. At the mid-term, the majority of respondents noted this capacity building has been helpful in building individuals’ capacity, but no new loan products have been deployed by the financial institutions as a result of the trainings. It is noteworthy that some MFI participants began to do larger individual loans and are gradually incorporating lessons learned in the form of increased assessments of an enterprise’s capacity to repay loans. Two MFIs are experimenting with crop loans to over 250 farmers with repayment dates timed to coincide with harvest periods. The DCA will be employed in this effort with a target of reaching 1000 farmers. PSDA plans to continue to work in earnest to promote cashflow-based lending over the second half of the Activity. Some additional examples of PSDA capacity building efforts at MFIs include work with GRET Chin on a review of lending procedures and in the development of a risk management policy and manual; financial support to Opportunities NOW for development of a financial literacy app for small entrepreneurs; and financial support to EdM to promote a microfinance program that will target vulnerable populations in Southern Yangon by providing financial services and trainings to develop their income-generating activities. Training was also provided to PGMF and MDP on credit scoring. All of these initiatives are works in progress and results will be fully known over the second half of the Activity (working towards increased access to finance). Capacity building at banks to encourage SME lending (particularly in the agribusiness sector) included a review of lending procedures and recommendations on revising the procedure to separately address SME credit requests; a four-day SME lending training; and development of an SME credit policy and procedures manual at both Myanmar Apex Bank (MAB) and Ayeyarwaddy Farmers Development Bank (A-Bank). At the mid-term, plans were in the works to deliver a four-day training program for Loan Officers and a one￾day SME lending training for senior managers at AYA Bank. PSDA also engaged in capacity building at the FRD in an effort to build capacity that was intended to strengthen staff capacity and ultimately to lead to economic policy reform, specifically in the microfinance sector. In recent months, an effort is being made to make inroads in positively affecting staff capacity and ultimately in affecting positive policy change in the insurance sector. As with capacity building with banks and MFIs, all FRD trainings were available to both men and women. Capacity building at FRD took the following forms: 1. PSDA provided funding for computers in local and national FRD offices and conducted a training on web management and on strengthening skills of microfinance regulatory staff in MS Office and 27 | USAID PSDA MID-TERM EVALUATION USAID.GOV Comptia A+. All FGD respondents were effusive in emphasizing the usefulness and relevance of this training. Trainees now use these skills on a daily basis, resulting in movement from a paper￾based office to one moving in the direction of becoming computer-based. The skills acquired from the training have also improved the FRD’s ability to collect information from MFIs and collate important data. 2. PSDA provided a draft of a comprehensive insurance business law that complies with the International Association of Insurance Supervisors’ Insurance Core Principles, consulted extensively with FRD on wording and provisions, and will continue to work with FRD on shepherding the law to Parliament. 3. PSDA supported capacity building for FRD insurance examiners. PSDA conducted the following trainings: a basic introduction to insurance; a Risk-based Insurance Supervision training; and a Reinsurance training. Additionally, PSDA developed an on-site and off-site supervision manual that was provided to FRD insurance examiners. All FGD respondents emphasized the usefulness of these trainings. 4. PSDA advocated for liberalizing the insurance industry to allow foreign insurance companies to enter the Myanmar market. 5. PSDA held a stakeholder workshop for FRD employees and local and international insurance companies on drafting the new insurance business law. FRD respondents expressed optimism concerning the chances of the insurance business law legislation being passed into law in addition to liberalization in the insurance sector which will eventually allow international insurance companies to participate in the Myanmar market. Despite all of the positive effects PSDA capacity building support has had with the FRD, the microfinance policy changes noted above took place before the majority of capacity building efforts at FRD had any major effect. PSDA policy advocacy support with MMFA was able to bridge the gap and directly advocate for policy change on behalf of its MFI members. Since that time, two key staff at the FRD are no longer present and have been replaced by an individual previously solely responsible for the insurance sector. A FRD respondent noted dissatisfaction concerning MMFA management’s lack of communication with FRD.19 Respondents mentioned that PSDA capacity building at the MMFA was effective. PSDA worked to build capacity that was intended to strengthen MMFA staff capabilities, support its MFI members in building MFI management and Loan Officer capacity, and ultimately to lead to economic policy reform in the microfinance sector and increased access to finance for those in need. All capacity building trainings were available to both men and women equally, and the vast majority of MFI borrowers are women. Capacity building at the MMFA took the following forms: 1. PSDA provided financial support to MMFA through a grant for capacity building, in cooperation with several other donors supporting the MMFA. 2. PSDA provided TA to help MMFA leadership to develop their White Paper around their policy recommendations, with support for a number of workshops to gain consensus among their members (and which for the first time brought the foreign and local, for profit and non-profit MFIs to work together as one). 19 The head of the FRD Insurance Division is now Acting Head of the FRD Microfinance Division. He now has authority over the Microfinance sector as well as the insurance sector. An evaluation respondent commented that MMFA often engages in activities without informing the FRD and is in possession of MFI data which should be reported to FRD but is not. Moreover, MMFA operates autonomously from FRD, which an evaluation respondent does not believe to be appropriate. USAID.GOV USAID PSDA MID-TERM EVALUATION | 28 3. PSDA supported MMFA’s first two Annual General Assembly meetings, which has facilitated open elections of MMFA leadership. MMFA staff were also trained in governance. 4. PSDA assisted MMFA in the development and quarterly delivery of a two-day training program on individual lending for MMFA members. This training responsibility will be transferred to local MMFA-related trainers over time. PSDA supported entrepreneurship and Angel Investing in the form of three grants to Phandeeyar, a business incubator. PSDA supported a Myanmar team’s participation in an international “Robot Challenge” where teams were tasked to build their own robots and compete against teams from around the world (Myanmar finished 6th out of over 120 competing countries), as well as for development of a “MakerSpace” where hardware and software innovation takes place and is developed (PSDA provided travel support and paid for the “Robot Challenge” fees through separate funding which was not part of the grant). Despite the success of Phandeeyar as a business incubator and angel investor (although PSDA did not support them with the intention of their providing financial support to entrepreneurs), the support was ineffective at promoting entrepreneurship beyond the relatively few individuals who received direct support. The TA given to and non-funding partnerships with Myanmar Young Entrepreneurs Association (MYEA) has met its mainly capacity building objectives within the grant framework. However, it is important to point out that the TA was not designed/did not contribute to broad-based economic opportunities. In addition, the grant to Myanmar Business Answers (MBA) has met its objectives in terms of trainings and awareness raising and attracting investors and providing some loans. Small businesses are better aware, also to apply for and attract investments though it is still at it starting phase and therefore too early to assess contribution to broad-based economic opportunities. That said, MBA aims to support loans for expansion of business and prioritizes promising sectors (e.g. agriculture, textiles, etc.). The majority of IP and USAID respondents stated that Angel Investing has had the least impact on improving financial services and related intended results relative to other approaches used. Respondents explained that there was a lack of attractive investment opportunities as well as a lack of legal framework. Additionally, the impact of this work was only visible for the few individuals that received direct support. CONCLUSIONS At the mid-term, PSDA has achieved four of the six quantitative Activity indicators related to ‘improving financial services and access to finance’ revealing adequate progress in increasing access to credit but limited progress to date toward the goal of increased access to savings (insurance). To its credit, PSDA has provided TA to advance the liberalization of the insurance sector as well as provided support to reform the insurance law. PSDA has also made efforts to build capacity to conduct risk-based insurance supervision and has endeavored to support regulations to allow for the increase in savings deposits by MFIs, which would be the foundation for increased savings among resource poor families. It is hoped these efforts will yield results over the second half of the project. Progress in increasing access to credit reveals both a high demand of credit in the country and the faster-than-expected uptake of new lending techniques by PSDA partners, while the lack of progress in increasing access to savings was largely attributed to the slow pace of reform found in the insurance sector in Myanmar. Qualitative data reveals that the most effective PSDA approaches to achieve these gains in increasing access to credit were policy advocacy in the microfinance sector and capacity building at MMFA. Capacity building at FRD was also noted by respondents as effective toward this goal. These approaches most directly resulted in relevant access to finance gains ranging from improved policy inputs to draft legislation as well as enhanced policy advocacy and dialogue capacity. 29 | USAID PSDA MID-TERM EVALUATION USAID.GOV While the deployment of the DCA loan guarantee scheme was viewed by some respondents as having a positive impact on improving financial services, the impact was not widespread at the mid-term relative to the whole microfinance sector in Myanmar given the relatively small amounts guaranteed. While short￾term capacity building to MFIs and Banks were effective at improving institutional strength and individual capacity and knowledge, at the mid-term these efforts had not yet directly resulted in economic policy reform nor can they be effective thus far in improving access to finance given the fact that PSDA’s efforts have not yet directly resulted in the deployment of new loan products. Despite the success of Phandeeyar as a business incubator and angel investor and the effective use of PSDA grant funds, the support was ineffective at promoting entrepreneurship beyond the few individuals who received direct support. Table 11 below organizes these approaches to facilitating Access to Finance by level of effectiveness, with summary details. Table 11: Effectiveness of Facilitating Access to Finance Approaches NO. APPROACHES LEVEL OF EFFECTIVENESS DETAILS 1. Microfinance sector policy advocacy Effective Concrete improvements observed in policy leading to increased access to finance for those in need. 2. Capacity building at MMFA Effective Leading to increased access to finance for those in need. 3. Capacity building at FRD Effective / Moderately effective Effective but policy reform started before PSDA support. 4. DCA Moderately effective In the early stage of its use, and relatively small amounts guaranteed at the mid-term. 5. Capacity building at MFIs and Banks Moderately effective Capacity has been improved, but no new loan products have been developed or deployed as a result of the trainings. 6. Angel Investing Limited effectiveness (ineffective) Lack of attractive investment opportunities as well as lack of legal framework. USAID.GOV USAID PSDA MID-TERM EVALUATION | 30 RECOMMENDATIONS Overall, PSDA is a well-managed Activity operating in a challenging and complex environment. After two and a half years of intensive work in an extremely slow-moving policy reform environment, PSDA has achieved success in notable areas. PSDA has successfully built the capacity of the majority of supported NGOs in 16 critical issue areas and enabled them to be active participants in policy dialogue where little or no ability to do so previously existed. PSDA contributed to advancing Myanmar’s compliance with ASEAN and World Trade Organization (WTO) requirements. PSDA has also improved the ability of Myanmar citizens, primarily women, to gain access to finance. USAID and Nathan Associates have agreed on a reorganization of Activity priorities which has incorporated lessons learned from the first half of the Activity and will likely improve activity performance over the second half of the period of performance. While the findings above noted positive progress against intended results, there are several areas where improvements can be made. The following recommendations are focused on improving Activity performance for the coming years. All recommendations are targeted at the IP, Nathan Associates, and USAID/Burma PSDA staff. Recommendations specifically related to M&E are in Annex VII. A table summarizing key findings, conclusions, and recommendations is included in Annex VIII. STRATEGIC • Focus on building capacity in targeted sectors, first through digitalization of GOM offices and optimally at ministries that have identified Government Champions of reform in place, using embedded advisors as and when it is feasible and has been requested by the ministry in question. • Continue the effort to hone Activity focus on fewer interventions that have a broader impact to a wider number of people, i.e. ‘Lighthouse’ issues that cast a large footprint in terms of impacting a broader part of the population. To qualify as a lighthouse issue, the intervention should be a major benefit for many people and should be high profile. PSDA is working in many issue areas based on midterm evaluation evidence. Additional resources should be focused on fewer lighthouse issues versus investing small amounts of money in a wide spectrum of disparate activities and issues. • Once the critical Lighthouse issues are clearly identified, identify strong personalities who are likely to have a long-term presence and policy reform impact in key ministries (Government Champions), then tailor a set of grants and TA around this personality or set of personalities to support them to carry out policy reform. The key to success is combining and linking the right Government Champions with lighthouse issues, when possible, and providing appropriate and timely support. It may be possible to work in multiple major issues with the right Government Champion(s) so long as the IP and USAID/Burma are able to build trust and a positive mutually beneficial relationship. Reform in Myanmar is about people and personalities more so than offices and institutions. Should a person not exist that fits the description of a Government Champion, PSDA should work to link with and develop those who may be considered to have the highest potential to advance policy reform. • Continue approaches which serve to help in accessing and building relationships with Government Champions e.g. through requested TA, capacity building, and training given for targeted ministries; and through requested technical analysis and reports. • Focus top priority near-term efforts on following up on in-progress work concerning Insurance reform, IPR law passage, and Gemstone policy legislation passage into law. • Be conservative and eschew early stage and start-up initiatives that have not proven to be viable. USAID/Burma should realistically assess its ability to achieve success in longer term 31 | USAID PSDA MID-TERM EVALUATION USAID.GOV initiatives such as Commercial Bank reform – interest rate liberalization (both in interest charged to borrowers and interest paid to depositors), Payments System – electronic transactions – interbank payments (cash handling is currently 40 percent of operating costs in Myanmar), and, if feasible, increase the government’s capacity to do proper and timely comprehensive financial feasibility studies of national infrastructure projects using international project assessment standards such as Environmental Impact Assessments and Social Impact Assessments. Regarding MDI, consider small scale support at the initial stage and only continue if tangible results are seen in the near term as a direct result of the support. Despite the great GOM need for such capabilities, USAID should avoid starting what it is unlikely to finish in the near to mid-term (activity closure in 2020) unless there are special circumstances and the initiative in question is considered to be in line with important USG strategic interests. EVALUATION QUESTION 1 AND 1A • Continue to develop strategic relationships with committed Government Champions (advocates for policy reform) in economic reform likely affecting the broadest population who work in targeted ministries (e.g. FRD/MOPF, DoT/MOC, DCU/MOC, and IP/MOE) where important positive policy change is most likely to occur. • Consider placement of long-term consultants (embedded advisors) in ministries where there is a demand to build capacity, improve subject matter expertise, establish trust and form relationships with individuals who work in targeted departments where important positive policy change is most likely to occur. While there is certainly no guarantee of success, the placement of a qualified and determined consultant at the right time in the right ministry, who has been personally selected by the GOM counterpart, enhances the likelihood of success. Given the pace of policy reform in Myanmar which would undoubtedly lead to much downtime for a long-term consultant, the cost-benefit efficacy of deploying an embedded advisor must be considered. A possible solution could be a long-term dedicated advisor based in Naypyidaw tasked with advising the selected most prospective GOM ministries on a roving basis. All of this needs to take into account the remaining limited PSDA timeframe and budget. • Continue to develop strategic relationships with committed Non-Government Champions (advocates for policy reform) in economic reform (individuals) in NGOs/CSOs working in targeted Issue Areas where important positive policy change is most likely to occur, as a secondary priority strategy. This should not be a main focal point in the strategy but should be done only in cases where it is known that relevant government ministries are receptive to the collective inputs of an interest group, as represented by a PSDA identified Non-Government Champion. • Continue to engage in short-term capacity building efforts only where it has proven to work in the past and primarily where it will have the highest probability of leading to economic policy reform, namely at government offices working in targeted Issue Areas where important positive policy change is most likely to occur. This should not be a main focal point in the strategy but should be done in a strategic way (see EQ.2.) • Continue to engage in grants to CSOs only where it has proven to work in the past and primarily where it will have the highest probability of leading to economic policy reform through use of policy dialogue e.g. MMFA. This should not be a main focal point in the strategy but should be done in a strategic way as and when needed (see EQ3). • Produce technical analysis studies and reports only when it has been specifically requested by a prominent individual in GOM and where it will have the highest probability of leading to economic policy reform. This should not be a main focal point in the strategy but should only be done as and when needed. On occasion, it may be a worthwhile investment to conduct a brief study in a high priority issue area (e.g. economic benefits to improved access to USAID.GOV USAID PSDA MID-TERM EVALUATION | 32 finance for SMEs) that has not been considered yet by GOM. Any reports should elucidate key subject matter and be succinct, with reasonable and attainable recommendations including steps to enhance the likelihood of achieving results. EVALUATION QUESTION 2 • Design capacity building/training plans primarily for departments of ministries that have been successful in other trainings (e.g. Trade Portal, Information Technology, Consumer Protection, Intellectual Property Rights). Base plans on the specific needs of each institution, to include IT training and support, training in specific subject matter, and English language training possibly in coordination with long-term volunteer organizations. • Deliver capacity building support in cases where economic reform policy is most likely to occur, i.e. where there is a committed and enlightened pro-reform government official (Government Champion) in place with the power and ability to drive a piece of legislation through Parliament to passage into law. The IP should endeavor to provide capacity building services, when practical, to support legislation which is currently in-progress. • Focus on introducing a comprehensive computer and internet package at targeted GOM ministries where there is a demonstrated commitment to use and maintain computers and the internet e.g. CBM) to support PSDA’s digitalization work that seeks to improve the effectiveness of regulatory processes and reduce their unnecessary burden on user, repeating what PSDA has done at the FRD of the MOPF. EVALUATION QUESTION 3 • Continue grants to organizations only where there is a track record of successful use of previous PSDA grant funds in achieving intended results, which require tracking and assessment, and if possible where the organization has a feasible plan to become self-sufficient within a reasonable timeframe. • Approve grants to CSOs only when it is considered highly likely that the CSO has the support of the relevant GOM ministry and that a linkage has been formed with a person or people at the ministry who are committed to economic policy reform. At the time of writing, PSDA has adjusted its grant policy in this direction. EVALUATION QUESTION 4 • Continue Microfinance policy advocacy work at the MMFA and FRD employing the same methods (primarily capacity building, assistance in comments to draft legislation, support to consultative workshops and conferences, identification of champions, and assistance with linking personalities a MMFA with their counterparts at FRD) to achieve additional positive results in policy which will lead to increased access to finance. • Continue to expand the DCA program in Myanmar and work to streamline procedures within MFIs to increase efficiency in the deployment of the scheme and to reach more financial institutions and more underserved targeted areas. • Discontinue support to Angel Investing as it has a low probability of achieving success in achieving broad-based economic activities. • Continue to provide capacity building at Banks and MFIs to build capacity and promote cashflow based SME lending to satisfy the cash starved middle market. Trainings should be focused on deployment of SME loan products when possible, not just capacity building at banks. Be selective in choosing banks that are committed to SME lending (A-Bank and 33 | USAID PSDA MID-TERM EVALUATION USAID.GOV AYA Bank). Avoid banks that are less committed to what PSDA can offer and that have signed up with other donors who employ different systems.20 • Provide capacity building at FRD and MMFA. o FRD: Provide increased similar trainings in improving the electronic capabilities of the staff in Naypyidaw and other offices and capabilities to link with MFIs for more accurate reporting and data collection. More trainings should also be given to FRD staff to help them understand the insurance sector as well as microfinance sector. English language training would better enable them to learn on their own by studying resources available on the internet. Continue to work with FRD to bolster previous efforts to catalyze insurance sector policy reform and liberalize the sector, paving the way for foreign investment. o MMFA: Provide additional cash flow based individual loan product training with an emphasis on actual products deployed. Provide additional support in better linking the new FRD Director with the MMFA so they can develop a more cohesive working relationship which will result in greater efficiency and increase the likelihood of additional economic policy reform. • Consider the use of an Embedded Advisor should the need arise; the case warrant and the budget be sufficient. This method should only be considered if the targeted ministry expressed a clear desire to receive a long-term advisor and the investment is deemed to be worthwhile in terms of likely impact. 20 MAB has just concluded an agreement with KfW and will have three embedded advisors training them in SME lending using a rigid and different system than that used by PSDA. USAID.GOV USAID PSDA MID-TERM EVALUATION | 34 ANNEXES ANNEX 1: EVALUATION STATEMENT OF WORK 35 | USAID PSDA MID-TERM EVALUATION USAID.GOV USAID.GOV USAID PSDA MID-TERM EVALUATION | 36 37 | USAID PSDA MID-TERM EVALUATION USAID.GOV USAID.GOV USAID PSDA MID-TERM EVALUATION | 38 39 | USAID PSDA MID-TERM EVALUATION USAID.GOV USAID.GOV USAID PSDA MID-TERM EVALUATION | 40 41 | USAID PSDA MID-TERM EVALUATION USAID.GOV ANNEX II: DATA COLLECTION TOOLS This annex includes all data collection tools used in conducting the evaluation, such as questionnaires, checklists, and discussion guides. GUIDELINES AND CONSENT SCRIPT FOR KIIS AND FGDS The Guidelines and Consent Script included here will be followed for each KII and FGD conducted by the ET during this evaluation. Additionally, basic information about each respondent will be collected via a Respondent Information Form which will include details such as gender, site, and ethnicity. Guidelines Introduction: Always begin with an introduction, explaining the following: 1. Introduction of team members 2. Purpose of KII/FGD 3. Summary of topics to be discussed 4. Confirmation of confidentiality (reading of complete consent script – see below) 5. Indication of how long KII/FGD will take 6. Distribute Respondent Information Form and collect from each informant/participant 7. Begin with a general question to open up the conversation (“What is your relationship to the PSD activity?” – see questionnaire) KII Implementations: ● All KIIs will take between 60 mins and 90 mins. ● During the KII, determine if informants are clear on the meanings of special technical terms ● Note down responses on the Interview Sheet (to be done by notetaker) FGD Implementation: ● All FGDs will take between 60 mins and 120 mins. ● Moderators of the FGD need to carefully read all of the content and information on each FGD sheet in advance before discussion. Detailed instruction and explanation on the sheet will help guide him/her to conduct the necessary facilitation process for the participants during the discussion. ● Before discussion starts, discussion group is formed as instructed on the FGD sheet. Point out special technical terms or words seen on FGD Sheet that may not be clear for participants and confirm their understanding. If necessary, explain (Some definitions are mentioned on FGD sheet.) KII/FGD Questions: 1. Proceed through questions as detailed in the KII and FGD protocols. 2. Keep it conversational rather than mechanical. But be sure to cover all questions. Questions provide structure to the conversation but can be discussed in any order if they come up sooner. Avoid repeating questions if they have already been answered. 3. Be sure to specify quantities, time periods, etc. in their response and if needed ask for clarification on this. USAID.GOV USAID PSDA MID-TERM EVALUATION | 42 4. Ask questions in a way that will solicit facts, not their opinions (unless perspectives are needed for the EQ). 5. Avoid making suggestions which might influence the response. 6. Explain questions if unclear or if the person interviewed has misunderstood the question. 7. Ask for clarifications and explanations from respondents, if needed. Ensure equal participation from FGD participants, as possible. 8. Conclude KII/FGD by summarizing main points and ask the interviewee/respondents for any additional comments/questions. Thank participants for their time. Post-KII/FGD: The ET should discuss: 1. Were all the questions answered? Are any changes to the protocols required? 2. Do the answers make sense? Were they facts or opinions? 3. Is there missing information or any doubts to the information received? 4. What are the main conclusions? Consent Script Hi, my name is [name], and I work for Social Impact, which is an independent research company based in the Washington, DC area. We are collecting data about the USAID/Burma Private Sector Development Activity that launched in 2015. As you may know, the activity was designed to foster an effective policy foundation for an inclusive market economy, support entrepreneurship, and expand access to finance and insurance. The evaluation is intended to inform the activity’s implementation for the remainder of the activity period and to inform USAID’s strategic decisions about future programming in this area. We selected you and other respondents to interview because we understand that you may have perspective on the activity itself and/or on relevant subject matter. We expect the duration of this [interview/focus group discussion] to be no more than 60 minutes. We plan to ask you about economic governance and access to finance in Burma, as well as related activities conducted with funding from Nathan Associates. There are no known risks or direct benefits related to your participation in this evaluation; however, your inputs may lead to recommendations that benefit economic growth in Burma— and, thereby, the general public. All information that you share will be kept confidential. We will aggregate and present our findings to USAID in a way that cannot be attributed to any individual. Therefore, please feel free to speak openly and candidly with us. Your participation is voluntary. Please feel free to ask to skip any question that you do not feel comfortable answering, end this interview at any point, or withdraw your responses after the interview. If you have questions about this evaluation, please contact Amanda Stek of Social Impact via e￾mail at astek@socialimpact.com. Do you confirm your consent to participate in this [interview/focus group discussion]? ⃝Yes ⃝No Do you have any questions before we begin? 43 | USAID PSDA MID-TERM EVALUATION USAID.GOV KEY INFORMANT INTERVIEW (KII) AND FOCUS GROUP DISCUSSION (FGD) QUESTIONNAIRES Questionnaires are organized by respondent category. All questionnaires began with the table below. For the purposes of this annex, the table is only included once. Names and Designation of Interviewee Name of Institution Date of Interview Location of Interview Name of Interviewer Name of Note Taker Interview start time Interview end time KII QUESTIONNAIRES USAID # Interview Question Response Grants Program 1. How is the program structured as part of the overarching regional ASEAN Connectivity through Trade and Investment (ACTI) program? How do the pillars/areas/components fit in (TA vs grants and nature/scope) 2. How, if at all, did PSD grants contribute to the following: Policy Reform? Economic growth? Entrepreneurship? Access to Finance? (Food security) Provide examples. Were they designed accordingly? How many people would you estimate PSD has helped, in terms the above as result of the grants only? 3. Describe the types of people helped. How have they been helped? Benefits to poor: In general, did the grants contribute to economic benefits in the form of poverty alleviation? Income enhancement – estimated % increase? Job creation (directly or indirectly)? Can you generally estimate the number of jobs created? For women? For youth? How do you know? 4. Are you aware of any difficulties that occurred in relation to the grant program? If so, describe. Grant Decision USAID.GOV USAID PSDA MID-TERM EVALUATION | 44 5. How were the organizations selected for grant approval? (Including size, nature and scope) Following which selection criteria? (Was there a numeric scoring system? 6. Will grantees receive further grants? Based on what criteria? 7. Were grant applicants requested to provide information on their prior engagements with international development sector? How did their having received prior assistance from the international development sector affect the grant decision? Was it viewed favorably? Unfavorably? Indifferently? 8. What are the most notable results that it helped you achieve in terms policy reform (Indirectly?), support to entrepreneurs (directly or indirectly?), employment creation (directly or indirectly?), and access to finance (directly?) 9. What are the outstanding challenges (for achievement of those)? 10. Did you ever notice any evidence of grant recipients developing any kind of “dependency” on aid from donors vs. independent generation of profits? If so, provide details? What % of grant recipients would survive without funding from PSD or other donors? 11. What do you consider to be the strengths and weaknesses of the grant program? 12. What % of grant recipient organizations are women-owned? What % of the budget is allocated toward grants to women-owned organizations? Do you have any recommendations regarding how a grants program could be improved to better enable grant recipients to promote policy reform, economic benefits, support to entrepreneurs and access to finance to women? youth? 13. Did the PSD TA contribute (directly or indirectly) to: Policy Reform – how? Economic growth – how? Entrepreneurship – how? Access to Finance – how? (Food security) How do you know? Provide examples. How many people would you estimate PSD has helped, in terms the above as result of the TA only? (directly or indirectly?) 14. How did the TA affect the economic state of most recipients (for which pillar, directly or indirectly)? To your knowledge, did TA in any way enable or improve the likelihood of the organizations to operate on a self-sustainable basis? Can you estimate the percent of cases where this worked and did not work? 15. Was TA ever a requirement in relation to a grant application in progress? (Fixed Obligation Grants) 16. Which types of TA were most effective? Least effective? Why? 17. What do you consider to be the strengths and weaknesses of PSD’s TA program in general? 18. Do you have any recommendations regarding how TA could be improved to better enable a grantee to improve the business environment through: policy reform (Indirectly?), support to entrepreneurs (directly or indirectly?), employment creation (directly or indirectly?), and access to finance (directly?), especially (including for) for women and youth? 19. Did you recognize any patterns concerning overall success and failure of the support rendered, (for each pillar, directly or indirectly) i.e. types of grants, types of TA, combinations of both, types of granted items, geographic locations, profile of management, grant use, grant size, industries or disciplines, characteristics of owners, stage of development? 20. Were most companies/organizations considered to be “private sector” at the time of the engagement? If not, did the grant and/or TA help them to move in the direction to become self-sustainable? If so, how? Provide examples. 45 | USAID PSDA MID-TERM EVALUATION USAID.GOV 21. Are you aware of any interactions that the organizations receiving assistance have had with other USAID projects? Have organizations / beneficiaries received benefits from any other USAID projects prior to engagement with PSD? After engagement with PSD? If so, describe – which projects and the nature of the assistance received? 22. Have PSD grants and TA affected the attraction of other private sector investors and partners? Other international donors? Who are the co-investors/partners, if any? Provide examples 23. How did the support (TA or grant) help beneficiaries to (directly or indirectly) improve their livelihoods? Examples? PROJECT MANAGEMENT SPECIFIC QUESTIONS Allocations by intervention type: -What % of the budget is allocated to grants? -What % of the budget is allocated towards TA? -What % of the budget is allocated toward the promotion of A2F and the Development Credit Authority (DCA) or is that separate from the budget? -Are there any forecasted changes to the above allocations? -What are the results from DCA support? Numbers. Allocations by targeted beneficiary: -What % of the budget and labor effort is allocated toward policy reform? Support for entrepreneurship? Access to finance? -Are there any forecasted changes to the above allocations? Context, measurement and indicators -In which context were the majority of indicators drafted – many of them are “custom” to this project. How does that relate to USAIDs overall global system of indicators and measurement of impact and results? -It appears there is a heavy emphasis on general indicators i.e. “number of media stories”, “number of visitors to websites”, “number of workshops” etc. Is there a priority concerning the indicators – whereby more of the budget and effort are allocated more toward some activities and less toward others (consider ACTI context and structure here)? Is that likely to change over the second half of the project? -Are you aware of any IRs that have been added or dropped since the beginning of the project? (why not a separate IR for grants?) -How much emphasis has been placed on FTF indicators? Is that likely to change? How would you gauge PSD’s ability to react and evolve in Myanmar’s changing environment? -How would you characterize the experience in working with Myanmar’s inexperienced NLD government? Project changes -Are you aware of any Contract Modifications that have occurred or are pending? Focus (gender, youth) -How much emphasis has the project placed on women? Youth? Is that likely to change over the second half of the project? What efforts were used to ensure women and youth were included in project activities? Donor coordination -Is there much overlap concerning multiple international donor support to the same organizations receiving support from PSD? Is there a due diligence process which determines to what extent specific organizations are receiving which specific support. Has there been any communication with other donors to identify potential overlap? Future design USAID.GOV USAID PSDA MID-TERM EVALUATION | 46 -Assuming you were not involved in drafting the original project documents and knowing what you now know, how would you create a fresh project in Myanmar to accomplish what PSD sets out to accomplish? -Specifically, what intervention type (grants, TA, DCA) in this project should be scaled up and what should be scaled back or abandoned? -Specifically, what targeted beneficiary efforts (policy reform, support to entrepreneurship, Access Two Finance (A2F) efforts in this project should be scaled up and what should be scaled back or abandoned? -Any further recommendations? IMPLEMENTING PARTNER (Nathan, JE Austin, Internews) # Interview Question Responses Grants Program 1. How is the program structured as part of the overarching regional ASEAN Connectivity through Trade and Investment (ACTI) program? How do the pillars/areas/components fit in (TA vs grants and nature/scope) ● 2. Did the PSD grants contribute to: Policy Reform – how? Economic growth – how? Entrepreneurship – how? Access to Finance – how? (Food security) How do you know? Provide examples. Were they designed accordingly? ● How many people would you estimate PSD has helped, in terms the above as result of the grants only? ● 3. Describe the types of people helped. How have they been helped? ● Benefits to poor: In general, did the grants contribute to economic benefits in the form of poverty alleviation? Income enhancement – estimated % increase? Job creation? (directly or indirectly)? ● Can you generally estimate the number of jobs created? For women? For youth? How do you know? ● 4. Are you aware of any difficulties that occurred in relation to the grant program? If so, describe. ● Grant Decision 5. How were the organizations selected for grant approval? (Including size, nature and scope) Following which selection criteria? (Was there a numeric scoring system? 6. How were the grant amounts determined? Will grantees receive further grants? Based on what criteria? ● 7. Were grant applicants requested to provide information on their prior engagements with the international development sector? How did their having received prior assistance from the international development sector affect the grant decision? Was it viewed favorably? Unfavorably? Indifferently? ● 8. What are the most notable results that it helped you achieve in terms policy reform (Indirectly?), support to entrepreneurs (directly or indirectly?), employment creation (directly or indirectly?), and access to finance (directly?) ● 9. What are the outstanding challenges (for achievement of those)? ● 10. Did you ever notice any evidence of grant recipients developing any kind of “dependency” on aid from donors vs. independent generation of profits? If so, provide details? What % of grant recipients would survive without funding from PSD or other donors? ● 47 | USAID PSDA MID-TERM EVALUATION USAID.GOV 11. What do you consider to be the strengths and weaknesses of the grant program? ● 12. What % of grant recipients are women? What % of the budget is allocated toward grants to women? ● Do you have any recommendations regarding how a grants program could be improved to better enable grant recipients to promote policy reform, economic benefits, support to entrepreneurs and access to finance to women? youth? ● Technical Assistance/Capacity Building/Training 13. Did PSD TA contribute (directly or indirectly) to: Policy Reform – how? Economic growth – how? Entrepreneurship – how? Access to Finance – how? (Food security) How do you know? Provide examples. How many people would you estimate PSD has helped, in terms the above as result of the TA only? (directly or indirectly (for all above)?) ● 14. How did the TA affect the economic state of most recipients (for which pillar, directly or indirectly)? To your knowledge, did TA in any way enable or improve the likelihood of the organizations to operate on a self-sustainable basis? Can you estimate the percent of cases where this worked and did not work? ● 15. Were grants ever contingent on TA having been delivered and some type of certification made? (Fixed Obligation Grants) ● 16. Which types of TA were most effective? Least effective? Why? ● 17. What do you consider to be the strengths and weaknesses of PSD’s TA program in general? ● 18. Do you have any recommendations regarding how TA could be improved to better enable grantees to improve the business environment through: policy reform (Indirectly?), support to entrepreneurs (directly or indirectly?), employment creation (directly or indirectly?), and access to finance (directly?), especially (including for) for women and youth? ● General 19. Did you recognize any patterns concerning overall success and failure of the support rendered, (for each pillar, directly or indirectly) i.e. types of grants, types of TA, combinations of both, types of granted items, geographic locations, profile of management, grant use, grant size, industries or disciplines, characteristics of owners, stage of development? ● 20. Were most companies/organizations considered to be “private sector” at the time of the engagement? If not, did the grant and/or TA help them to move in the direction to become self-sustainable? If so, how? Provide examples. ● 21. Are you aware of any interactions that the organizations receiving assistance have had with other USAID projects? Have organizations received benefits from any other USAID projects prior to engagement with PSD? After engagement with PSD? If so, describe – which projects and the nature of the assistance received? ● 22. Have PSD grants and TA affected the attraction of other private sector investors and partners? Other international donors? Who are the co-investors/partners, if any? Provide examples ● 23. How did the support (TA or grant) help beneficiaries to (directly or indirectly) improve their livelihoods? Examples? ● USAID.GOV USAID PSDA MID-TERM EVALUATION | 48 PROJECT MANAGEMENT SPECIFIC QUESTIONS Allocations by intervention type: -What is the reasoning and process behind each budget decision? - How does your organization prioritize reforms? - How does your organization prioritize activities? Grants, TA, DCA - How does your organization prioritize budget allocation? -What % of the budget is allocated to grants? -What % of the budget is allocated towards TA? -What % of the budget is allocated toward the promotion of A2F and the Development Credit Authority (DCA) or is that separate from the budget? -Are there any forecasted changes to the above allocations? - What are the results from the DCA support? Numbers. - What are the objectives of the DCA guarantees? (objectives of the DCA loans are 1) more individual loans; and 2) larger size). Have the objectives been met successfully? Allocations by targeted beneficiary: -What % of the budget and labor effort is allocated toward policy reform? Support for entrepreneurship? Access to finance? -Are there any forecasted changes to the above allocations? Context, measurement and indicators -In which context were the majority of indicators drafted – many of them are “custom” to this project. How does that relate to USAIDs overall global system of indicators and measurement of impact and results? -It appears there is a heavy emphasis on general indicators i.e. “number of media stories”, “number of visitors to websites”, “number of workshops” etc. Is there a priority concerning the indicators – whereby more of the budget and effort are allocated more toward some activities and less toward others (consider ACTI context and structure here)? Is that likely to change over second project half? -Are you aware of any IRs that have been added or dropped since the beginning of the project? (why not a separate IR for grants?) -How much emphasis has been placed on FTF indicators? Is that likely to change? -How would you gauge PSD’s ability to react and evolve in Myanmar’s changing environment? -How would you characterize the experience in working with Myanmar’s inexperienced NLD government? Project changes -Are you aware of any Contract Modifications that have occurred or are pending? Focus (gender, youth) -How much emphasis has the project place on women? Youth? Is that likely to change over the second half of the project? What efforts were used to ensure women and youth were included in project activities? Donor coordination -Is there much overlap concerning multiple international donor support to the same organizations receiving support from PSD? Is there a due diligence process which determines to what extent specific organizations are receiving which specific support. Has there been any communication with other donors to identify potential overlap? Future design -Assuming you were not involved in drafting the original project documents and knowing what you now know, how would you create a fresh project in Myanmar to accomplish what PSD sets out to accomplish? -Specifically, what intervention type (grants, TA, DCA) in this project should be scaled up and what should be scaled back or abandoned? -Specifically, what targeted beneficiary efforts (policy reform, support to entrepreneurship, A2F) efforts in this project should be scaled up and what should be scaled back or abandoned? -Any further recommendations? 49 | USAID PSDA MID-TERM EVALUATION USAID.GOV GOM Institutions (National Government) # Interview Question Responses 1. What do you know about the PSD activity? ● 2. What has been your interaction or involvement with the program? How long have you been engaged with the program? ● 3. Who are some of the key PSD personnel that you had an opportunity to interact with? What was your responsibility at that point of interaction? ● 4. Whom would you consider to have been the biggest beneficiaries of the PSD program? ● 5. Specifically describe the TA assistance you received from PSD: Capacity building Training Gov’t champion Technical analysis and report Embedded advisor Other? ● 6. What were the intended results of the TA? ● 7. Which factors influenced the achievement or non-achievement of intended results? ● 8. How has your office benefitted specifically? Have you received help in: Policy Reform? Assistance with drafting legislation or policy? What? Assistance with comments and advice? Organizing stakeholder meetings/workshops? Capacity building training? Other? Please describe: How is this being applied in practice? Examples ● 9. In your opinion, would you consider PSD to have achieved its goals of improving support to policy reform, which will lead to economic growth? Why or why not? (i.e. What has been the benefit of having PSD operating in your county?) How? Examples? ● 10. What would you say have been PSDs major policy reform contributions which have led to the betterment of livelihoods to those in need: ● a. Economic benefits & employment (poverty alleviation/ Income enhancement) to the beneficiaries? especially the women and youth? ● b. Food Security and nutrition? For women? ● c. Local or National? (Practical examples?) 11. Are you aware of any PSD efforts to improve A2F, mobilize savings, expand insurance programs? Which? Are you aware of the Development Credit Authority (DCA) and its USAID.GOV USAID PSDA MID-TERM EVALUATION | 50 function? To what extent would you say PSD activities have improved access to financial services? Recommendations? 12. Are you aware of any PSD efforts to support entrepreneurship through the encouragement of start-ups and promotion innovation? Recommendations? 13. What tangible benefits would you consider the beneficiaries to have received from this program? 14. What would you consider to have been major challenges to the achievement of the program goals? ● 15. How would you recommend PSD alter its activities, if at all, to better achieve policy reform results? ● 16. Do you think the contributions made by PSD will be able to stand on their own after the end of the program i.e. be self-sustainable? ● 17. What support did the government provide to PSD’s efforts to assist in policy reform? 18. Do you have any thoughts on how the Myanmar government can improve in its cooperation with PSD to improve the policy environment? ● 19. If USAID were to fund an activity similar to PSD, what kind of support would you suggest as a government institution? ● 20. What other International donor funded programs are being implemented in the country which are doing similar work as PSD, if any? Overlap? ● 21. Are you aware of any interaction or integration of PSD with any of the above￾mentioned projects? Which projects? ● 22. Do you believe PSD worked closely enough with the GoM to jointly accomplish the objectives? ● Microfinance Institutions (MFIs) Background Information Introduce and provide history of your MFI: Established: Who are your shareholders by %? Loan book size in terms of outstanding loans in USD amount and number of loans? Default rate? % NPLs? Profile of LOs: How many LOs: How many branches? Borrower profile? Loan assessment methodology: Average loan size? Interest rate? Term? Collateral requirement? Group or individual? Is your MFI profitable? Self-sustainable? How close to profitable? Is the MFI itself in debt? Future plans of the MFI: 51 | USAID PSDA MID-TERM EVALUATION USAID.GOV How did you first make contact with PSD? Describe this encounter: # Interview Question Responses Grants Program – Describe the grant terms and conditions. 1. How did the PSD grant to your MFI contribute to: Policy Reform – how? Economic growth – how? Entrepreneurship – how? Access to Finance – how? (Food security) How do you know? Provide examples. Were they designed as such, accordingly? ● 2. Describe the types of people helped (and how selected?). How have they been helped? ● Benefits to poor: In general, did the grant to your MFI contribute to economic benefits in the form of poverty alleviation? Income enhancement – estimated % increase? Job creation? (directly or indirectly) How? ● 3. Can you generally estimate the number of jobs created? For women? For youth? How do you know? ● Grant Decision (these questions only asked to assess familiarity with DCA) 4. Are you aware how were the MFIs were selected for grant approval? Following which selection criteria? Was there a numeric scoring system? 5. Are you aware how the grant amounts were determined? Will grantees receive further grants? Based on what criteria? ● 6. Were grant applicants requested to provide information on their prior engagements with the humanitarian assistance sector? How did their having received prior assistance from the international development sector affect the grant decision? Was it viewed favorably? Unfavorably? Indifferently? ● 7. Have you received grants or loans from IFI’s? If so, which ones? How much? ● 8. What do you consider to be the strengths and weaknesses of the grant program? Recommendations? ● 9. DCA: Are you working with the Development Credit Authority (DCA)? Describe your interaction with this organization and your experience in general – how does it all work? - What % of your portfolio is DCA guaranteed? - How do you decide which loans get a DCA guarantee? - Are these registered or approved in some way with the DCA so they can be later identified in the event of a default? (so cannot slot in all NLPs) ● 10. Have you had defaulted loans reimbursed under the DCA? If so, how much in terms of USD? How many loans in total? ● 11. Have you made loans that you otherwise would not have made had you not had the DCA guarantee? If so, what % of the loans receiving DCA guarantee would you otherwise not have made? How many individuals would you estimate you’ve helped as a result of DCA? Do you have any recommendations regarding how the DCA program could work better? ● 12. What % of grant recipients are women? What % of the budget is allocated toward grants to women? ● Do you have any recommendations regarding how a grants program could be improved to better enable MFI grant recipients to promote policy reform, economic benefits, support to entrepreneurs and access to finance to women? youth? ● USAID.GOV USAID PSDA MID-TERM EVALUATION | 52 Technical Assistance/Capacity Building/Training – describe the TA you received: 13. Did the PSD TA contribute (directly or indirectly) to: Policy Reform – how? Economic growth – how? Entrepreneurship – how? Access to Finance – how? (Food security) How do you know? Provide examples. ● How many people would you estimate PSD has helped, in terms the above as result of the TA only? (directly or indirectly?) ● 14. How did the TA affect the economic state of your MFI? Did TA in any way enable or improve the likelihood of your MFI to operate on a self-sustainable basis? ● 15. Were grants ever contingent on TA having been delivered and some type of certification made? (Fixed Obligation Grants) ● 16. Which types of TA were most effective? Least effective? Why? ● 17. What do you consider to be the strengths and weaknesses of PSD’s TA program in general? ● 18. Do you have any recommendations regarding how TA could be improved to better enable your MFI to improve the business environment through: policy reform (Indirectly?), support to entrepreneurs (directly or indirectly?), employment creation (directly or indirectly?), and access to finance (directly?), especially (including for) for women and youth? ● General 19. Was your MFI considered to be “private sector” at the time of the engagement? If not, did the grant and/or TA help you to move in the direction to become self￾sustainable? If so, how? Provide details. ● 20. Has your MFI received assistance from international donors? Who and what kind of assistance? – USD amounts. ● 21. Has your MFI received benefits from any other USAID projects prior to engagement with PSD? After engagement with PSD? If so, describe – which projects and the nature of the assistance received? ● 22. Have PSD grants and TA affected the attraction of other private sector investors and partners or donors? Other international donors? Who are the co￾investors/partners, if any? Provide examples. ● 23. What are the most notable results (for both grants and TA) that helped you achieve in terms policy reform (Indirectly?), support to entrepreneurs (directly or indirectly?), employment creation (directly or indirectly?), and access to finance (directly?), ● 24. What are the outstanding challenges (for achievement of those)? ● 53 | USAID PSDA MID-TERM EVALUATION USAID.GOV International Financial Institutions (IFIs)/Other Donors # Interview Question Responses 1. What do you know about the Private Sector Development (PSD) activity? ● 2. What has been your organization’s interaction with PSD? What has been the level of interaction? Field level or Office of Economic Growth partners’ meetings level in Yangon? ● 3. What are some of the key PSD supported efforts/interventions that your organization has had the opportunity to interact with, if any? Policy Reform Economic growth Entrepreneurship Access to Finance Describe each of the interactions: ● 4. Who would you consider to be the biggest beneficiaries of these interventions? How? How do you know? ● 5. In your opinion, would you consider PSD to have achieved its goals of improving the - economic policy environment, - support to entrepreneurship - access to finance for MSMEs? Has food security/nutrition been a consideration to your knowledge? Has there been any emphasis specifically on benefits women and youth as envisaged at the start of the program? What have been the nature of jobs created for women? Are they different than the jobs created for men? How? ● 6. Based on your interaction with PSD, what would you say have been the major contributions of the PSD in terms of: ● Policy Reform Entrepreneurship Economic growth/Economic benefits & employment (poverty alleviation/ Income enhancement) to the beneficiaries? Especially the women and youth? Access to Finance Which of the interventions have been most successful? Least successful? 7. What would you consider the major overall achievements of the PSD in Burma? What would you consider to have been major challenges to the achievement of the program overall goal(s)? ● 8. From your interaction with the PSD, what would you consider to be the most important lessons learned? ● 9. Have you given grants and/or TA to the same organizations that PSD is supporting? Which? Based on which criteria? Are they producing positive results? ● 10. Do you think the innovations already supported by your organization and PSD will be able to stand on their own after the end of the program i.e. be self-sustainable? Why, why not? What else is needed? ● 11. Coordination: Are there many IFI operators working in the same space: policy reform, entrepreneur support, A2F? Have there been any synergies that have been developed during the period of interaction? What are the implications of this? ● 12. Do you have any recommendations on how to improve PSD’s operations in order to make greater impact? Do you plan further cooperation with PSD? If so, in what way? If USAID were to fund an activity similar to PSD, what approach would you propose? ● USAID.GOV USAID PSDA MID-TERM EVALUATION | 54 NGOs, Business Associations, MSMEs, Entrepreneurs, Producers Organizations, Women’s Groups, Community-Based Organizations (CBOs), Journalists The ET utilized questions as relevant for each respondent type. All questions were not asked to each respondent type. # Interview Question Responses Introduce your organization and its history and goals: Grant details: 1. How did you first hear about the PSD program? ● 2. When did you receive the grant? ● 3. Describe the grant assistance you received from PSD? i.e. What was the size of your grant and for which purpose (exactly)? ● 4. What is your understanding of why the USAID/PSD provided you with a grant? ● 5. Did your organization meet/fulfill this purpose? If so, how? ● 6. Describe the mechanics of receiving the grant, i.e. the application process and the terms and conditions of the grant as well as the dedicated uses of the funds and how the use of funds contributed to the betterment of livelihoods of targeted beneficiaries. How was this verified and confirmed? ● 7. What was the economic state of your company /organization prior to the grant? Was your company/organization profitable? Self-sustainable? ● 8. How did the grant contribute to (provide best and worst examples – distinguish between direct and indirect contribution to economic opportunities?) ● 8a. Job creation? ● 8b. Improved household income? 8c. Improving policy reform? 8d. Entrepreneurship support? (examples) 8e. Access to finance? How do you know? How many people would you estimate your project has helped, in terms of improved economic livelihood as a result of the grant only? (Distinguish between direct and indirect contribution?) Or did it support economic reforms that indirectly contribute to facilitating employment? ● 9. Women: Did the grant contribute to poverty alleviation for women? Income enhancement? Job creation? (direct or indirectly)? How do you know? How many women do you think your project has helped, in terms of economic benefits as a result of the project support only? (Number of jobs created? Estimated increase in income as a percent?) (Distinguish between direct and indirect contribution) Can you tell us about women’s groups which were able to receive grants? Did that face any challenges different from non-women only grant recipients? If so, which challenges? ● 10. How did the grant affect the state of your company/organization in terms of revenue generation, profitability and self-sustainability? Which main improvements have been realized if any? ● 11. What factors contributed to the success/ failure of your grant? Impact on “Broad￾based economic activities” in Myanmar? ● 12. Do you have any recommendations regarding how a grants program could be improved to better enable yourself/ company/organization (or any company) to accomplish intended goals? ● Technical Assistance/Capacity Building/Training details: 13. Specifically describe the TA assistance you received from PSD? ● 55 | USAID PSDA MID-TERM EVALUATION USAID.GOV 14. Describe any training sessions or workshops you attended in detail. Did you receive any benefits for attending? What were the primary lessons? What skills did you acquire? How are you using/applying them? Are they checking on how you are applying them? ● 15. How did the TA affect the economic state of your company/organization? Which improvements were realized if any? Has the TA served to increase the effectiveness of your company/organization? (How/examples)? ● 16. Is your company/organization considered to be “private sector”? If not, did the TA help you to move in the direction to become self-sustainable or in any way? If so, how? ● 17. What were you taught that you still use? What do you no longer use? Or what are you now using /doing differently? Applied how? Checked how? ● 18. What challenges did you face in applying the knowledge from the TA? Was the training relevant? ● 19. How did the Technical Assistance (directly or indirectly) contribute to: a. Improving policy reform? How? b. Entrepreneurship support? How? c. Job creation / Improved household income? How? d. Access to finance? How? How do you know? ● How many people would you estimate your project has helped, in terms of improved economic livelihood as a result of the project support (grant / TA) only? (Directly or indirectly)? How many were women? ● 20. Which TA or training would be useful to you in the future? Would you be willing to pay for TA? Why? Why not? How much would you pay? What for? ● 21. Have you had any interactions with other donor projects as a result of your engagement with PSD? Have you received benefits from any other donor projects? If so, describe – which projects and assistance received. Easy /how to apply for it? Do you anticipate receiving benefits from other donor organizations? When/how? ● 22. Do you have any recommendations regarding how a grants or TA program could be improved to better enable recipients to promote policy reform, economic benefits, support to entrepreneurs and access to finance to women? youth? ● USAID.GOV USAID PSDA MID-TERM EVALUATION | 56 FGD QUESTIONNAIRES Technical Assistance/Capacity Building/Training Recipients # Interview Question ● 1. Specifically describe the TA assistance you received from PSD: ● Business Reporters Training ● A2F (MFI) ● A2F (DCA 1) ● IPR ● A2F (FRD) ● Consumer Protection ● A2F (Insurance) ● National Quality Infrastructure (Standard Law) ● Trade and Investment (SPS) ● Entrepreneurship (IT) ● A2F (SME lending) ● Food (Food Law) ● 2. What were the intended results of the TA? ● 3. Which factors influenced the achievement or non-achievement of intended results? ● 4. Describe any training sessions or workshops you attended in detail. Who provided the training? Did you receive any benefits for attending? What were the primary lessons? What skills did you acquire? How are you using/applying them if at all? Are they checking on how you are applying lessons learned? ● 5. Has the TA served to increase the effectiveness of your company/organization? (How/examples)? How did the TA affect the economic state of your organization (if applicable)? Which improvements were realized if any? ● 6. What were you taught that you still use? What do you no longer use? Or what are you now using /doing differently? Applied how? Checked how? ● 7. Have you encountered and challenges in applying the knowledge? If so, what? Was the TA relevant and useful? ● 8. How did the Technical Assistance (directly or indirectly) contribute to: a. Improving policy reform? How? b. Entrepreneurship support? How? c. Job creation / Improved household income? How? d. Access to finance? How? How do you know? ● How many people would you estimate the PSD training has enabled you to help, in terms of improved economic livelihood as a result of the project support (grant /TA) only? (Directly or indirectly)? How many were women? ● 57 | USAID PSDA MID-TERM EVALUATION USAID.GOV 9. Which TA or training would be useful to you in the future? Would you be willing to pay for TA? Why? Why not? How much would you pay? What for? ● 10. Have you had any interactions with other donor projects as a result of your engagement with PSD? Have you received benefits from any other donor projects? TA? Grants? If so, describe – which projects and assistance received. Easy /how to apply for it? Do you anticipate receiving benefits from other donor organizations? When/how? ● 11. Do you have any recommendations regarding how a TA program could be improved to better enable recipients to promote policy reform, economic benefits, support to entrepreneurs and access to finance to women? youth? ● 12. Is your company/organization considered to be “private sector”? If not, did the TA help you to move in the direction to become self-sustainable or in any way? If so, how? ● Is there anything that we missed that you would like to talk about? What is the most important point we discussed? USAID.GOV USAID PSDA MID-TERM EVALUATION | 58 DCA Borrowers # Interview Question ● 1. Describe your business. ● 2. What was your first interaction with the MFI? Which MFI? Who, what, where, when, why, how? ● 3. Individual or group? ● 4. What interest rate? Amount? Term? Repayment schedule? Collateral? ● 5. Have you been able to repay the loan? Why not? With or without difficulty? ● 6. Prior to your contact with the lender, have you taken loans in the past? With what result? Had you approached other financial institutions or been approached by them? Had you experienced difficulty in obtaining finance in the past? Are there many lenders in your area? Describe and compare them? Why are MFIs working in your area? ● 7. Why do you think you were able to qualify for this loan (despite x,y,z which may have precluded you from receiving finance in the past)? ● 8. Would you like to receive another similar loan? Why? Why not? ● 9. Describe any training sessions or workshops you attended in detail. Who provided the training? Did you receive any benefits for attending? (Cash, lunch) What were the primary lessons? What skills did you acquire? How are you using/applying them if at all? Is anyone checking on how you are applying lessons learned? ● 10. How did the TA affect the economic state of your family? Which improvements were realized if any? Improvement in livelihood? Job creation? ● 11. What were you taught that you still use? What do you no longer use? Or what are you now using /doing differently? Applied how? Checked how? ● 12. Have you encountered and challenges in applying the knowledge? If so, what? Was the TA relevant and useful? ● Is there anything that we missed that you would like to talk about? ● What is the most important point we discussed? ● 59 | USAID PSDA MID-TERM EVALUATION USAID.GOV ANNEX III: DISCLOSURE OF CONFLICT OF INTEREST FORMS USAID.GOV USAID PSDA MID-TERM EVALUATION | 60 61 | USAID PSDA MID-TERM EVALUATION USAID.GOV USAID.GOV USAID PSDA MID-TERM EVALUATION | 62 63 | USAID PSDA MID-TERM EVALUATION USAID.GOV ANNEX IV: EVALUATION TEAM MEMBER PROFILES Qualifications and Experience of Core Evaluation Team Members Team Leader and Senior Evaluation Specialist, Mr. Benjamin Ryan: Benjamin Ryan is a senior-level accomplished USAID evaluation team leader and specialist with more than two decades of experience in private sector development. He brings significant Burma private sector work experience and country knowledge, having worked and lived in Burma since 2008. As Saratoga Limited’s Chairman in Burma, Mr. Ryan has provided extensive due diligence of Burmese companies financial institutions in the private sector to determine suitability for direct investments. In this capacity, he has conducted comprehensive research of the Yangon, Burma property development market, established high-level relationships with government officials, mediated private equity deals for international mining investors, researched and mapped promising mineral deposits for foreign investors. Mr. Ryan has also established high-level relationships with Burmese government officials. Mr. Ryan brings significant experience in conducting performance evaluations, including experience and knowledge related to similar reforms in countries in transition. As Evaluation Team Leader with MSI, he conducted a structured evaluation of a USAID agribusiness SME innovation development project, where he evaluated the performance and ability of grant and technical assistant recipients to sustainably improve food security and improve long-term economic prospects for intended beneficiaries. Also, as Evaluation Team Leader in Liberia, he assessed and selected the most prospective agribusiness SMEs to receive finance and other interventions with the aim to create long-term employment in self-sustaining and profitable agribusinesses as part of USAID’s Feed the Future (FTF) private sector development initiatives. Mr. Ryan has also held high level SME banking positions in Azerbaijan, Eastern Europe, Afghanistan and Pakistan. Additionally, Mr. Ryan ran a private equity fund in Uzbekistan, Turkmenistan and Tajikistan. Mr. Ryan holds a Master of Business Administration from Thunderbird School of Global Management, with a focus on Finance, and a Bachelor’s degree from Providence College, with a major in Economics. Mr. Ryan was also a Commissioned Officer in the United States Army and is fluent in Mandarin Chinese and English. Private Sector Analyst, Mr. Evert de Witte: Mr. de Witte has over 15 years of demonstrated evaluation experience related to the private sector in Asia/South-East Asia, particularly in Burma, where he currently lives and works. As Lead M&E Advisor on DFID’s Business for Shared Prosperity Program, he developed and maintained an M&E system to establish the Burma Enterprise Opportunity Facility in partnership with government, businesses, civil society, and other development partners to finance innovative new projects. Mr. de Witte also brings strong economic growth expertise and documented experience in the business and financial sectors in Burma. As Evaluation Team Lead on DANIDA’s Responsible Business Project, he developed and piloted baseline tools to support the development, engagement, and investment of private sector through responsible investment and business practices to ultimately lead to sustainable and inclusive growth. Mr. de Witte is familiar with USAID’s evaluation policies and guidance, having served as Evaluation Team Lead on a Mid-Term Evaluation of USAID’s Mid-term Evaluation of Shae Thot “The Way Forward.” In this capacity, he designed a quantitative household survey, using a mixed methodology approach, conducting ten focus group discussions (FGDs) and 57 in-depth key informant interviews (KIIs) to provide qualitative data on program outcomes and integration. Mr. de Witte holds a Masters in Economic Development and International Cooperation and is fluent in English. Local Evaluation Specialist, Mr. Ye Min Thant: Mr. Thant has over 7 years of demonstrated experience in monitoring and evaluation, project planning, and impact monitoring. As a National Research Evaluation Consultant, he conducted a mid-term evaluation of the My-Justice Program, implemented by British Council in partnership with more than 30 INGOs and CSOs, and funded by European Union (EU). Additionally, he participated in desk reviews, and facilitated key informant interviews (KIIs), and focus group discussion (FGDs) with project stakeholders in Yangon, Nay Pyi Taw, Taunggyi and Mawlamyine. USAID.GOV USAID PSDA MID-TERM EVALUATION | 64 Mr. Thant’s monitoring and evaluation experience stem from sectors including infrastructure, agriculture, fisheries, nutrition, and governance. As a National Evaluator for the Livelihood and Food Security Trust Fund (LIFT), managed by UNOPS, he conducted a mid-term review of the Kan Latt project under Tat Lan II program funded by Livelihoods and Food Security Trust Fund (LIFT) and implemented separately by Save the Children, International Rescue Committee (IRC) and Care Myanmar. In addition to his monitoring an evaluation experience, Mr. Thant also brings strong research experience in fields such as agriculture and market system development. As a Research Assistant for the World Vegetable Centre (AVRDC), he conducted a study on the value chains and post-harvest practices of key vegetables in Myanmar. Mr. Thant holds a master’s in policy studies from the University of New Brunswick and speaks Burmese, English, Mandarin Chinese and French. Local Private Sector Development Specialist, Ms. Aye Aye Khaing: Ms. Khaing has over 14 years of demonstrated experience in monitoring and evaluation, specifically in the agricultural sector. As a Team Leader, working for the Asia Development Bank, she prepared detailed work-plan and implementation schedule, reviewed and extracted lessons from related projects on agribusiness value chains, developed selection criteria and evaluation procedures for related investment subprojects. As an Independent Consultant, working for the International Trade Centre, Palas des Nation, she developed baseline questionnaires for beneficiary groups and conducted key informant interviews (KIIs) and focus group discussions (FGDs) with farmers, collectors, storage facilities, processors, exporters, regulatory bodies and other relevant institutions. Ms Khaing has also experience working with the Institute for International Development-Australia. While serving as National Agricultural Specialist, she prepared regional development plans for sustainable economic development for the Danu zone and Shan State and conducted research on economic development opportunities for the Danu community. Ms. Khaing holds a bachelor’s degree in agriculture from Agriculture University in Burma and speaks Burmese and English. The evaluation team was supported by an in-country logistician and interpreters, and by Social Impact’s Headquarters (HQ) Management Team (including the SI Gender Specialist). Team Member Roles The ET, both in-country and HQ backstopping members, with their roles and responsibilities are presented in Table 12 below. Table 12: Evaluation Team Roles and Responsibilities Role Responsibilities Evaluation Team Team Leader & Senior Evaluation Specialist Mr. Benjamin Ryan ● Provide overall ET leadership and technical direction for the evaluation ● Serve as primary technical liaison with USAID/Burma ● Manage implementation of approved evaluation activities ● Oversee field data collection ● Lead briefing with USAID/Burma and other stakeholders ● Lead data analysis and development of high-quality deliverables Private Sector Analyst Mr. Evert de Witte ● Provide technical input into all evaluation tools and deliverables ● Lead sub-team during fieldwork ● Conduct data collection in collaboration with the TL ● Contribute to data analysis and development of high quality deliverables Local Evaluation Specialist Mr. Ye Min Thant ● Advise on the methodology and data collections tools ● Advise on local context ● Participate in desk review and data collection in collaboration with the TL ● Contribute to data analysis and development of deliverables 65 | USAID PSDA MID-TERM EVALUATION USAID.GOV Local Private Sector Development Specialist Ms. Aye Aye Khaing ● Advise on the methodology and data collections tools ● Advise on local context and private sector actors ● Participate in desk review and data collection in collaboration with the TL ● Contribute to data analysis and development of deliverables Logistician Ms. Faye Simons (Thei Thei Zaw Win) ● Provide logistical support including travel arrangements, meeting arrangements, and other administrative tasks as needed Interpreters (2) Ms. Su Wai Phyo Mr. Henry Aung Tun ● Translate written materials, interpretation in meetings (e.g., KIIs, FGDs) HQ Backstopping Staff Program Director Ms. Amanda Stek ● Ensure USAID satisfaction via routine check-ins ● Provide high-level guidance on methodology, major deliverables’, and any problems ● Oversee TL and HQ management staff ● Conduct in-depth review of deliverables using EQUI® quality assurance checkpoints Program Manager Mr. Soham Banerji ● Onboard the team and train them on SI procedures and quality standards. ● Issue and ensure contractual adherence to partner subcontract. ● Manage the budget by tracking LOE usage and other spending. ● Provide quality assurance on all deliverables Program Assistant Ms. Euphonise Loiseau ● Provide administrative and logistical support—arrange travel and lodging, process expense reports ● Copyedit and format deliverables Gender Specialist Ms. Noshaba Zafar ● Review all major deliverables to ensure appropriate addressing of gender considerations. USAID.GOV USAID PSDA MID-TERM EVALUATION | 66 ANNEX V: BIBLIOGRAPHY 1. USAID PSDA: Consolidated Quarter 1 and First Annual Report (July 1, 2015 to September 30, 2015) 2. USAID PSDA: Annual Report, FY 2016 10-31-16 submitted to USDA (Final) (4) 3. USAID PSDA: Year 1 Annual Work Plan (July 1,2015 to September 30,2016) 4. USAID PSDA: Second Annual work plan, FY 2017: (Mid-term Update) 5. USAID PSDA: Work plan, FY 2017: October 1, 2016 to September 30, 2017). 6. USAID PSDA: Third Annual Work plan, FY 2018: October 1, 2017 to September 30, 2018). 7. USAID PSDA: FY 16-1 Quarterly Report (October 1, 2015 to December 31, 2015. 8. USAID PSDA: FY 16 Q2 Quarterly Report (January1,2016 to March 31, 2016) 9. USAID PSDA: Quarterly Report First Quarter, Fiscal Year 2017 (October 1 to December 31, 2016) 10. USAID PSDA: Quarterly Report, Fiscal Year 2017: January to March 2017 11. USAID PSDA: Quarterly Report, Fiscal Year 2017: April 1, 2017 to June 30, 2017. 12. USAID PSDA: Quarterly Report, First Quarterly, Fiscal Year 2018: October 1 to December 31, 2017) 13. USAID PSDA: work plan plus M&E Plan- March 2017. 14. USAID PSDA: Monitoring and Evaluation Plan. 15. USAID PSDA: Grants Program. January 8,2018 16. USAID PSDA: Grants Manual Volume 1. 17. USAID PSDA: Grants Manual Volume 2. 18. Update on PSDA Grants 14, (October 24, 2017) 19. USAID PSDA: List of Grantees’ operation areas 20. Statue Report on PSDA Grants – Oct 25, 2017 (update on PSDA Grants, October 24, 2017) 21. “The obligations of membership: Myanmar in ASEAN” (June 2016) by Nathan Associates, commissioned by USAID, https://www.nathaninc.com/wp-content/uploads/2018/02/Buehrer￾Obligations-of-Membership.pdf 22. “The private sector and the governance of business in Myanmar” (Feburary 2016) by Nathan Associates, commissioned by USAID, https://www.nathaninc.com/wp￾content/uploads/2018/02/Bissinger-Private-Sector.pdf 23. “ICT development for innovation and growth in Myanmar” (June 2016) by Nathan Associates, commissioned by USAID, https://www.nathaninc.com/wp￾content/uploads/2018/02/MooreMadden-ICT-reducedsize.pdf 24. “Banking and finance in Myanmar: Present realities, future possibilities” (January 2016) by Nathan Associates, commissioned by USAID, https://www.nathaninc.com/wp￾content/uploads/2018/02/Turnell-Banking-and-Finance.pdf 25. “Strengthening macroeconomic management in Myanmar: Understanding Myanmar’s economic reform and growth dynamics” (February 2016) by Nathan Associates, commissioned by USAID, https://www.nathaninc.com/wp-content/uploads/2018/02/Bolnick-Strengthening-Macroeconomic￾Management-in-Myanmar.pdf 26. “New economic horizons for Myanmar: Scenarios for inclusive and sustainable growth” (January 2017) by Nathan Associates, commissioned by USAID, https://www.nathaninc.com/wp￾content/uploads/2017/12/Roland-Holst-New-Economic-Horizons-for-Myanmar.pdf 27. “Managing change: Executive policymaking in Myanmar” (May 2018) by The Asia Foundation 67 | USAID PSDA MID-TERM EVALUATION USAID.GOV ANNEX VI: LIST OF RESPONDENTS Organization M/F Type, date & location USAID 1 USAID M KII 27-Apr YANGON 2 USAID M KII 30-Apr YANGON Implementing Partners 3 Nathan Associates M KII 25-Apr YANGON 4 Nathan Associates F KII 25-Apr YANGON 5 Nathan Associates M KII 25-Apr YANGON 6 Nathan Associates F KII 25-Apr YANGON 7 Nathan Associates F KII 25-Apr YANGON 8 Internews M KII 26-Apr YANGON 9 Nathan Associates F KII 27-Apr YANGON 15-May 10 Nathan Associates M KII 27-Apr YANGON 11 Nathan Associates M KII 27-Apr YANGON 12 Nathan Associates F KII 27-Apr YANGON 13 Nathan Associates M KII 27-Apr YANGON 15-May 14 Nathan Associates F KII 27-Apr YANGON 15-May Grantees 15 Opportunities NOW M KII 2-May YANGON 16 Opportunities NOW M KII 2-May YANGON 17 Phandeeyar M KII 26-Apr YANGON USAID.GOV USAID PSDA MID-TERM EVALUATION | 68 18 Myanmar Microfinance Institute Association M KII 27-Apr YANGON 19 Intellectual Property Proprietors' Association (IPPAM) M KII 28-Apr YANGON 20 IPPAM F KII 28-Apr YANGON 21 Myanmar Consumers Union (MCU) M KII 30-Apr YANGON 22 MCU F KII 30-Apr YANGON 23 MCU F KII 30-Apr YANGON 24 Pyu Sin Saydana Action Group (PSSAG) F KII 30-Apr YANGON 25 Development for Environmental￾friendly Agriculture and Rural Life of Myanmar (DEAR Myanmar) M KII 1-May YANGON 26 Food Science and Technology Association (FoSTA) F KII 1-May YANGON 27 FoSTA F KII 1-May YANGON 28 FoSTA F KII 1-May YANGON 29 FoSTA F KII 1-May YANGON 30 Entrepreneurs du Monde (EdM) F KII 2-May DALA 31 EdM F KII 2-May DALA 32 EdM M KII 2-May DALA 33 National Young Woman's Christian Association (YWCA) F KII 2-May YANGON 34 YWCA F KII 2-May YANGON 69 | USAID PSDA MID-TERM EVALUATION USAID.GOV 35 YWCA F KII 2-May YANGON 36 YWCA F KII 2-May YANGON 37 YWCA F KII 2-May YANGON 38 Myanmar Engineering Society (MES - Grantee) M KII 2-May YANGON 39 Myanmar Engineering Society (MES - Grantee) M KII 2-May YANGON 40 Myanmar Engineering Society (MES - Grantee) F KII 2-May YANGON 41 UMFCCI (YANGON) M KII 30-Apr YANGON 42 Myanmar Business Answers, Micro Empire (Grantee): M KII 4-May YANGON 43 Myanmar Business Answers, Micro Empire (Grantee): F KII 4-May YANGON 44 Myanmar Development Institute (MDI) F KII 8-May Nay Pyi Taw 45 Development Resources International (DRI) M KII 8-May Nay Pyi Taw 46 DRI M KII 8-May Nay Pyi Taw 47 DRI M KII 8-May Nay Pyi Taw MFIs/Banks 48 Myanmar Development Planners (MDP) F 1-May YANGON 49 Proximity Finance M KII 26-Apr YANGON USAID.GOV USAID PSDA MID-TERM EVALUATION | 70 50 PACT Global Microfinance Fund M KII 2-May YANGON 51 PACT Global Microfinance Fund M KII 2-May YANGON 52 Opportunities NOW M KII 2-May YANGON 53 Opportunities NOW M KII 2-May YANGON 54 Pyu Sin Saydana Action Group (PSSAG) F KII 30-Apr YANGON 55 Entrepreneurs du Monde (EdM) F KII 2-May DALA 56 EdM F KII 2-May DALA 57 EdM M KII 2-May DALA 58 National Young Woman's Christian Association (YWCA) F KII 2-May YANGON 59 YWCA F KII 2-May YANGON 60 YWCA F KII 2-May YANGON 61 YWCA F KII 2-May YANGON 62 YWCA F KII 2-May YANGON 63 A-Bank M KII 30-Apr YANGON 64 LOLC (Sri Lankan MFI) M KII 4-May YANGON 65 LOLC F KII 4-May YANGON 66 ASA (Bangladeshi MFI) M KII 4-May YANGON 67 ASA M KII 4-May YANGON 68 ASA M KII 4-May YANGON 69 MAB Bank M KII 4-May YANGON 70 GRET Delta F 9-May Skype 71 Yoma Bank M KII 11-May YANGON 72 AYA Bank M KII 11-May YANGON 73 AYA Bank M KII 11-May YANGON 71 | USAID PSDA MID-TERM EVALUATION USAID.GOV IFIs/Donors 74 LIFT, UNOPS M 11-May YANGON 75 World Bank KII 11-May YANGON 76 IFC F KII 11-May YANGON 77 IFC F KII 11-May YANGON 78 ADB M 11-May Skype Business Associations 79 Myanmar Young Entrepreneurs Association (MYEA) M KII 11-May YANGON 80 MYEA M KII 11-May YANGON 81 MYEA M KII 11-May YANGON FGD (NGO Trainees) FGD at Internews with Business Journalists 82 Seven Day Journal F FGD 30-Apr YANGON 83 Mizzima (English issue) M FGD 30-Apr YANGON 84 Myanmar Times M FGD 30-Apr YANGON 85 Kumudra Journal F FGD 30-Apr YANGON 86 MRTV F FGD 30-Apr YANGON 87 Internews F FGD 30-Apr YANGON FGD at Phandeeyar with Start-up challenge participants 88 Recyglo F FGD 2-May YANGON 89 Amazing Express M FGD 2-May YANGON 90 MMtutor M FGD 2-May YANGON FGD at Phandeeyar with Founder Institute participants 91 Giant Pay M FGD 2-May YANGON 92 Beautiful Smiles (Myanmar) M FGD 2-May YANGON 93 Asia Art Connect F FGD 2-May YANGON 94 Vyra F FGD 2-May YANGON FGD at Phandeeyar with Makers' Space participants USAID.GOV USAID PSDA MID-TERM EVALUATION | 72 95 Phandeeyar M 2-May YANGON 96 MACALESTER COLLEGE’’ 22 M FGD 2-May YANGON FGD at Myanmar Engineering Society 97 Myanmar Engineering Society (MES - Grantee) M FGD 2-May YANGON 98 Myanmar Engineering Society (MES - Grantee) F FGD 2-May YANGON FGD (Government Trainees) FGD at Plant Protection Division under Dep of Agriculture 99 Livestock, Breeding and Veterinary Dept (LBVD) under Ministry of Agriculture, Livestock and Irrigation F FGD 4-May YANGON 100 Department of Agriculture (DoA), under Ministry of Agriculture, Livestock and Irrigation F FGD 4-May YANGON 101 DoA under Ministry of Agriculture, Livestock and Irrigation F FGD 4-May YANGON 102 Ministry of Agriculture, Livestock and Irrigation F FGD 4-May YANGON 103 Ministry of Agriculture, Livestock and Irrigation F FGD 4-May YANGON FGD at FRD (computer training participants) 104 Financial Regulatory Department under MOPF M FGD 10-May Nay Pyi Taw 73 | USAID PSDA MID-TERM EVALUATION USAID.GOV 105 FRD M FGD 10-May Nay Pyi Taw 106 FRD M FGD 10-May Nay Pyi Taw 107 FRD F FGD 10-May Nay Pyi Taw 108 FRD F FGD 10-May Nay Pyi Taw 109 FRD F FGD 10-May Nay Pyi Taw 110 FRD F FGD 10-May Nay Pyi Taw 111 FRD M FGD 10-May Nay Pyi Taw 112 FRD F FGD 10-May Nay Pyi Taw FGD at FRD (Risk-based Supervision Insurance Training participants) 113 Financial Regulatory Department under Ministry of Planning and Finance F FGD 10-May Nay Pyi Taw 114 FRD F FGD 10-May Nay Pyi Taw 115 FRD F FGD 10-May Nay Pyi Taw 116 FRD F FGD 10-May Nay Pyi Taw 117 FRD F FGD 10-May Nay Pyi Taw FGD (DCA borrowers) FGD at MMFA with training participants 118 LOLC M FGD 27-Apr YANGON 119 LOLC M FGD 27-Apr YANGON 120 Proximity Finance M FGD 27-Apr YANGON FGD with staff of A-Bank 121 A-Bank F FGD 30-Apr YANGON 122 A-Bank M FGD 30-Apr YANGON 123 A-Bank F FGD 30-Apr YANGON 124 A-Bank M FGD 30-Apr YANGON 125 A-Bank F FGD 30-Apr YANGON 126 A-Bank F FGD 30-Apr YANGON USAID.GOV USAID PSDA MID-TERM EVALUATION | 74 FGD with staff of MAB Bank 127 MAB Bank M FGD 4-May YANGON 128 MAB Bank F FGD 4-May YANGON 129 MAB Bank F FGD 4-May YANGON 130 MAB Bank M FGD 4-May YANGON GOM 131 Myanmar Customs Department (under Min of Planning and Finance) M KII 26-Apr YANGON 132 Department of Research and Innovation (under Ministry of Education) M KII 28-Apr YANGON 133 DRI under Min of Edu F KII 28-Apr YANGON 134 DRI under Min of Edu M KII 28-Apr YANGON 135 DRI under Min of Edu M KII 28-Apr YANGON 136 Dep of Trade (DoT), MoC KII 7-May Nay Pyi Taw 137 Dep of Trade (DoT), MoC KII 7-May Nay Pyi Taw 138 Dep of Trade (DoT), MoC KII 7-May Nay Pyi Taw 139 Dep of Trade (DoT), MoC KII 7-May Nay Pyi Taw 140 Dep of Trade (DoT), MoC KII 7-May Nay Pyi Taw 141 Dep of Trade (DoT), MoC KII 7-May Nay Pyi Taw 75 | USAID PSDA MID-TERM EVALUATION USAID.GOV 142 Department of Research and Innovation - Intellectual Property Division (under Ministry of Education): F KII 7-May Nay Pyi Taw 143 IP Division F KII 7-May Nay Pyi Taw 144 IP Division F KII 7-May Nay Pyi Taw 145 IP Division F KII 7-May Nay Pyi Taw 146 IP Division F KII 7-May Nay Pyi Taw 147 IP Division F KII 7-May Nay Pyi Taw 148 IP Division M KII 7-May Nay Pyi Taw 149 IP Division F KII 7-May Nay Pyi Taw 150 Department of Consumer Affairs (DoCA), MOC M KII 7-May Nay Pyi Taw 151 DoCA M KII 7-May Nay Pyi Taw 152 DoCA M KII 7-May Nay Pyi Taw 153 DoCA M KII 7-May Nay Pyi Taw 154 DoCA M KII 7-May Nay Pyi Taw 155 DoCA M KII 7-May Nay Pyi Taw 156 DoCA M KII 7-May Nay Pyi Taw 157 DoCA M KII 7-May Nay Pyi Taw 158 Myanmar Gem Enterprise (MGE), Ministry of Natural Resource and Environment Conservation (MoNREC) M KII 8-May Nay Pyi Taw 159 MGE under MonREC M KII 8-May Nay Pyi Taw 160 MGE under MonREC M KII 8-May Nay Pyi Taw USAID.GOV USAID PSDA MID-TERM EVALUATION | 76 161 MGE under MonREC M KII 8-May Nay Pyi Taw 162 MGE under MonREC M KII 8-May Nay Pyi Taw 163 MGE under MonREC F KII 8-May Nay Pyi Taw 164 MGE under MonREC M KII 8-May Nay Pyi Taw 165 MGE under MonREC M KII 8-May Nay Pyi Taw 166 MGE under MonREC M KII 8-May Nay Pyi Taw 167 Myanmar Development Institute (MDI) F KII 8-May Nay Pyi Taw 168 Financial Regulatory Department under Ministry of Planning and Finance M KII 10-May Nay Pyi Taw 169 FRD M KII 10-May Nay Pyi Taw 170 FRD M KII 10-May Nay Pyi Taw 171 Myanmar Insurance (state owned enterprise), Ministry of Planning & Finance M KII 9-May Phone 77 | USAID PSDA MID-TERM EVALUATION USAID.GOV ANNEX VII: MONITORING AND EVALUATION ADDITIONAL FINDINGS AND RECOMMENDATIONS This section was included based on the request by USAID/Burma. The findings are based on ET desk review and detailed investigation into the IP M&E plan and system. Findings and practical recommendations are included. ● Measurements of desired policy change need to be clarified: PSDA currently has indicators that target improved and increased inputs to policy-making rather than indicators related to literal policy change. The activity goal indicators in the original contract are written in terms of “citizens’ input into economic policymaking increased” and more opportunities for “dialogue and “engagement,” rather than ‘reforms attained’. The measurement plan should therefore equally measure both inputs to policy and actual policy change. As this activity and other governance projects have experienced, it is easier to achieve policy input goals relative to policy change goals due to the often transitioning and changing national government environment. ● Individual grant objectives are not clearly aligned with overall activity goal: Over the first three years of the activity, a grant fund of $2.5 million (limited to a maximum amount of $100,000 per each one-year grant) was available for non-government organizations (NGOs). This key instrument was used to: 1) complement and leverage technical assistance with key government policy makers to promote economic reforms, 2) build the capacity of nongovernment groups to participate effectively in economic reform policy dialogue, and 3) provide NGO members with better services and understandings for them to take most advantage of the economic reforms. This following order above of interdependent intermediate objectives is important to be respected in the design and measurement towards the IRs and overall objectives, in order to measure how grants contribute to broad-based economic opportunities. For example, policy change cannot materialize without policy dialogue and actual reform. It is acknowledged however that these processes take time, are dependent on external factors in an already slow-moving and opaque policy environment. In addition, the ET recommends that targets be revisited considering revised assumptions about local organizations’ networks and reach, based on progress mid-way through implementation. ● The Theory of Change does not incorporate all current aspects of the Activity: The Theory of Change does not contain an Intermediate Result for grants that directly links to the overall objective (increased broad-based economic opportunities). Currently only two components have Intermediate Results. EQ 3 asks to what extent the grants contributed to broad￾based economic opportunities, however few of the grants were designed to do so. It is acknowledged, however, that this is, in essence, a policy reform program. At the same time, the Access two Finance (A2F) component does not go beyond increased access to credit, i.e. what it is being used for21, e.g. for setting-up or expanding businesses (economic opportunities), although this is beyond the scope of the program. Nonetheless, these were the only types of grants that at least indirectly led to broad-based economic opportunities. ● Cross cutting indicators provide accountability but foster little learning: While indicators regarding number of people trained can help an activity remain accountable to planned activities, these indicators do not allow for learning about the impact of activities. For example, did people apply material from the training in their day-to-day work? How? When? The IP should consider adding measures of training and technical assistance value (e.g. on use and application of 21 Generally, it is difficult to get reliable data in some areas (e.g. for getting ‘access to finance’ related data; they must rely on banks and MFIs to provide that data). Similarly, it is also hard to get reliable/consistent reporting from MFIs which do not have direct relationships with Nathan (e.g. the relationship with some MFIs through GRET; GRET training to MFIs). USAID.GOV USAID PSDA MID-TERM EVALUATION | 78 knowledge gained, on-the job, as well as towards specific relevant Activity goals and objectives)22 so that they can better assess the Activity’s impact. ● The overall evaluation design for PSDA allows for statements of contribution but not attribution: There are many donors in Myanmar working in the same sectors. They are often working in siloes, not coordinating and collaborating together. This makes it difficult for any one donor to take sole credit for successful economic policy reform without utilization of an impact evaluation approach to program measurement (with a clearly defined counterfactual). While it is no longer feasible to set up an impact evaluation design for PSDA to allow for better estimations of activity impact, USAID and the IP can consider conducting a landscape analysis for geographic areas and issue areas within the scope of PSDA. This could be done by following guidance available for landscape analyses of similar Policy Reform initiatives (including mapping out who does what, what works well, why and based on that overall direction of relevant stakeholders, subsequent program design and operating principles for the future are decided). 23 This will allow USAID and the IP to speak more confidently about what other initiatives or donors may have influenced measured/achieved results, as well as to (continue to) direct focus in a complementary and coordinated fashion (e.g. on the financial sector reform), following the findings, conclusions, and recommendations from this report. 22 The Kirkpatrick model provides simple guidance to developing and evaluating basic training effectiveness indicators, from learning the right knowledge (relevance of training), to generating the required behavior, to contribution to and measurement of the improved performance accordingly. https://www.skillbuilderlms.com/wp-content/uploads/2017/08/Measuring-Training-Effectiveness-and-ROI￾eBook.pdf?mkt_tok=eyJpIjoiTXpaa09UVTRaV0poT0RZMyIsInQiOiJyN0lhdTVEOEdoMWE4TFwvUTJkUXhVNXB mK0ZOY2p6ZlFFNUtiOURqWVdTWjFyMFlJbmQzZVpQeDh5Z2VxNERZNzY2dE1wWnB6cG5zSzN3ZGs5MkFL ck9BTWxcL05hOURyb3V5U1QwMEQ4RVJKUExTcGxqVERRdHVGSFFCS3dRYXBsIn0%3D. Regarding the evaluation of training effectiveness, another example is the following more detailed/elaborated reference – Conducting evaluation to determine training effectiveness: http://www.iaea.org/inis/collection/NCLCollectionStore/_Public/34/062/34062611.pdf. 23 The ET refers to USAID’s ‘Landscape Analysis of Learning’ that contains some relevant and interesting guidance (e.g. to consult and coordinate around policy initiatives and strategic decisions, as well as dissemination of results, usage and application). https://usaidlearninglab.org/sites/default/files/resource/files/landscape_analysis_report_04_10_17.pdf. 79 | USAID PSDA MID-TERM EVALUATION USAID.GOV ANNEX VIII: SUMMARY OF FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS EVALUATION TOPICS MOST EFFECTIVE APPROACH ISSUE AREA WITH MOST RESULTS CONCLUSIONS RECOMMENDATIONS Economic Policy Reform (EQ1) Government Champion Trade, Food safety, IPR The activity is working on many areas (broad) on a surface level (with the exception of A2F) Focus on fewer Issue Areas and interventions with the goal of having a deeper impact on the population. Embedded Advisors A2F, Trade, MDI Capacity Building (EQ2) Trainings MF sector, Food safety, Trade, IPR, Role of media 1) PSDA has been effective in increasing capacity/knowled ge and less in reforming policy (which it wasn’t always intended to do). 2) The majority of PSDA funded CSOs are not self￾sustainable without donor support (despite their relevant mandate). 1) Continue to develop strategic relationships with committed champions of economic reform (individuals) in targeted ministries where important positive policy change is most likely to occur. 2) Support GoM or organizations which have support from GoM through interventions tailored to their specific needs (e.g. IT support, specific trainings). 3) Continue policy reform efforts with high likelihood of success e.g. Trade Liberalization, (ASEAN/WTO), Food Safety (Food and Drug Administration (FDA), IPR (MoE), Grants (EQ3) Grants to MFIs (EdM, GRET Delta) and associations (MMFA) MF sector / mobilize credit The majority of grants achieved general objectives (capacity building) but did not (intend to) impact broad-based economic opportunities. Continue effective grants (to take most advantage of the economic reforms) Access to Finance (EQ4) MF sector policy advocacy MF sector / mobilize credit A2F work has vastly outperformed other policy reform efforts in terms of achieving intended results and was the only work that contributed to broad based economic opportunities Focus on proven success: - A2F MFIs: increase individual lending, - Banks: SME lending, DCA - Ag) Capacity building at MMFA (IT support) MF sector / mobilize credit DCA Agriculture, A2F USAID.GOV USAID PSDA MID-TERM EVALUATION | 80 ANNEX IX: ADDITIONAL GENDER FINDINGS AND CONCLUSIONS FINDINGS While the ET has noted EQ-relevant gender notes and findings in the main report, this annex presents a more complete picture of PSDA’s progress in this topic. Throughout the activity history, PSDA has shifted gender into different Activity topics and focus areas. The topic of gender and ethnic minorities were first jointly placed under PSDA’s Entrepreneurship work. From the PSDA Annual Report Year 2 (FY2017) Issue Area 11, it states: “Enable Entrepreneurial Development, including for Women-Owned and Ethnic￾Owned Business.” By the 2nd Quarterly Report (FY 2018), the topic was addressed as “Support to Gender and Ethnic Minorities in Economic Development.” Gender was then removed from the topic of entrepreneurship and became a separate and distinct Intermediate Result (IR) category called “Support Entrepreneurial Drive and Policy Dialogue” in the PSDA Year 3 (FY 2018) Workplan and was re-categorized in the 2nd Quarterly Report (FY 2018) as “Building Capacity for Effective Economic Policy Dialogue and Entrepreneurism”. The ET learned through KIIs and activity documents at the mid-term that PSDA considered the “Support of Gender and Ethnic Minorities in Economic Development” as one cross-cutting issue. According to the PSDA Annual Workplan FY 2018, while the Activity purses a holistic approach where inclusion of women is an underlying objective of all tasks, the Activity seeks specific opportunities to implement initiatives where gender and disadvantaged groups are beneficiaries. Quantitative: There is one relevant indicator for cross-cutting issues, namely Indicator 4.2 (Support Gender and Ethnic Minorities in Economic Development) relates to the number of women’s associations participating in PSDA activities (see Table 13: Cross-Cutting Considerations Indicators3). At the mid￾term, the Activity has substantially over-achieved the target. The majority of the organizations are related to microfinance work, for example the National YWCA implemented trainings for 223 women borrower groups in 2017. In 2018, a large number of trainings were conducted by grantees in the area of food safety and intellectual property. Table 13: Cross-Cutting Considerations Indicators RESULT INDICATORS PROGRESS TO DATE (MID-TERM) Intended Result 4.2: # women’s associations participating in activities -Target MET (achievement is significantly higher than targeted); 81 (targeted) vs. 557 (achieved) Qualitative: While gender was not a central strategic focus of PSDA and considered one of many (sixteen) issue areas, women did participate equally in the Activity and benefit from the Activity; and PSDA kept gender disaggregated data. The ET has organized relevant findings on gender below by EQ. Evaluation Question 1: Economic Policy Reform According to the Contract between USAID and Nathan, USG prioritizes the inclusion of women and marginalized populations as part of an inclusive reform process in Myanmar. As such, the IP explored the challenges and opportunities facing women and rural ethnic populations, among other historically 81 | USAID PSDA MID-TERM EVALUATION USAID.GOV disadvantaged populations, in all assessments, baselines, and activity approaches (particularly in trainings and in the grants component). In activity assessments of the gender context in Myanmar, the IP notes that the most powerful person in Myanmar is a woman – State Counsellor Aung San Suu Kyi. Furthermore, women occupy many high-level positions in Myanmar government offices and commercial enterprises. Though presence at these levels does not automatically correlate with influence, the influence that PSDA’s women champions have had gives some insight into this important aspect of women’s equal influence in GoM. Some of the central champions PSDA supports are women. Evaluation Question 2: Capacity Building The ET learned through KIIs and activity documents that PSDA used their partnership with Internews to provide women and minorities with information. The PSDA Quarterly Report2 (FY 2018) states: “One way the Activity supports gender and ethnic minorities in economic development is to help give them a voice and better access to information. Various technical support activities by subcontractor Internews, which include the strengthening of individual ethnic/regional journalists and tailored in-house training and mentoring for ethnic and regional media, have led to an increased supply of information on economic issues in regions and ethnic minority states.” Both trainers and trainees confirmed equal participation of men and women in all outlined trainings and capacity building activities, with many exceeding targets for women’s involvement (due to the fact that many staff in recipient institutions are female, including at DDG and DG level). It was furthermore noted, however, based on desk review and interviews with IP staff, that engaging women in technical trainings was more challenging and that PSDA aims to ensure a more equal participation in IT and high-tech sector trainings, where possible. As a specific example of engagement of women in capacity building work, PSDA conducting a trade facilitation workshop where a trade portal was created. This was created and presented by the Deputy Director General to senior generals with much success. Evaluation Question 3: Grants PSDA seeks specific opportunities to implement initiatives where gender and disadvantaged groups are beneficiaries through grant awards and efforts to expand banking and MFI lending to women and ethnic￾owned SMEs. In the case of Pyu Sin Saydana Action Group (PSSAG) and Internews, gender has been considered from the assessment and disbursement through to the measurement of participation and gender-disaggregated results particularly concerning women’s’ participation in the peace process. Over 50 percent participation of women was the target in the following grants: YWCA, MES, MFIs, DEAR Myanmar, and DRI. Grants were also found to be accessible to men and women equally. PSDA also developed and helped Myanmar Young Entrepreneurs Association to implement a survey of entrepreneurs covering almost 300 enterprises. Almost 50 percent of survey respondents were women. Additionally, PSDA extended grants to the Young Woman’s Christian Association (YWCA) and Opportunities NOW to train women in business skills as well as grants to MFIs with women in high leadership positions to include MDP and PSSAG. Evaluation Question 4: Access to Finance In total, PSDA supported ten MFIs that extend microfinance loans almost exclusively to women through a comprehensive combination of capacity building/training, grants, and DCA credit guarantees. The USAID.GOV USAID PSDA MID-TERM EVALUATION | 82 majority of respondents confirmed adequate participation of female borrowers in trainings. PSDA has kept records of the number and percentages of women accessing finance from supported MFIs and banks. CONCLUSIONS The topic of gender throughout the Activity period of performance thus far has been treated and viewed as an ‘add on’ and has been moved and shifted throughout the activity framework. This evolvement has resulted in gender becoming a more defined activity goal (and issue area) of economic policy reform, capacity building for policy dialogue, and increased access to finance. While gender is not a central strategic focus of the activity and is considered one of many (sixteen) issue areas at present, benefits were certainly accrued to women as a result of project activities and gender disaggregated data was kept. The most noteworthy success was achieved in microfinance where the vast majority of borrowers are women. At the mid-term, PSDA achieved the single indicator related to ‘supporting gender in economic development’ by a wide margin.