Final Report Performance Evaluation of the USAID/Mozambique Support Program for Economic and Enterprise Development (SPEED) Credit: Romualdo Uaila Final Report 1 Performance Evaluation of the USAID/Mozambique Support Program for Economic and Enterprise Development (SPEED) Order Number: AID-656-O-14-00001 Requisition/Reference Number: REQ-656-13-000024 Submitted to: Ms. Shannon Howard Contracting Officer’s Representative (COR) Prepared by: Jeffrey Nash, Team Leader Jeffrey Dorsey, Technical Writer Contractor: FLAG International LLC 315 Severn Ave Annapolis, MD 21403 USA www.flaginternational.com DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. 2 Table of Contents ACRONYMS EXECUTIVE SUMMARY …………………………………………………………………6 1.0 EVALUATION PURPOSE & EVALUATION QUESTIONS ……………………..10 2.0 SPEED Activity BACKGROUND .…………………………………………………..12 3.0 EVALUATION METHODS & LIMITATIONS …………………………………....13 4.0 FINDINGS & RECOMMENDATIONS…………………….………………………..20 4.1 Qualitative …………………………………………………..……………………...18 a. Effectiveness ………………………………………………….………………...20 b. Stakeholder Collaboration…………………………………………….…………23 c. Sustainability …………………………………………………….……………...26 d. Gender ………………………………………………………………………......30 4.2 Quantitative………………………………………………………………..………..37 a. Initial Activity Design……………….…………………………………………...37 b. Management of SITT Tracking Tools……………………..……………………..38 c. Outcomes ………………………………………………………..…………….....42 4.3 Literature Review (excerpt from full report).………..…………………………..…44 5.0 ADDITIONAL RECOMMENDATIONS……………………………………………..44 ANNEXES …………………………………………………………………………………..54 Annex 1: Qualitative and Quantitative Data Collection Design Matrices…………………...55 Annex 2: Evaluation Participants…………………………………………………………….60 Annex 3: Key Informant Participants………………………………………………………...61 Annex 4: Focus Group Participants…………………………………………………………..62 Annex 5: Workshop Participants……………………………………………………………..63 Annex 6: Summary of SPEED Capacity-Building Activities………………………………..64 Annex 7: Mozambican Macroeconomic Indicators 2010 – 2014……………………………65 Annex 8: Foreign Direct Investment in Mozambique by Sector……………………………..66 Annex 9: PMP Summary Table Quarter Three Fiscal Year 2013……………………………67 Annex 10: Indicator Status…………………………………………………………………...68 Annex 11: SPEED Policy Stage Achieved Per Year………………………………………...70 Annex 12: List of Institutions by Sector, Type, and Form of Assistance……………………71 Annex 13: Questionnaires……………………………………………………………………74 Annex 14: Main Database……………………………………………………………………75 Annex 15: Organizations Participating in Data Collection…………………………………..77 3 Annex 16: Lessons Learned from Other Countries…………………………………………..78 Annex 17: Literature Review Tables 1, 2A, and 2B…………………………………………80 Tables Figure 1: Parallel Analytical Method………………………………………………………...13 Figure 2: Sectoral Employment Disaggregated by Sex……………………………………….35 4 Acronyms AR Parliament ATB Agriculture, Trade and Business BM Banco de Moçambique, (central bank) COR Contacting Officer´s Representative CSO Civil Society Organization CTA Confederation of Business Associations FEMME Women´s Business Association FEMOTUR Tourist Federation FLAG FLAG International LLC GoM Government of Mozambique IPEX Export Promotion Institute M&E Monitoring and Evaluation MF Ministry of Finance MIC Ministry of Industry and Commerce MINAG Ministry of Agriculture MITUR Ministry of Tourism MRM Ministry of Mineral Resources NGO Non-Governmental Organization PGR Attorney General´s Office PMP Performance Management Plan PSWG Private Sector Working Group RFQ Request for Quotation SITT SPEED Indicator Tracking Table SOW Scope of Work UEM Eduardo Mondlane University USAID United States Agency for International Development WB World Bank 5 EXECUTIVE SUMMARY The US Agency for International Development (USAID) Mission in Mozambique contracted a third-party firm to conduct a performance evaluation of its Support Program for Economic and Enterprise Development (SPEED). SPEED implementation began on August 30, 2010 and will end on September 30, 2014, with the possibility of several months’ extension. The program is implemented by Development Alternatives International (DAI) as the prime contractor, together with sub-contractors Nathan Associates, The QED Group, Cimpogest, and SAL & Caldeira. The total life-of￾activity funding is $19.5 million USD. SPEED is the third generation of USAID/Mozambique’s trade and investment activities focused on the business enabling environment. It works primarily to influence governmental policy advocacy, reform, and implementation. SPEED has a national scope and is designed to increase export diversification, support job creation, and contribute significantly to income generation. The purpose of the performance evaluation was four-fold. First, the evaluation team was asked to determine whether the current approaches and strategies are working well or not. Second, they were to capture important information on lessons learned and best practices from the implementation of SPEED activities. Third, they were to offer recommendations on any necessary, immediate modifications that would re-focus and strengthen the activity for the remaining life of SPEED. Fourth, they were to offer recommendations and findings that would inform the design of future USAID / Mozambique policy advocacy support activities. Methodology In order to carry out the evaluation, qualitative and quantitative matrices that were developed by USAID to outline evaluation questions, sub-questions, indicators, data source collection methods, sampling criteria, and analysis methods were used to guide the evaluation team. The qualitative matrix featured four key questions on effectiveness, stakeholder collaboration, sustainability, and gender. The quantitative matrix focused on activity design, management, outcomes, and a literature review. The evaluation team used a mixed / parallel analytical method to synthesize and triangulate qualitative and quantitative data to form conclusions. The evaluation was carried out by a three-person expatriate and two-person local team over an eight week period, with field work taking place in Mozambique for five weeks. Qualitative data was collected primarily through key informant interviews, focus groups and a workshop, reaching 49 stakeholders. Stakeholders were clustered into informant types (direct beneficiaries, non-beneficiaries, Partners, USAID, SPEED) and Sector Focus Areas (Government of Mozambique, United States Government, International Government (non￾US), Private Sector, Non-profit, and Academia). 6 Effectiveness Findings Findings under the question of effectiveness revealed that SPEED is building local capacity in policy dialogue and changes. It is balancing appropriately between a demand-driven approach and other task order requirements. It has supported Mozambique on a macroeconomic level by facilitating policy reforms in the areas of foreign exchange regulation, exchange rate policy, a sovereign wealth fund policy, and a mining and petroleum tax law. In regard to determining the allocation of initiative activity by sector, the evaluation team had difficulty interpreting the data based on the documents it received. According to the indicators tracking table for FY 2013 Q3, 51 out of 59 policy, regulation and administrative procedure initiatives addressed were related to agriculture, food security, and climate change. However, according to the indicators tracking table for FY 2013 Q4, only 19 of 57 SPEED initiatives addressed agriculture, food security and climate change (Annex 17). An explanation for the significant difference in sector designations by initiative was not found. Stakeholder Collaboration Findings Findings regarding stakeholder collaboration found that SPEED has successfully engaged its stakeholders, including key ministries and government offices. Stakeholders reported a positive experience working with SPEED. Non-beneficiaries interviewed, however, reported difficulty figuring out how to access SPEED if they were not first contacted or introduced by a third party. Sustainability Findings Questions to determine the sustainability of SPEED’s capacity building activities with the public and private sector resulted in feedback that suggested SPEED’s beneficiaries felt the services they received from SPEED will result in sustainable change. The evaluation team did not, however, find conclusive evidence of long-term sustainability deriving from capacity building within the scope of its evaluation. When analyzing whether policy reforms and policy processes supported by SPEED will create long-lasting economic benefits for Mozambique, sources showed that it will. Gender Findings Findings on gender presented the greatest opportunity for SPEED to expand its policy reform efforts. SPEED did not refrain from focusing on gender but under its demand-driven model it received limited gender-specific requests from stakeholders. The evaluation team found gender-related policy issues to be a source of confusion for the evaluation participants. Quantitative Findings Inconsistencies were detected among SPEED’s tracking tools. Consistent inclusion of key fields for categorizing or breaking down activity data for all activities and initiatives would make analysis easier for activity managers in USAID and for external actors (evaluators, members of Congress, etc.) to more easily relate activities to one area of interest or another. Further, all activities and initiatives should be organized in one database in order to enable comprehensive and accurate monitoring and analysis of all data. In review of indicators, the evaluation team found that USAID and SPEED’s decision to eliminate the number of jobs created indicator is warranted, especially considering the variances in how this indicator can be measured under a policy program. The World Bank “Doing Business” indicator, which USAID and SPEED do plan to keep, is currently missing from the tracking tools. 7 Literature Review Findings The literature review was meant to evaluate the quality of SPEED’s publications posted on their website and how relevant and timely they are to the stakeholders’ needs and interests, to include assessing them within the context of the four key questions. Each publication was carried out with a specific purpose linked to a SPEED initiative, and this was found by the evaluation team to be a very valuable tool for SPEED to engage with their stakeholders. The publications posted on the SPEED website were found to be of high quality and overall feedback from stakeholders who regularly access the publications was that the publications were of value to them and their work. A full literature review report was submitted separately to USAID. Key Recommendations Under the qualitative data section, the evaluation team recommends that SPEED’s annual and quarterly reports delineate between demand driven and task order / results based activities to better enable the SPEED team, USAID, and other stakeholders to better measure the balance between the two. This would help both in terms of documenting lessons learned as well as in preparing upcoming work plans. In addition, it is recommended that a broad-based survey be administered by the follow-on activity to ask beneficiaries what they’ve achieved in terms of successful trade agreements, business deals, or new business models as a result of their support from SPEED. This would enable the follow-on activity to better assess the impact of capacity building activities and adjust accordingly during the life of the new activity. It is recommended that SPEED and its follow-on program increase their ability to work through a ‘gender-lens’ when carrying out their policy work. The first step is to understand gender implications within the policy arena and educate stakeholders. Next would be to take a more inclusive approach to try to engage more with Ministerial Gender Units, the Donor Gender Working Group, and women-focused business associations. Finally, clearly identifying sectors that women have a significant role, yet clearly are under-represented, such as agriculture, trade and finance, and the service industry. Under the quantitative data section, a key recommendation includes re-incorporating the World Bank’s “Doing Business” indicator back into the Performance Management Plan (PMP) and other tracking tools. SPEED is also encouraged to add footnotes to all tracking tools that define how the indicators are measured or calculated similar to how the five policy stages are outlined for easy reader interpretation. In addition, there are a number of ways that the tracking tools can be made more consistent and better linked, specifics of which can be found in the “Management of SITT Tracking Tools” section below. The evaluation team also cautions that not all outcomes of policy changes can be measured quantitatively, even if the results derived from them are recognizably positive, therefore the addition of new quantitative indicators is not recommended. Under the literature review, SPEED publications were found to be high quality and as a result, there are few recommendations for improvement in this area. The evaluation team recommends SPEED continue to employ a communications expert, increase the alert distribution list to include more stakeholders, plan for how the site will be managed during 8 the potential gap between SPEED and a follow-on activity, and actively publish reports speaking to gender in the policy arena and private sector. Regarding the use of social media, SPEED should take advantage of potential synergies between the publication site and its social media networks to increase traffic on its site and the use of its publications. The evaluation team strongly supports an extension of the SPEED program until February 2015. Any gap period between SPEED and a follow-on activity should be minimized and alternative mechanisms to continue activities during the gap should be explored. It is recommended that the follow-on activity be demand-driven and extended to the longest duration possible. Finally, it is recommended that the follow-on activity take into account a need for support to the business enabling environment at the sub-national level and also strive to take advantage of the presidential inauguration period, post-October 2014 national elections, to raise the profile of SPEED-supported policy reforms. 9 1.0 EVALUATION PURPOSE AND QUESTIONS This evaluation is being conducted towards the end of the third year of activity implementation. Because USAID plans to continue to support policy advocacy work in the future, the main purpose of this evaluation is the following: 1. To determine whether the current approaches and strategies are working well or not, and why this is. 2. To capture important information on lessons learned and best practices from experience implementing SPEED activities. 3. To recommend any necessary immediate modifications that would re-focus and strengthen the activity during the remaining life of SPEED. 4. To provide findings and recommendations that would inform the design of future USAID/Mozambique policy advocacy support activities. The primary audience for this performance evaluation is the Agriculture, Trade, and Business (ATB) Office of USAID/Mozambique. Secondary audiences include the implementing partners and their stakeholders, the GoM, relevant donor groups, the private sector, and other stakeholders. There may be some areas where some evaluation findings will be shared with these and other stakeholders for discussion, as determined by USAID. The evaluation was built off of the SPEED Evaluation Scope of Work (SOW). The data collection design matrices were built for both qualitative and quantitative data collection (Annex I). The qualitative design matrix was based on the four USAID key question areas of 1. Effectiveness, 2. Stakeholder Collaboration, 3. Sustainability, and 4. Gender. The questions under the qualitative design matrix are as follows: 1. Effectiveness How and to what extent has SPEEDs programming been effective in the following areas and what were the contributing factors? a. Building Local capacity in policy dialog and policy implementation. b. Balancing between demand driven approach (external USAID stakeholders) and other task order requirements (internal USAID stakeholders). c. How can SPEED indicators be improved? d. How effective has SPEED been at the Macroeconomic level and sector specific work? 2. Stakeholder Collaboration How and to what extent has SPEED engaged, collaborated with, and leveraged stakeholders? a. How do stakeholders view SPEEDS work? b. Is SPEED engaging with appropriate key Government stakeholders? 3. Sustainability What sustainable practices has SPEED implemented? a. How Sustainable is SPEED’s work on Strengthening institutional capacity of private and government institutions, especially CTA? 10 b. To what extent will the policy reforms and processes create long lasting economic benefits? 4. Gender How has SPEED integrated Gender Issues with its work efforts? a. How has SPEED addressed the different Gender roles and needs when selecting policy areas, conducting policy analysis or advocating policy change? And how could any changes or improvements be made in this area? b. Are there particular policy areas that have to date or could in the future especially benefit women traders or entrepreneurs? All illustrative indicators were built around the evaluation. Specific performance indicators were reviewed relevant to SPEED reports and other tracking documents as identified during the Inception Plan review. The quantitative design matrix is based on questions associated with the design, management of tracking tools, and outcomes of the SPEED activity. Additionally, within the quantitative design matrix, USAID requested a literature review that will “evaluate the quality of SPEED’s publications and how relevant and timely they are to the stakeholders’ needs and interests,” to include assessing them within the context of the four key questions. For this deliverable, USAID requested the following publications be reviewed: “Documents to be reviewed are Notes, Reports, Presentations (see http://www.speed￾program.com/ourwork/all-publications), and blog postings (see http://www.speed￾program.com/blogs). There are a total of at least 25 Notes, 36 Reports, 20 Presentations, one Web posting (there is some duplication in this total count; this is inclusive of both English and Portuguese versions), and 20 total blog postings. Because of limited resources, we request that a sampling of half of the publications, from each category, be evaluated for the Literature Review.” 11 2.0 SPEED ACTIVITY BACKGROUND SPEED implementation began on August 30, 2010 and will end on September 30, 2014 with the possibility of an extension. It is implemented by Development Alternatives International (DAI) as the prime contractor, together with Nathan Associates, the QED Group, Cimpogest and SAL & Caldeira. The total life-of-activity funding is approximately US$19.5 million. SPEED contributes to USAID/Mozambique’s Assistance Objective (AO), “Inclusive growth of targeted economic sectors,” illustrated on the ATB Results Framework. SPEED supports ATB’s Intermediate Result 2 (IR2), “Enabling environment improved.” This AO and IR2, and the concept behind the SPEED activity, is founded on the hypothesis that an improved enabling business environment is a necessary condition that, combined with increased agricultural productivity and sound management of natural resources, will lead to broad￾based economic growth. An effective legal and institutional framework that reduces costs of doing business and promotes transparency and fairness will attract investment, which will lead to the creation of jobs and an increase in the volume and value of exports. It also will enhance the productivity and competitiveness of agriculture, tourism, and related small and medium enterprises (SMEs). SPEED supports the creation of a private-sector friendly enabling business environment that leads to inclusive economic growth. The rationale of the activity is that through an improved business climate, Mozambican markets will be able to attract investments, increase exports and also create jobs. The activity’s success depends on two outcomes: 1. Adoption and Implementation of Reforms: Mozambique will successfully adopt and implement an array of business environment reforms that enhance trade and investment generally, but specifically in tourism (service sector), agriculture, and agribusiness (manufacturing). 2. Improved Capacity: Private sector associations and government institutions are able to effectively promote and implement policies that foster trade and investment. 12 3.0 EVALUATION METHODS AND LIMITATIONS The FLAG consultancy team began preliminary remote work on the evaluation just after the signing of the contract on December 2, 2013 and the team was mobilized in Maputo, Mozambique on January 13, 2014. Survey instruments were developed and submitted to USAID for review. Testing of the survey instruments took place from January 23 – 24, 2014. Data collection efforts then began on January 24, 2014 and continued until February 7, 2014. The majority of data collection took place in Maputo. One trip to Beira took place February 3 – 5, 2014. The evaluation team members are listed below: ● Jeffrey Nash, Team Leader ● Jeff Dorsey, Lead Technical Writer ● Maria Witz, Qualitative M&E Specialist ● Romualdo Uaila, Quantitative and Qualitative M&E Specialist ● Nélio Cândido, Interpreter and Survey Specialist ● Sergio Macuacua, USAID/Mozambique ATB M&E Specialist, participated in the Beira data collection. The evaluation team employed a mixed method approach that utilized a parallel analytic process. This was determined due to the fact that USAID was seeking to collect primary and secondary data for different questions and from different sources which required more than one method to collect the information. By comparing both qualitative and quantitative data sets, it improved the results of the evaluation and allowed for the identification of unanticipated results. The evaluation team followed the qualitative and quantitative matrices in figure one below to carry out the analysis: Figure 1. Parallel Analytical Method Both a Qualitative and Quantitative Design Matrix was given to the evaluation team per the RFQ by USAID (Annex 1), which clearly identified which data would fit under which general heading. The evaluation team used these matrices as the foundation for carrying out the performance evaluation. Although the Design Matrix showed the intent for the type of Quantitative Data Analysis Quantitative Data Collection Quantitative Data Collection Synthesis Triangulation of Data Qualitative Data Analysis Qualitative Data Collection 13 analysis that would be performed on specific data sources, adjustments were made with regard to the type of analysis performed on data when findings presented opportunities to optimize or expand upon analysis to the benefit of the overall evaluation. Qualitative data was collected primarily through key informant (KI) interviews, focus groups, and a workshop, reaching 49 stakeholders (Annex 2) and saved into a database maintained by the evaluation team (Annex 14). There was an interpreter present at each type of data collecting event and the participant(s) were given the opportunity to use English and/or Portuguese for communication. Stakeholders were clustered into informant types and sector focus areas. The informant types included: • Direct Beneficiary (22 interviewed): Individuals/organizations receiving direct support, either financial or technical, from SPEED. • Partner (9 interviewed): Individuals/organizations working in parallel with SPEED, but neither receiving nor providing direct assistance. • Non-beneficiary (10 interviewed): Individuals/organizations not receiving formal support from SPEED and may or may not have been familiar with the program. • USAID sources (5 interviewed): USAID offices directly involved with SPEED. • SPEED sources (3 interviewed): Direct employees of the SPEED implementation team. It was important to balance the input of those that have worked directly with SPEED with those that had not, in order to capture data on how well SPEED is known and of what value such a program is to those engaged with SPEED, or could be to those that may not yet have had the opportunity to work directly with SPEED. The evaluation team therefore interviewed both types of informants to enable any comparisons to be made. The evaluation team feels a representative and balanced sampling of informant types was captured. A total of 49 Stakeholders were interviewed, also clustered by sector area of work: • Government of Mozambique (GoM) (19 interviewed) • International Government (Gov. Intl) (3 interviewed) • United States Government (USG) (8 interviewed) • Private Sector (13 interviewed) • Non-profit (5 interviewed) • Academia (1 interviewed) It should be noted that the evaluation team broke down the public sector into the various forms of government (GoM, USG, and International Government), since each would have a unique experience and perspective on SPEED. It was also necessary to break down CSOs into private sector, non-profit, and academia categories for the same reason. The evaluation team feels a representative and balanced sampling stakeholders by sector was captured. A full list of the organizations that participated in the data collection effort can be found in Annex 15. 14 Many interviewees agreed to meet with the evaluation team on the condition that their names and, in some cases, their position titles and even organization names remain anonymous when citing their responses. This was discussed with USAID, who agreed with this approach. The type of survey instrument used per stakeholder was dependent on their informant type and sector. As noted, the intent was to have a balanced sample of informant types and sectors. The evaluation team set as criteria of one survey instrument per stakeholder, but for some organizations, there was more than one department that was relevant to SPEED; hence more than one representative from same organization may have been involved in the data collection. This was also in line with USAID’s recommendation for stakeholders. Another criterion for stakeholder selection was to have a balanced mix of stakeholders that knew or worked with SPEED at differing times of their project life cycle, in order to assess the evolution through comparing perception of stakeholders on distinct stages of SPEED. The evaluation team succeeded at interviewing over 90% of the stakeholders from USAID’s original stakeholder list. Key Informant (KI) Interviews A total of 30 KI interviews were carried out among 23 male and 7 female participants that represented a variety of informant types and sectors (Annex 3). The survey tool used for KI interviews was in the form of a questionnaire. A questionnaire was developed for each informant type (Annex 13), since it was recognized that relevancy of questions would vary across these clusters. KI interviews were carried out mostly at the stakeholder’s office or nearby. Each interview took 45 minutes to one hour. Two to four members of the evaluation team were present at each KI interview. It was guaranteed that at least one Portuguese speaker be present at every interview. This allowed the interviewee to choose either English or Portuguese for the interview. The majority of the KI interviews were carried out in English, but approximately seven were interpreted between English and Portuguese. All data collected during these interviews was entered into a master database. Focus Groups The second data collection tool utilized was that of focus groups (FGs). A mix of informant types were invited to the FGs, but USAID and SPEED direct staff were not invited so that the participants would feel more at ease to express their views in support of the evaluation. These interviews were with more than one stakeholder and were on more targeted topics than the KI interviews. The FGs provided a more in-depth understanding of a sub-topic within the overall evaluation. The questionnaires utilized for the KI interviews were not used for the FGs, rather a more open discussion was encouraged. The facilitator encouraged discussion on certain topic points but allowed for an open dialogue. Three FGs were carried out during the data collection period; two in Maputo and one in Beira. Each was held at a café in an informal setting to encourage open discussion on the focus topics. At each FG, the participants were 15 given the opportunity to choose to use English or Portuguese for main mode of communication. The titles, locations, and descriptions of each FG are listed below: 1. Gender, Business, Policy and SPEED. FG took place in Maputo (five confirmed attendance; two participated). Conducted mostly in English, with a few parts interpreted into Portuguese. This focus group was meant to address key evaluation question #4: Gender. In order to understand the overall reality of women’s position within the business and policy arena of Mozambique as compared to men’s, the first group of questions was meant to capture this data. • Level of women compared to men working in decision-making levels within the business and policy arenas of Mozambique. • Perceptions of women vs. men working in the business and policy arenas in both urban and rural areas of Mozambique. • Level of focus on gender appropriate language within policy papers. • Challenges of both women and men in the business and policy arenas of Mozambique. The second grouping of questions was meant to measure SPEED’s level of focus on issues of gender within their work. • Did any of your work with SPEED address issues of gender? If yes explain. • If not, do you see ways that SPEED or a SPEED-like program could address issues of gender? 2. Private Sector, Policy, and the Resource Boom. Took place in Maputo (six confirmed attendance; two participated). Conducted in Portuguese. In light of the fact that the “resource boom” did not come into the spotlight until 2012, when SPEED had already been working for two years, their major focus areas have been less on this specific topic. Since the resource boom is such a critical issue for the country in both business and policy, it is critical it is addressed by SPEED or a future SPEED-like activity. The evaluation team felt measuring the level of focus SPEED has had on the resource boom would be a measurement of key evaluation question one focused on effectiveness. If it could be shown that they have been able to keep up with the changing environment and needs of the country in their work, then this will speak to effectiveness. Since this is a fairly new topic, it was felt that more valuable data could be collected in a forum such as a focus group. The first set of questions addressed the overall reality of the Resource Boom’s effects on the private sector and on policy. • What have been the strongest sectors for the economic growth of Mozambique? • Has policy supported or limited growth in these areas? 16 • What effect will the Resource Boom have on the business and policy environments of Mozambique? The second grouping of questions more closely looked at how SPEED has addressed the resource boom, as well as what future actions could be taken to better address the resource boom. • What has been SPEED’s impact in this area? • Is there opportunity for an increased focus? 3. Private Sector, Policy and SPEED. Took place in Beira (six confirmed attendance; four participated). Conducted in Portuguese with use of interpreter. The evaluation team felt that the topics of gender and of the resource boom deserved extra attention, therefore they were the topics for the two focus groups in Maputo. In Beira, the evaluation team felt it was important to keep to a more general discussion, but target the unique experience of those at the sub-national level. Moreover, since three of the four in attendance were non-beneficiaries of SPEED and working directly in the private sector, the evaluation team felt it would be of value to keep the focus more general as to identify the challenges and successes each experienced working in the private sector of Beira that could help identify how SPEED may be of value in the future. Questions focused on • What have been the strongest sectors for the economic growth of Mozambique? • Has policy supported or limited growth in these areas? • What has been unique to Beira vs. Maputo in regard to the 2 above questions? • Do you see SPEED or a SPEED-like program being responsive to the needs of Beira? If yes, explain. If no, please suggest how such a program could be more responsive at the sub-national level. The number for all three FGs, disaggregated by informant type, sector and gender, interviewed are listed in Annex 4. Workshop As requested by USAID, the SPEED evaluation team organized one workshop as part of the evaluation process in Maputo, but not in Beira. Even though the original SOW requested a workshop in both Maputo and Beira, it was decided by both USAID and the evaluation team that there was not an adequate number of relevant stakeholders in Beira for a forum of this type to be of value. The workshop in Maputo hosted 11 participants (7 male / 4 female). The participants are broken down by informant and sector type (Annex 5). The workshop targeted more senior level stakeholders. The evaluation team did not invite USAID and SPEED direct staff so that the participants would feel more at ease to express their views. The workshop permitted an in-depth discussion of the evaluation questions and generated valuable inputs for recommendations on future directions of any USAID-supported policy advocacy activities. This was particularly useful as a forum for discussing important issues pertaining to the four evaluation questions that arose as a result of the desk review, key informant interviews, and focus group discussions. The workshop participants chose to speak in English even though two interpreters were provided and speaking in Portuguese as the main language was initially 17 suggested. Quantitative Analysis Per the RFQ, a Quantitative Design Matrix (Annex 1) was provided by USAID to the evaluation team that focused on the initial activity design, management and tracking of the SPEED indicator tracking table mechanisms, outcomes, and a literature review To conduct the quantitative review, the evaluation team analyzed the following sources: • Relevant GoM economic data sources • WB Doing Business ranking and Transparency International’s Corruption Index • Specific sector (GOM) annual reports and databases • SPEED internal/external documents • Documents relevant to design phase of SPEED • SPEED Indicator Tracking Tools • SPEED Results Framework • SPEED Website Where possible, the evaluation team came up with quantified outcomes. Targets were compared with observed outcomes. When targets were not set, the team used data to see how closely linked they were correlated with SPEED policy interventions. Since the activity is demand driven, not results driven, target outcome comparisons were mostly absent. Rigorous quantitative data analysis such as statistical software modeling was not appropriate for this evaluation because the data reviewed was not statistically significant. While SPEED’s capacity building activities reached thousands of individuals, as did the number of jobs estimated to be created as a result of SPEED’s efforts, it was not in the evaluation team’s mandate to investigate such large numbers of beneficiaries through large scale surveys. Limitations One challenge the evaluation team recognized in its discussions with SPEED and USAID are that the Regional Inspector General (RIG) recently carried out an in-depth audit on the SPEED activity and met with many of the same stakeholders the team met. The SPEED COP was concerned that this would not be the most opportune time for stakeholders to be approached, being asked as they were to provide information a second time within such a short time frame on similar topics. The concern was also that stakeholders may perceive all these evaluations as a signal that SPEED was being investigated for some wrong doing. The evaluation team made it a point to clarify to each interviewee that the team is not related to the RIG in any way and to explain the purpose of our performance evaluation and how it differs from the RIG audit. The timing for the evaluation was a challenge since it was peak holiday season for many stakeholders. Many of them were on holiday and unable to be reached. There were also two national holidays during the projected time of data collection, one being a last-minute Federal Mozambican holiday only announced 48 hours ahead of time. The team had interviews 18 scheduled for that day, but all had to be canceled once the holiday was declared. The team was also advised that generally speaking, Mondays and Fridays were not good days to schedule meetings. This also greatly limited the team’s ability to schedule meetings during the limited time frame of two weeks. Questionnaires were developed for each informant type, but as the data collection progressed, the evaluation team identified questions that were not as relevant and not of as much value as first thought. For three KI interviews, the decision was made not to work through the full questionnaire but rather to have a general broad-ranging discussion on SPEED. The cases where this was done was when 1) the interviewee had limited time and it would not have been appropriate for the team to try to work through the questionnaire, and 2) when the formal setting of the interview was not conducive to this format of an interview. 19 4.0 FINDINGS & RECOMMENDATIONS 4.1 Qualitative Data There are four main evaluation questions and sets of sub-questions that have been answered by this evaluation. They include a. Effectiveness, b. Stakeholder Collaboration, c. Sustainability, and d. Gender. a. Effectiveness The first key question the evaluation team sought to address was that of program effectiveness. In conducting qualitative analysis of the key informant interviews, focus groups, and workshop results, the team examined the following questions: 1. Is SPEED building local capacity in policy dialogue and policy implementation? 2. Is SPEED balancing between a demand-driven approach (external USAID stakeholders) and other task order requirements (internal USAID stakeholders)? 3. How effective has SPEED been at the macroeconomic level and sector specific work? Findings The findings related to these three questions are explored below: a.1. Is SPEED building local capacity in policy dialogue and policy implementation? It was found that SPEED has built local capacity in policy dialogue and implementation. Hundreds of beneficiaries have been trained to develop technical skills that enable them to participate more broadly on policy dialogues seeking to improve trade and investment. It was generally believed that SPEED helped to promote dialogue amongst stakeholders, disseminate information on legislation, and provided institutional support. SPEED’s outreach and capacity building activities reached 5,062 individuals in three years (Annex 6). All (ten of ten) direct beneficiaries interviewed with a standard questionnaire felt they received significant capacity building support from SPEED, including technical support in learning how to prepare project activity work plans that meet USAID’s requirements, logistical support, and support for lobbying GoM institutions to develop policies to facilitate investment into Mozambique’s private sector. The following comments demonstrate how support from SPEED was perceived by several beneficiaries, specifically with building their capacity. 20 Beneficiary Comments – SPEED Perceptions “SPEED did help them set the foundation; this has allowed them to build networks and partnerships. When asked how they engage or not with CTA (a much larger and more well established organization of business associations), they said they have communicated with CTA and plan to work on coordinated issues so not seen as competitor. Also, their target populations are women and youth entrepreneurs.” - Tiri Pamodzi (a small business association for women and youth) “Without SPEED’s support we may have been forced to close down”-Anonymous* “CTA has stronger influence (technical capacity) than SPEED in some areas such as with the Private Sector Working Group and with the Mozambican Government.” -Anonymous* a.2. Balancing between a demand-driven approach (external USAID stakeholders) and other task order requirements (internal USAID stakeholders). A demand-driven activity differs from a results driven project on a number of key points including design, implementation, funding, and measurement of impact. The first and most important difference is the manner in which the projects are designed. In the results-driven model, the solicitation and contract will articulate the purpose and requirement for concrete, prioritized deliverables to be achieved over a given timescale. In the demand-driven model, the design will focus on positioning a program to target general policy areas of interest and to exploit opportunities unforeseen during design. Installing such flexibility and responsiveness in the strategy of a program is a difficult task and so is effective management of the program by the client and the contractor. The evaluation, by all accounts, has found that SPEED, as a demand-driven project, has been effectively implemented. This is speaking from a general consensus by considering the positive nature of comments yielded from the data collection process, as well a review of secondary data (annual reports, activity sheets, etc.) that demonstrate tangible results. More specifically, the SPEED activity worked within its areas of interest, responded to program stakeholders and supported various entities in their efforts to create, modify or remove a policy. a.3. How effective was SPEED at the macroeconomic level and sector-specific work? Macroeconomic Level: The final question the evaluation team examined to determine program effectiveness is to look at what SPEED has done at the macroeconomic level. According to key informant interviews, as well as the discussions from the Focus Groups and Workshop, it is generally thought that SPEED has influenced the Mozambican business environment positively. Several key informants commented that SPEED addressed crucial issues in the business environment effectively, offered good business practices, and inspired them through linking them to the US. 21 When looking at macroeconomic indicator estimates and projections below covering SPEED’s implementation period of 2010 – 2014 (Annex 7), one can see that GDP growth remains stable, inflation trends down, the deficit slightly decreased as a percent of GDP, and revenue increased as a percent of GDP. These indicators imply that overall the Mozambican economy has been favorably trending over the duration of the SPEED activity. However, it should be noted that the Current Account Deficit, mostly due to good and service trade deficits, is projected to grow 562% from 2010 to the end of 2014. As shown by the positive Overall Account Balance estimates and projections, this potentially detrimental trend has and is predicted to be offset by an equally large growth in the Capital & Financial Account Balance, which is fueled mainly by FDI directed toward the Extractive industry in Mozambique (Annex 8). Should FDI trends significantly decrease for any reason and fail to offset the trade imbalance, the Overall Account Balance of Mozambique will experience a significant shock and swing in the direction of deficit. Four policy reforms supported under SPEED were macroeconomic level by nature. These include foreign exchange regulation, exchange rate policy, a sovereign wealth fund policy, and a mining and petroleum tax law. While these reforms are significant and will help level the playing field for the private sector in Mozambique as well as contribute to GDP growth and revenue increase over time, these were the only four policies out of the total 59 supported to date that specifically correlate to macroeconomic indicators. Since the main focus of the supported policies was to improve the business environment in the sectors supported by SPEED, namely the agriculture sector, it would be unfair to expect the majority of policies to support macroeconomic change. Such focus is secondary to SPEED, although the activity’s work with the Tax Authority and Customs offices to improve foreign trade systems to spur competitiveness and economic growth are an example of how SPEED has been able to effectively address issues that cross between both improving the business environment and supporting macroeconomic growth. Since key informants viewed SPEED’s influence on macroeconomic policy as positive and the activity did support identifiable macroeconomic policy reforms, one would be led to believe that the activity contributed towards the favorable trajectory of macroeconomic indicators from 2010 – 2014. Trying to further isolate SPEED’s impact to specific national macroeconomic indicators such as GDP, however, is outside of the scope of work of this evaluation and it may not be possible to attribute SPEED’s contribution to these changes. Regarding sector-specific work, the evaluation team had difficulty interpreting the data based on the documents it received. According to the indicators tracking table for FY 2013 Q3, 51 out of 59 policy, regulation and administrative procedure initiatives addressed agriculture, food security, and climate change. However, according to the indicators tracking table for FY 2013 Q4, only 19 of 57 SPEED initiatives addressed agriculture, food security and climate change. Further, only 33 initiatives were referenced in the SITT Summary Table in Annual Report #3, and these were not categorized by sector. An explanation for the 22 difference in initiative numbers and designations by sector was not found. Despite this confusion, several policies were found to address tourism, another focus sector under SPEED, in addition to the macroeconomic level policies and several miscellaneous ones. SPEED’s support to agricultural policies pushed forward reforms on tax regulation, paperwork reduction, removing barriers to agriculture, streamlining of license processes, and others. An extractives strategy, business environment improvement strategy, tourism development policy, land administration procedures, and an agriculture investment plan were among procedure types that SPEED helped prepare for legislature. Recommendations The evaluation team recommends that SPEED deliverables such as annual and quarterly reports delineate between demand-based and task order / results based activities to enable the balance between the two to be better measured. While the evaluation team received only positive remarks about how well SPEED balanced between the two, it was not possible to conduct a side-by-side comparison of each due to the lack of their being tracked as one or the other. b. Stakeholder Collaboration The second key question area the evaluation team sought to address in the SPEED Evaluation was that of Stakeholder Collaboration. More specifically, the focus of the data collection and analysis within this topic were the sub-questions: 1. How and to what extent has SPEED engaged, collaborated with, and leveraged stakeholders? 2. How do stakeholders view SPEED’s work? 3. Is SPEED engaging with appropriate key Government stakeholders? Findings b.1. Extent of Collaboration SPEED has successfully engaged, collaborated with and leveraged its stakeholders. Based on data collected through Key Informant Interviews, Focus Groups and a Workshop, the following conclusions were made:  One hundred percent of all direct beneficiaries stated they were involved in identifying their needs.  One hundred percent of direct beneficiaries felt that their existing networks/partnerships had been strengthened due to the support they received from SPEED.  Fifty percent of direct beneficiaries stated they provided input into the design of SPEED. The reasoning for this being under 100% was that some interviewed were new to the 23 program and were not aware as to whether or not the institution which they represented was involved in the design input to SPEED. Also, it is clear that those involved with TIP did have the opportunity to provide input into the design of SPEED and subsequently continued to be involved with SPEED; those that did not provide input were new to SPEED after it had been designed and awarded. SPEED does have mechanisms that allow for appropriate feedback into its activity planning, mostly in the form of ‘retreats’ and workshops. Such meetings are valuable mechanisms for bringing stakeholders together to allow for a more collaborative process.  One hundred percent of direct beneficiaries felt SPEED is a flexible activity. SPEED was very open to working with the stakeholders to address their needs as they evolved. Moreover, being demand-driven, flexibility was defined as being able to address new and/or unexpected policy issues that were having or could have a detrimental impact on the private sector. One example of this was with the resource boom. This was not a key conversation when SPEED first began in 2010, but as it became more of a significant issue in 2012, SPEED was able to turn its attention to this issue. SPEED was flexible enough to refocus time and resources to working with stakeholders on issues relevant to the private sector and the resource boom, and has succeeded at becoming a leader in this dialogue. SPEED is flexible in its response to emerging issues.  Fifty percent of direction beneficiaries felt SPEED’s response time was quick, whereas 40% felt it was moderate and 10% felt it was slow. Response time is an important aspect of ability to collaborate well; therefore this is an important piece of information. The majority did feel SPEED responded quickly and those that felt it was less than quick, did add the disclaimer that they understood SPEED was limited by USAID administrative requirements and that they responded as quickly as they could. Those that felt it was less than quick stated they were aware of the time frame requirements; therefore guaranteed they planned appropriately. SPEED’s response is timely.  One hundred percent of non-beneficiaries felt their organization would benefit from engagement with SPEED or with an activity like SPEED. This clearly supports greater outreach of a future SPEED-like program, if indeed budget allows. SPEED’s original SOW required a broader range of stakeholder collaboration than its predecessor program, TIP, as stated below: Successful reform will require collaboration with a wide-range of stakeholders, employing methodologies that reinforce capacity and understanding. SPEED implementer(s) will need to engage various government, private sector, civil society, and international donor actors as participants in the processes as well as recipients of technical assistance in some cases. During the past five years, USAID has worked primarily with and through CTA and the Ministry of Industry and Commerce (MIC). Under SPEED, the Contractor will have a far wider mandate, engaging any stakeholders necessary to ensure successful implementation. b.2. How Do Stakeholders View SPEED’s Work? Overall, stakeholders had a very positive view of the SPEED activity and their engagement 24 with it. One hundred percent of direct beneficiaries interviewed felt SPEED positively helped to change their organization. The evaluation team felt this question spoke to SPEED’s ability to leverage their work with stakeholders because as their direct beneficiaries became strengthened, SPEED was better able to carry out their work within the policy realm. Eighty percent of the direct beneficiaries felt their work with SPEED strengthened how others viewed their organization. Only 20%, while recognizing the positive support from SPEED, also feel that some outside their organization may have negative views regarding this partnership, which some view as USG intervention. These 80% that felt their reputation had been strengthened as a result of their association with SPEED. Comments such as those below illustrate their positive view of SPEED’s impact on their reputation: Direct Beneficiary Comments “GoM recognizes their work as CTA even though they really know SPEED is in the background of some of its activities. SPEED’s support has helped strengthen their reputation, but GoM is still sensitive to anything foreign driven.” -Anonymous “Yes, SPEED’s support had helped positively influence the private sector's perception of the government.” - Anonymous “SPEED’s engagement has helped with others having a more positive view of them. “Oh, you are working with SPEED, that’s good.”” -Anonymous Out of the non-beneficiaries, the majority had heard of SPEED but had not been contacted directly by SPEED. Several did feel they could benefit from support from SPEED, but did not know how to submit a request or didn’t know if they were even allowed to do so. Many understood the close relationship between CTA and SPEED and said they would welcome more support via CTA, but that CTA was limited in their inclusivity. Certainly, within the limitations of both, SPEED was limited in what outreach it could carry out since its interventions were not ‘demand-driven’ with reference to the entire Mozambican business community. Assuming that greater resources are available for a follow-on SPEED activity, outreach could be broadened and collaboration extended to a greater number of participants through new channels of public outreach or social media. b.3. Engagement with Government Stakeholders The evaluation team finds that SPEED has adequately engaged with appropriate Mozambican government entities in relation to what have become their target sectors areas of focus. The key focus sectors for SPEED since 2010 have been: • Trade and Investment • Agriculture 25 • Tourism, Biodiversity and Natural Resources • Human Rights and Governance • Minerals Below are the Government of Mozambique key partners that SPEED has engaged: • The Ministry of Agriculture • Ministry of Tourism • Ministry of Mineral Resources • Ministry of Justice • Ministry of Transport and Communication • Treasury • Ministry of Finance • Tax Authority • Ministry of Industry and Commerce • Attorney General’s Office In sum, as sectors have become significant focus areas for SPEED, the activity has successfully been able to adequately engage with relevant ministries. Recommendations The evaluation team does not have recommendations for this section due to the positive findings outlined above. c. Sustainability The third key question area that the evaluation team sought to address in the SPEED Evaluation was that of Stakeholder Sustainability. More specifically, the focus of the data collection/analysis within this topic was the following sub-questions: 1. How sustainable is SPEED’s work on strengthening institutional capacity of both private and GoM institutions, especially CTA? 2. To what extent will the policy reforms and policy reform processes create lasting economic benefits? Findings c.1. How sustainable is SPEED’s work on strengthening institutional capacity of both private and GoM institutions, especially CTA? SPEED-supported capacity building activities have resulted in capacity strengthening of public and private institutions. Stakeholders also confirmed that SPEED has contributed to initiating policies, providing corrective input to policies as they develop and killing harmful policies before they are enacted. Furthermore, SPEED has promoted dissemination of relevant information through its website. SPEED publications are appropriate in targeting those involved in policy and the private sector development of Mozambique and present information at the level they need. Such capacity building activities are 26 effective in updating and informing stakeholders in a demand-driven model. Below are quotes from direct beneficiaries that support the case for SPEED having strengthened institutional capacity: Direct Beneficiary Comments “The planning exercise was extremely important. We were trained in a number of issues and have replicated these. We learned how to prepare project activity work plans to meet USAID's requirements. Our organizational capacity improved because of SPEED.” -Anonymous “CTA has stronger influence (technical capacity) now, such as within the Private Sector Working Group and with the Mozambican Government.” -Anonymous “SPEED’s studies have helped them influence policy; therefore has strengthened the Ministry’s efforts and ability to influence policy reform. SPEED opened a platform for dialogue between the government and private sector.” -Anonymous The prospect of sustainability of SPEED’s institutional capacity work depends on the rate at which staff among the supported GoM institutions and private sector firms turnover and whether tangible changes to the way they operate or achievements that happened during SPEED will live beyond such turnover. Staff who participated in SPEED’s many events and forums will likely apply what they learned for the duration of their time in the GoM or at their firm. They will draw upon the connections they made at networking events to advance their directives. Those who traveled internationally may be re-invited to participate in trade agreement meetings or international forums that will raise Mozambique’s profile and attract more investor attention. Those who participated in SPEED-facilitated retreats, such as the EMAN II retreat, will have been exposed to the process of developing a strategy inclusive of public and private sector participants and donor agencies. They may be therefore more likely to adopt inclusive decision making models. The quotes above speak to the positive experience that beneficiaries had with SPEED, but these do not mean sustainable change has, or will, take (n) place. To measure such an effect, a broad-based survey could be administered by the follow-on activity to ask beneficiaries to cite what they’ve achieved in terms of successful trade agreements, business deals, or new business models as a result of their support from SPEED. Such a survey could take place at several intervals over time to measure short or long-term change if this is a high priority for USAID. c.2. To what extent will the policy reforms and policy reform processes create lasting economic benefits? Question: In what way has SPEED influenced the Mozambican business environment? All direct beneficiaries and all partners, except one who had no comment, felt SPEED’s work had a positive effect on the Mozambican business environment. Comments show a consensus 27 on SPEED’s ability to have long lasting effects on Mozambique’s business environment, even after the project comes to an end. Other responses by the interviewees show broad agreement to the propositions that SPEED should support efforts for the private sector to address issues within the purview of government action that limit private sector growth. They also felt SPEED has helped lay the foundation not only for their business association, but also for other businesses, which have had a very positive impact on the business environment of Mozambique. Finally, they agreed that SPEED is shaping discussion and leading dialogue and SPEED speaks a “clear language.” The positive remarks from those interviewed by the evaluation team can be traced to several efforts by SPEED to influence the business environment. SPEED’s support to, and facilitation of, nearly 60 policy, regulation and administrative procedure reforms to date are meant to contribute to improving the overall business environment. These reforms support accelerated international trade, introduction of new regulations, reduction of land conflict issues, improvement to the business registration process, and increases in investments in certain sectors. By providing technical assistance and acting as an intermediary between public and private sector actors to move these reforms forward, SPEED acts as a guiding compass and venue for discussions and analysis to take place with third party assistance. Another way in which SPEED has influenced the business environment is to conduct research and create tools that can help the GoM and other public/private sector actors measure the progress of their initiatives and reforms. In 2013, SPEED conducted a study that looked at the evolution of the business environment in Mozambique and posted the draft on its website with a plan to disseminate the results to a broad set of stakeholders. It also developed a performance assessment framework for the government and sought buy-in from the government to maintain focus on framework. In addition, SPEED worked with several ministries, the CTA, and International Finance Corporation to draft a business environment action plan/annual strategy with key policy reforms to be implemented. This plan was adopted by the Council of Ministers in 2012. In these ways, SPEED has not only helped create a timeframe for acting upon, and a sense of urgency for addressing, otherwise slow moving reforms, but it has also gone through the process of creating buy-in from the highest levels of government. SPEED’s “Communication for Policy Change (C4PC)” strategy is another effort that has sought to influence the business environment and policy reform through improving public awareness of the need to adopt specific reforms. The strategy explicitly includes a framework to influence decision makers. Finally, it can be expected that SPEED’s capacity building efforts and organization of knowledge exchange forums indirectly influence the business environment, particularly when participants were able to travel outside Mozambique to meet their counterparts in other countries, be exposed to new ideas, and conduct bilateral meetings on the side to generate 28 interest in foreign investments in Mozambique. The ways in which SPEED has sought to influence the business environment as an enabler, facilitator, and source of technical assistance can therefore be tied to the positive comments received by interviewees above. The fact that they independently re-affirmed what was already suspected by USAID, which is that SPEED is a positive influence on the business environment, can serve as additional evidence that USAID’s investment in SPEED is worth continuing. Question: In what way has SPEED influenced policy reforms in Mozambique? Do you know of any policy reforms SPEED has influenced? And, do you think these will create long lasting economic benefits? Four out of ten direct beneficiaries felt SPEED directly helped their lobbying efforts, whereas others felt it did not do so directly but has made their organization technically stronger, thus has worked toward strengthening their ability to lobby for policy reform. If beneficiaries begin to see results to their lobbying efforts, they’re more likely to continue lobbying and gain more traction, which could lead to sustainable change if policy reforms are made as a result. SPEED’s initiation of these efforts may have therefore enabled these beneficiaries, but signs of sustainability could be better taken from results achieved. Being able to point to policies passed directly as a result of SPEED-supported lobbying by the private sector and implementation of the new laws would be the best indicator of sustainable change. However, an attempt to trace an effort by SPEED to help a direct beneficiary to initiate lobbying and then confirm that a policy was not only passed, but implemented, as a result did not yield results by the evaluation team. The beneficiaries interviewed did not state they were aware of specific policies under which SPEED supported lobbying efforts. It can, however, be seen from SPEED’s most recent Performance Management Plan (PMP) in late 2013 that 16 SPEED-supported policy reforms have been passed to date and implementation has begun on 17 recent reforms (some presumably passed independently of SPEED’s assistance but with SPEED support for implementation). Below are several quotes from direct beneficiaries regarding the ways in which SPEED has influenced policy reforms: Direct Beneficiary Comments “SPEED has positively influenced policy reforms in Mozambique that have ultimately worked toward support private sector growth.” -Anonymous “SPEED is of most value in initiating dialogue on policy reform” -Anonymous A greater number of direct beneficiary interviews may speak to greater awareness of specific 29 policy reforms SPEED has influenced than what was recorded by the evaluation team. If that would be the case, such interviews could dig deeper into beneficiaries’ opinions about the likelihood of the reforms creating long lasting economic benefits. The evaluation team treated the questions in this section as opinion questions and therefore analyzed the results of the interviews, not conducted its own its own independent study of the likelihood of long-lasting economic benefits stemming from SPEED-influenced policy reforms. The evaluation team does believe that the reforms that have been passed and implemented through SPEED’s support deserve close follow-up and monitoring. Perhaps by the close of the program, or under a follow-on activity, factual evidence of long lasting economic benefits can be collected. Recommendation To measure institutional capacity strengthened by SPEED, a broad-based survey could be administered by the follow-on activity to ask beneficiaries to cite what they’ve achieved in terms of successful trade agreements, business deals, or new business models as a result of their support from SPEED. Such a survey could take place at several intervals over time to measure short or long-term change if this is a high priority for USAID. d. Gender The fourth key question area the evaluation team sought to address in the SPEED Evaluation was that of Gender. More specifically, the focus of the data collection/analysis within this topic was on two sub-questions: 1. How has SPEED addressed the different Gender roles and needs when selecting policy areas, conducting policy analysis or advocating policy change? And how could any changes or improvements be made in this area? 2. Are there particular policy areas that have to date or could in the future especially benefit women traders or entrepreneurs? Findings d.1. How has SPEED addressed the different Gender roles and needs when selecting policy areas, conducting policy analysis or advocating policy change? It should be noted that this sub-question could not be assessed in its entirety, since SPEED is a demand-driven activity, meaning the focus of SPEED initiatives were determined by the requests presented to them by stakeholders (internal and external to USAID). Another aspect of being demand-driven was determined by which sectors rose to the top of the priority list for the country as a whole, and SPEED would try to refocus their work as best determined within the current realities. Since the dialogue on ‘gender’ is quite limited on both national and local levels, specific gender and women-focused policy priorities were not often in the forefront. For these reasons, SPEED had limited decision making authority as far as selecting policy areas of focus that specifically focused on gender and women’s empowerment. The only place gender is referenced in the original SPEED SOW is on page 15, under 30 Activity 4D where it reads “Investment, stating that SPEED needs to look at the Gender Impact of Regulation (inappropriate regulations sometimes hurt women more than men).” The original SOW also suggested as a possible SPEED stakeholder to engage with women business associations and SPEED has indeed done this. SPEED’s response to their addressing the gender requirement under this Activity was as follows: “We (SPEED) have addressed the gender dimension in the case of some policies, like illegal taxes and access to finance. We have also supported several organizations that are oriented toward women's issue as they relate to entrepreneurship and the business environment (Tiri Padmozi, AWEP, NFNV, etc.).” Even though SPEED was not mandated by USAID to take on a certain number of women￾focused initiatives, they have received requests to work on several. Below points out specific initiatives that SPEED has worked on where they feel they have been able to focus on specific issues of women in the private sector: ● Support for the African Women’s Entrepreneurship Program, who are trying to form an Association in Mozambique. This is an effort being carried into 2014 and more detail should be expected in the next round of reporting. Current reports do not speak to this activity. ● A significant amount of SPEED’s work has been on the agriculture sector. A few key initiatives by SPEED in the agriculture sector have been: 1. SPEED-sponsored the position of a senior economic advisor to the Minister of Agriculture; 2. SPEED has focused on issues of the impacts of tax regulations on the agriculture sector; and 3. per request from ACIS, SPEED produced "The Legal Framework for Recognizing and Acquiring Rights to Land in Mozambique in Rural Areas - a Guide to Legalization and Occupation 3rd Edition (Land Manual)" in 2012. In 2013, SPEED increased their efforts to disseminate the land manual. This expanded effort is in direct coordination with the Ministry of Agriculture and aims to target provinces that are experiencing significant land tenure issues and provide training programs. In this sector, women provide 65% or more of the labor force depending on the region. SPEED’s work has not had a direct focus on women within the agriculture sector, but the work SPEED has done in this sector has had the potential to impact both males and females alike. Even though SPEED has stated this work has the potential of impacting males and females working within the agriculture sector, there is no reporting of differentiated approaches based on gender. ● Support in the area of access to finance at the policy level for “New Faces New Voices.” Current reports do not speak to this initiative, but it is expected that the results of this work will be reported during 2014. 31 ● Support for Tiri Pamodzi, an association for youth and women entrepreneurs. SPEED provided a consultant to Tiri Pamodzi to work with them to develop their Strategic Plan. Furthermore, they provided $26,720 worth of equipment to include computers and other training materials for their Training Center. Within this work, there was no specific programming oriented toward women, but since Tiri Pamodzi is an association targeting youth and women, this initiative can be said to support women in the private sector. The PMP shows only two areas where data has the option of being disaggregated by sex. One is in jobs created and the other consists of participants in USG supported trade and investment capacity building trainings.1 SPEED did indeed have several efforts that focused on women-specific issues within the policy and business arenas of Mozambique; but as stated, since this was not a results-driven activity, there were no exact target numbers for such a focus. Understanding this, the majority of what could be assessed was whether or not SPEED staff looked through a ‘gender lens’ when carrying out their overall policy work. The evaluation team determined that there is space for improvement for SPEED and the follow-on activity to better address policy concerns through a “gender lens”. Gender Units in the Ministries exist, as does a Donor Gender Working Group, but SPEED has not engaged with either entity. The question of a gender-focus within SPEED initiatives was brought up during the data collection with SPEED stakeholders. During the data collection when the word gender was first brought up, the initial reaction by many interviewees was more in terms of “we have plenty of women working with us or it does not matter if man or woman, if qualified, we hire.” Many also responded with a bit of a snicker when first hearing the word gender. It should be noted that 64% of the stakeholders involved in the data collection were male and 36% were female. The one Focus Group in Maputo that focused on the topic of Gender had two female participants. There was not a significant difference between male and female perspectives to the word ‘gender’. 100% of Direct Beneficiaries and Partners interviewed stated their work with SPEED had nothing to do with ‘gender.’ They did not see the relevance or need for any gender specific focus to be incorporated into their requests. Most 1 The first indicator is “Jobs Created.” Referencing the data in PMP from the end of quarter three, fiscal year 2013 (Annex 9), there is no target set for either males or females in 2011 for this indicator, but an actual is given as 16,000 but not disaggregated by sex. In 2012, a target number is set for males and females, but no actual is given. For 2013, there is no target and not actual given. There is an inconsistency in numbers depending on which PMP document is referenced. If one references the PMP data from quarter four, fiscal year 2013 (Annex 9) the numbers vary in that the actual for 2011 is only given in the slot for males and the actual for 2012 is given as 1,118 for males, whereas the other PMP showed no actual for 2012. Whereas the 2013 target in this document is set at 5,000 for males only and the actual is given as 2,627 for males only. This PMP document sets a target for males at 5,000 for 2014, whereas the other document lists no target. The second is indicator 4.2.2 “Number of participants in USG supported trade and investment capacity building trainings.” Targets were neither set for males nor females in 2011, but actuals were recorded as 1,118 males and 276 females. For 2012, targets were set for 1,200 males to be trained and 400 females to be trained; but this indicator was subsequently dropped; therefore no actuals were recorded after this time. This data is consistent in both PMP documents. 32 felt that policy was gender neutral and that there were no differences to the challenges men or women faced in Mozambique’s business environment. Moreover, one of the participants interviewed that works with registering businesses stated that an equal number of men and women register businesses. A copy of this data was requested so that this statement could be verified, as well as the team could assess if males and females tended to register businesses in different sectors. This data was never supplied to the team. These comments speak to the perception of stakeholders as to the need for a focus on gender considerations within private sector development and policy reform within Mozambique. There is a limited awareness and understanding of gender and its true meaning. The World Health Organization (WHO) defines as: "Sex" refers to the biological and physiological characteristics that define men and women. "Gender" refers to the socially constructed roles, behaviors, activities, and attributes that a given society considers appropriate for men and women. Gender within the context of a development project requires a focus on males and females throughout all phases of a project’s life cycle. A focus on gender does not mean one sex will receive more attention; rather an honest effort will be made to address the unique needs and capabilities of males and females within the target communities. Moreover, strategies will vary when need be, to best address these needs and a balance of efforts will be realized to guarantee that neither sex falls in the shadow of the intended work. Understanding gender from this perspective, then allows one to understand that policy is not innately gender neutral. It is a reality in current and post-conflict societies that women are under-represented at both the macro and micro levels. The lack of an enabling business environment is even less so for that of women since their unique situation tends not to be taken into account. Policy that does not recognize this, does not reach the entire populous. Failing to create and implement policies which actively support women, minorities and rural entrepreneurs has resulted in costing nations in productivity worth billions of dollars. Even though most interviewees felt there were no major differences in women’s or men’s situation in the policy and business arenas of Mozambique initially, once the meaning of ‘gender’ was defined more clearly, they did acknowledge a need for a closer look to be taken at this issue. They also recognized the possible implications at the policy level that may differ between men and women, specifically as relates to private sector development. A few comments from the interviewees on how women may be disproportionately impacted in this context included: Interviewee Comments on Gender “In the agriculture sector, the overall obstacles may be the same for both males and females but women are disproportionately impacted in several areas such as with the illegal fees in trade.” -Anonymous “There is a significant problem of financial literacy among both males and females, but it is more prevalent with women.” -Anonymous 33 “Access to finance does differ among males and females. The banks do see the value of investing in females because see them as being more responsible with following loan terms and are under the perception that women reinvest in family and save, whereas men expand more businesses. However, women’s access to credit tends to require a longer application process; the more lengthy process is necessitated by the fact that less than 20% of women possess proper identification and have to go through a lengthy and costly process to obtain their identification documents before they can open accounts.” -Anonymous “Limited access to collateral is also an obstacle for many women, few of whom own land or even have tenure rights recognized under communal tenure.” d.2. Are there particular policy areas that have to date or could in the future especially benefit women traders or entrepreneurs? According to a World Bank report on Gender Equality and Development in Mozambique (2012)2 , there are several significant gender considerations that are relevant to the policy and business environments of Mozambique. The study states that these considerations have evolved as a result of war, urbanization, labor migration, process of commodification of social relations, and impoverishment. The World Bank states that as the country moved to a liberalized economy after the war, the changes from a socialist economy resulted in many government employees being laid off. The first to get laid off were women. There is no link between the number of women working in government positions and women-friendly policies. There was also a decrease in the number of women associations after the war. The World Bank’s report states that both the Mozambican Government and international donors do not take the issue seriously enough and there are not adequate gender policies and interventions in place in Mozambique. Mozambique has ratified several international treaties that support women’s empowerment, such as the “Convention on the Elimination of all Forms of Discrimination against Women” (CEDAW), yet enforcement is weak. Even with the treaties/agreements Mozambique has become a signatory to that work toward women’s equity and equality, Mozambique still does not have a specific policy framework for women’s enterprise development. (Cilo Consulting 2011). According to the Global Entrepreneurship Monitor (GEM)3 , women’s presence is growing at twice the rate in developing nations as compared to developed nations in the private sector.4 One reason given for this is that the majority of women have limited access to formal employment, specifically in Mozambique. Worldwide, it can be argued that for this reason, many women tend to become entrepreneurs and make their own opportunities; therefore it is critical that an enabling environment must be built around gender equity in order to promote female empowerment. 2 Background Paper, Mozambique Country Case Study: Gender Equality and Development for the World Development Report (World Bank 2012), Tvedten, Inge, 2011 3 Minitti, M., Arenius, P., & Langowitz, N. (2005). Global Entrepreneurship Monitor: 2004 report on women and entrepreneurship. Babson Park, MA and London: Babson College and London Business School. Retrieved from http://www.bdc.ca/fr/Documents/woman/GEM_Womens_Report.pdf 34 Of most importance is that any work in the policy arena be approached in a manner that clearly identifies the variations of how the same policy will impact differently males compared to females in any given sector. In Mozambique 12% of men but only 1% of women have access to private sector employment (World Bank 2012). Women make-up 59% or greater of the informal working sector (Cilo Consulting 2011)5 . Many policies do not reach the informal sector, which means women working in the labor force will disproportionately not have access to the benefits of policy reform work done by SPEED that will support private sector development. Just as significant is that out of the 1% of women that do enter formal employment in the private sector, many women have succeeded in growing their businesses to extend outside the range of micro lenders. There is limited policy in place that specifically supports women-owned businesses at this level. There are significant differences between women living in rural vs urban areas and the policy impacts on each population must be considered. Moreover, there are even regional differences from north to south of the everyday reality women face. According to a 2012 Mozambican Labor Market Profile released by Germany’s Ulandssekretariatet as shown in figure two below, women have the largest presence in the sector of trade, restaurants and hotels; second largest in finance, real estate and business services, with 3rd largest in public administration, education and health. Women also have a presence in mining/quarrying, manufacturing and construction, even though significantly less than other referenced sections. Figure 2: Sectoral Employment Disaggregated by Sex Source: Labor Market Profile 2012. Germany’s Ulandssekretariatet http://www.ulandssekretariatet.dk/sites/default/files/uploads/public/Afrika/Landeanal yser/labour_market_profile_2012_-_mozambique_web.pdf 5 The Enabling Environment for Women in Growth Enterprises in Mozambique. Assessment Report. Prepared by Cilo Consulting and edited by SA Ideas. (January 2011) 35 Recommendations It is critical that the first step is to build awareness among those working in the private sector and policy arenas of Mozambique on defining gender and its implications within these arenas. Law and policies must be reviewed through a gender lens to guarantee they are not discriminatory to a greater degree to either men or women. Specifically speaking about women’s role within the private sector, it is critical to recognize the key sectors where women have a strong presence and properly target these sectors in future work. Furthermore, since there is variation among women depending on their socio￾economic level and their region of residence, these variations must be considered when assessing the potential impacts of a said policy. During SPEED’s remaining time and under any follow-on activity, there are a number of initiatives that could be carried out in regards to gender and working through a gender lens. One primary example would be for the activity to carry out a full analysis of what the potential policy and business implications are from a gender perspective. This information can be of much value for providing guidance on how future programming can better carry out targeted gender-related work. An initial assessment could consider the following research areas: 1. An analysis of market constraints (i.e. regulation, product affordability and appropriateness, physical distance, financial capability, cultural constraints, infrastructure, skills gaps), highlighting where constraints affect women disproportionately (GIZ Tool Kit: Promoting Women’s Financial Inclusion (pg. 50) 2. Review of international best practices in policy in regards to gender 3. Women in labor vs at decision making levels by sector 4. Number of women registered businesses per sector and number of women focused business associations 5. Women’s access to credit 6. What are the policy implications on women in the informal sectors, specifically where their labor is unpaid and uncounted in the GDP. SPEED could also build awareness among stakeholders on women in policy and women in business specific issues. Current awareness levels are extremely limited. At the start of the new activity, it is recommended that follow-on activity staff introduce the activity to Gender Units in the GoM. This will allow opportunity for those Gender Units to provide guidance on how they can or cannot support such an effort. One example in the agriculture sector may be if a SPEED-like activity has been requested to review a draft bill, they could then consult the Gender Unit of MoAG to learn of any policy considerations that may differ between men and women working within this sector. The Gender Unit will also be able to provide guidance on the Gender Strategy paper that was developed by MoAG. It may be true that the Gender Units are weak, but one complaint is that they have not been adequately recognized by both the Mozambican Government and the international donors. In order for such a mechanism to be strengthened, the first step is recognition. Once strengthened, these Gender Units can be of much value for future international aid work in the policy arena. 36 SPEED has been demand driven, but within a limited pool of stakeholders. If indeed the follow-on activity’s budget allows and this pool can be expanded, then there will be a need for increased outreach initiatives. These initiatives should guarantee they target relevant women working in the private sector, as well as those focusing on issues of policy within a gender context, especially if their literacy levels are low and they live in rural areas. Another area where SPEED or the follow-on activity could focus efforts would be on access￾to-credit issues for women. Any initiative that would result in a quicker loan process for women is recommended. Additionally, in the agriculture sector, financial literacy is known to be particularly weak among women. Training on loan application processes should be considered for this target group via CSO’s or other SPEED stakeholders. In general, given the number of events, knowledge forums, and training that SPEED has provided, it’s recommended that an event be held specifically for women in the private sector. SPEED could also invite gender focal point participants to the Private Sector Working Group and having a representative join the Donor Gender Working Group. The follow-on activity may also want to include, in its short-term technical assistance budget and roster, a role for a Gender Advisor to be utilized as needed. As noted in the findings section, there was a significant difference in the number of male vs female participants in SPEED training programs. There are reasons why these numbers cannot be equal, but some consideration should take place as to whether a quota for women or greater outreach to attract women to the initiatives could take place. It is recommended that a follow-on activity develop indicators that will better measure the impact of policy reform specifically on women traders and entrepreneurs. Furthermore, it is recommended that SPEED’s future reporting to include activity sheets, quarterly and annual reports, include a section specifically focusing on their work directly with issues of gender and specifically women’s empowerment within the private sector and policy arena of Mozambique. Due to the sectoral cross-cutting focus of many of SPEED’s efforts such as in areas of taxation and the business registration and licensing, more consideration should go into how such policy aspects affect women entrepreneurs different than male entrepreneurs. With the “Institute for the Promotion of Small and Medium Enterprises” (IMPEME), housed in the Ministry of Industry and Commerce, being responsible for developing the most conducive environment for the growth of SMEs, it is suggested a follow-on SPEED activity increase its work with IPEME that will focus specifically on issues relevant to women-owned SMEs. 4.2 Quantitative Data The evaluation team conducted an analysis of quantitative data by looking at the design, development, and implementation of the SPEED activity. Our findings are presented in the sections that follow. 37 a. Initial Activity Design This section looks at SPEED’s original scope of work and strategic design framework to identify and assess the adequacy of stakeholders involved in the design process and targets and results to be achieved. Findings a.1 Available Design Documents The only document available for this period is the initial solicitation. Other design-phase documents which must have been prepared at the time the project was being designed are not (or no longer) available on the USAID website. It has not been possible to locate copies of these documents at the time this report was being written. a.2 Solicitation Document (RFP) The only initial design documents available to the evaluation team is the RFP (RFTOP 674- 10-020 SPEED Mozambique), which states the overall objectives, means for achieving them, suggested approaches, possible partnerships and resources necessary to achieve clearly defined overall policy reform objectives. SPEED was to operate in selected policy areas: 1) trade, 2) investment, 3) finance, 4) agriculture and 5) tourism. The approach outlined for SPEED was systematic and recognized the need for both policy development and measures to implement those policies which were adopted. The RFP called for an approach which substantially and comprehensively changed targeted systems, involved participation by the stakeholders most affected, championed by powerful actors capable of moving their agenda forward, and based on the demand of those in power and of those affected by policies. The approach offered in the RFP is results-oriented focusing on the policy issues most relevant at the time the RFP was written. However, the RFP contains an interesting clause which notes the willingness of the Mission to accept some flexibility in achieving broad policy reform objectives: “SPEED implementation will enable reformers to identify demand but also to take advantage of opportunities when the political winds shift and create unexpected reform opportunities.” In fact, massive and unforeseen changes have occurred in Mozambique which have made the inclusion of this phrase prescient and the Mission’s willingness to taking advantage of policy opportunities as they arise, a factor in the success which SPEED eventually did achieve. a.3 Entities involved in design and completion Ten organizations who were direct beneficiaries of SPEED were interviewed by the team with respect to their participation in its design. Three of them had to be eliminated because double counting and respondents from the agriculture area (Ministry of Agriculture and CEPAGRI) who could not confirm or deny their institutions’ participation. Of the seven valid respondents, five had some measure of participation in SPEED’s design. But only three of these valid respondents subsequently participated annually in SPEED’s annual planning 38 exercise for policy work to be carried out. It appears that there was substantial input from outside USAID into the design of SPEED, but that participation in determining the issues SPEED worked on in any given year is lower. SPEED did host ‘retreats’ that allowed for USAID and stakeholders to provide input into future initiatives. It should be noted that such retreats were not open to all stakeholders, only key stakeholders of SPEED, such as CTA, which the evaluation team found to be appropriate. Recommendations Not every stakeholder needs to be consulted at the launch of a new activity, therefore the evaluation team recommends that the follow-on activity use the same approach as what was used under SPEED. b. Management and Tracking of SITT Mechanisms This section analyzes the consistency among how the tracking tools utilized by SPEED are defined and reviews the consistency of targets and data reported using these tools. Findings b.1. Consistency in Defining Tracking Tools SPEED uses a number of decentralized and non-integrated tracking tools that closely overlap. The PMP tables and activity sheets found in the annual and quarterly reports contain some matching targets and results while methods for collectively categorizing initiatives and activities by sector appear to be missing. The introduction of SITT indicators seems to have been a change in name only from "PMP" to "SITT" as shown by comparing the annexes for each in the second and third annual reports. All the information found in the SITT summary table in annual report three seems to have been pulled directly from the PMP worksheets. The use of multiple tracking tools or tracking tool names may serve a purpose for coordination between the SPEED team and USAID COR, however, the evaluation team found difficulty in determining which tracking tool is best suited for which purpose, aside from isolating data from certain reporting periods. The PMP appears to be the best source of cumulative data. During interviews with SPEED and USAID team members, the evaluation team noticed that the PMP and SITT were sometimes referred to interchangeably, which makes sense, but at other times they were perceived to be different indicators, which they do not seem to be. It was stated in the second annual report that SPEED “adopted the DAI TAMIS system to more readily provide reports on ongoing activities and enable different data cuts – by instrument, by staff member, by category (agriculture, tourism, DG), etc.” This appears to be another tracking tool that was not made available to the evaluation team, therefore consistency cannot be verified, but TAMIS adds to the number of tracking tools being used. The major finding here is that tracking tools cannot be consistently defined by the evaluation team despite their tracking of the same data from the same activity. Please refer to the recommendations below for an elaboration of what the evaluation team recommends based on this finding. b.2. Consistency in targets and results throughout tracking tools 39 The evaluation team reviewed the targets and results reported in each of the tracking tools for consistency and accuracy across time periods and tools. Below is a review of each tracking tool. Performance Management Plan (PMP) Summary Table No targets were set for the first year of SPEED in 2011. Data reported in the PMP begins in 2011 and the setting of targets started in 2012. Out of eleven indicators total, four indicators were tracked consistently from the beginning of 2011 until the end of 2013 (Annex 10). Other targets were dropped or added in 2012. Performance Management Plan Activities The evaluation team found that policy reform names, descriptions, legal instrument categories, policy type, stage achieved during the reporting period, status, significance, and results achieved were consistently completed with full information for 59 policies as of the end of quarter three in 2013. However, in the indicators tracking table for FY 2013 Q4, policy type classifications were drastically changed and two initiatives seemed to go missing. For example, the record for Anti-Corruption Legislation admin procedure and Support of New Alliance projects in Mozambique admin procedure seem to be missing from the Q4 indicator tracking table, the former also from the Q2 indicator tracking table. The evaluation team also reviewed how SPEED tracked whether the policies were advanced through CSOs, GOM reformed with USG support, or related to sustainable natural resource management or democracy & governance and found that this information was complete. A separate worksheet on “Policy Advocacy CSOs Strengthened” that tracks assistance by type for specific CSOs was found to have been completed consistently and across all three years. As previously referenced, a tracking tool that appears to be a more updated version of the PMP from the end of quarter four in 2013 may supersede the PMP for the third quarter of fiscal year 2013. The label categorizes it as an SITT, but everything else about the table resembles the PMP tracking tools, therefore it’s unclear whether this data should be compared against previous PMP data or whether it stands alone or is meant for the quarterly report. Activity Sheets The activity sheets mirror the PMP tracking tools in their reporting of targets and results but also include additional financial information, narrative descriptions, explanations of activities, and information about upcoming activities. They also include a section for “Evaluations, Assessments, Studies, and Audits” that the PMP tracking tools do not contain. There is less consistency among activity sheets due to the variations in narrative descriptions. Annual and Quarterly Reports Annual reports two and three contain what appear to be similar indicator tracking summary tables in format and structure that identify initiatives (policy, regulation or admin procedure) number, name, legal instrument, and stages achieved during the reporting period. However, in annual report two, this section is labeled as the “PMP Summary Table” and in annual report three; it is labeled “SITT Summary Table.” This change in name from PMP to SITT was also made in the last quarterly report for FY 2013. The data reported in these tables are mostly consistent with what’s reported in the PMP worksheets and activity sheets but the presentation, format, and reporting periods vary. Due to the various reporting periods 40 (quarterly, annual, and cumulative), it’s difficult to assess the consistency of each policy and indicator across the entire set of tracking tools using one comparison method. World Bank Doing Business Ranking The World Bank’s Doing Business ranking is mentioned separately because the evaluation team learned through the data collection process that this is an indicator closely tracked by both the SPEED team and USAID, however, it’s not included in the tracking tools above. . The SPEED evaluation SoW includes the WB Doing Business Ranking as an indicator. According to the World Bank’s Doing Business ranking, in 2010, Mozambique was in 135th position and fell to 146th position (out of 185 countries) in 2013. Its 2014 ranking improved to 139th place. This improvement is solely attributable to SPEED’s work construction permits which dramatically decreased (by 247 days) the time it takes to get a construction permit in Maputo. SPEED has also worked on the Insolvency Law governing bankruptcy which had been based in a very old Code of Civil Procedure and was time cost and onerous; as a result of its work, the new insolvency law was passed. Despite shortcomings, the country’s Doing Business ranking is likely to remain a matter of concern for GOM, for activities like SPEED which attempt to improve the business environment, and for donors such as USAID which support these changes. Jobs Created Indicator The “jobs created” indicator was reported for FY 2011, when no target was set and an estimate was made of 16,000 jobs created, while for FY 2012 a target of 15,000 jobs was set as a target but no “actual” was reported. For FY 2013, this indicator which is problematic at best was dropped. In fact, it is difficult on a policy project to estimate how many jobs are created by any given or any set of policy reforms, whether they are permanent, or whether other jobs are lost as a result. Dropping this indicator was a wise decision, since it can only be estimated and only very imperfectly at that. Recommendations It would be helpful if the SPEED Annual reports presented breakdowns of the number of policies amended, developed and/or blocked by “Legal Instrument Category” and “FTF Policy Type”. Policies are in fact labeled and categorized accordingly by these two category descriptors, however, the policies are presented in list fashion without aggregations dedicated to emphasizing the number or percent of policies by “Legal Instrument Category” and “FTF Policy Type.” Such aggregations would improve the reports ability to more clearly demonstrate SPEED programmatic focus and impact. It would also be helpful if SPEED annual reports included tables that highlighted and displayed major indicators of its program in a consistent manner. For instance, the tables at the end of FY 2011 Annual report are not repeated in format in FY 2012 and FY 2013, which makes comparison from year to year more difficult. 41 In addition, the World Bank “Doing Business” indicator should be re-incorporated into the tracking tools it is said by USAID that the indicator will be retained in the follow-on policy reform activity. Despite shortcomings, the country’s Doing Business ranking is likely to remain a matter of concern for Government, for activities like SPEED which attempt to improve the business environment and for donors such as USAID, which support these changes. Linking data between PMPs where possible would reduce entry error. Second, linking data between PMPs would also offer trails to auditors which would show them where the numbers and counts are coming from. These links and trails have the effect of making it easier for them to verify or discount the data. Ideally, all the PMPs from day-one of a new policy reform project might be kept in a single MS Excel workbook. Developing, maintaining and tracking indicators for policy change activities is not an easy task. A results-based approach where outputs and outcomes are narrowly defined would be easier to measure but also would be counterproductive. At the time policy or regulatory interventions are proposed, indicators specific to that policy or regulatory reform should be determined and their method of calculation agreed upon. However, reporting and analysis could be facilitated if newly added indicators are estimated backwards in time wherever possible to allow a long time series to view changes over time in areas that are of interest. Indicators should allow measurement of outcomes related to specific policy changes over the life-of-the-project and beyond Through indexing by areas of interest related to policy interventions pursued, it will be possible to show which interventions were related areas of interest to specialized offices such as Feed the Future, which need to be able to report on policy changes having positive outcomes for their specific areas of concern. All Indicators: The process for measuring/calculating indicators needs to be transparent in order for third parties to be able to assess their effectiveness. This is also important for anyone internal to the program that may need to review the data and follow the trail from raw data to aggregates. This is not the case for many aggregates with the SPEED PMP Worksheets. Example: How are # of jobs created and money saved through reforms calculated? Recommendations for SITT management: • All SPEED initiatives (policies, regulations, and admin procedures) and activities should be assigned to or categorized under appropriate sector, funding source, target policy area, etc. and located in a central database. • Base/source/raw data for all aggregates should be easy to access and link to aggregates. Spreadsheet links and formulas should be apparent for trouble shooting purposes. 42 • Special formulas and methodology used to calculate/generate aggregates should be clearly identified, explained and justified. • Publications should be assigned to the same topics/sectors as initiatives and activities. • There should be consistent categorization of all activities across all versions of M&E documents or tracking systems by year, quarter, etc. • Drastic changes in categorization (such as those seen between FY 2013 Q3 and Q4 for initiatives) should be identified and explained. Continuity should be maintained to the greatest extent possible in order to allow comparisons over time. c. Outcomes This section identifies the policies addressed under SPEED and the civil society organizations, private sector companies, and government institutions that received assistance from SPEED. Findings c.1. Policies Amended, Developed, or Blocked Due to SPEED’s Involvement At the end of quarter three in 2013, a total 59 distinct policies have been amended, developed, or blocked due to SPEEDS involvement according to the PMP (Annex 11). According to the PMP, 24 policies have been analyzed since the launch of SPEED. A total of 26 policies have been presented for consultation. A total of 17 policies have been presented for legislation. Sixteen policies have passed/been approved and implementation has begun on 17 policies. The PMP shows that targets were met or exceeded in 2012 but not met in 2013. c.2. List of Institutions by Sector, Type and Value of Grant or Other Form of Assistance SPEED assisted 43 institutions under the life of the activity to date (Annex 12). Nine institutions which received SPEED assistance were interviewed during the course of fieldwork. They appear to be representative of the CSOs and other organizations assisted which are listed in the SPEED Project Activities of the 2011 annual report and are referred to in the text of other reports. Those interviewed are: • ACIS Associacao de Comercio e Industria • IPEME Small Medium Enterprise association • CTA Confederation of Business Associations • Tiri Pamodzi (Instituto de Desenvolvimento e Empreendedorismo) • CACM (Centro de Arbitragem, Conciliação e Mediação) • CEPAGRI (Agricultural Promotion Central ) • AT/Customs Service 43 • Ministry of Agriculture • Ministry of Tourism The interviewed CSOs include four business associations, one agricultural promotion organization and the center for arbitration and mediation; on the GOM side, it includes two ministries and the government’s customs service. The distribution of institutions interviewed generally reflects the types of Mozambican associations and Government agencies dealing with business which SPEED has assisted. In 2011, SPEED reported having worked with 9 CSOs, the Customs Service and Institute of Tourism, Ministry of Tourism, the Ministry of Culture. For business associations, SPEED provided financial assistance to these associations. This assistance took the form of grants for construction and equipment, organizational and business development support for the growth and improved effectiveness of the organizations themselves, support for meetings, seminars and conferences, hiring consultants for specialized studies which they participated with SPEED in writing. SPEED also provided salaries to allow some staff to be imbedded in both CSOs and government ministries. For ministries, SPEED has assisted in them in preparing studies on specialized topics of interest (Open Skies and visa studies for Ministry of Tourism). Recommendations Analysis of SPEED policy interventions should be simplified in the SPEED SITT and with reference to the annual reports. Currently, annual reports do not aggregate similar types of policies nor do they provide breakdowns of the number of policies amended, developed and/or blocked by “Legal Instrument Category” and “FTF Policy Type.” Policies are in fact labeled and categorized accordingly by these two category descriptors, however, the policies are presented in list fashion without aggregations dedicated to emphasizing the number or percent of policies by “Legal Instrument Category” and “FTF Policy Type.” Such aggregations would improve the reports ability to more clearly demonstrate SPEED programmatic focus and impact. Fiscal year 2011 tables are not found in fiscal year 2012 or 2013, or are found, but in a different format with different information within: • SPEED Project Activities Table • SPEED Project PMP - Summary Table • SPEED Project PMP – Worksheet 1: Policy Reform • SPEED Project PMP – Worksheet 2: Policy Advocacy CSOs strengthened 4.3. Literature Review *Below is an excerpt from the full literature review report submitted separately to USAID in 44 conjunction with the SPEED Performance evaluation* This Literature Review is a component of the overall performance evaluation intended to assist USAID/Mozambique to conduct a performance evaluation of its Support Program for Economic and Enterprise Development (SPEED). As stated by USAID, a Literature Review is an important component of the overall program performance evaluation, as SPEED is one of the only organizations in Mozambique that makes available policy, economic, and other analyses of this kind in Mozambique. In addition to informing on possible programming needs for SPEED, these publications are also an important source of information for stakeholders and a promotional tool for the program. Methodology A combination of stakeholder survey methods and technical analysis was employed in this literature review to assess four major focus areas that were also used in the overall SPEED Performance Evaluation: • Effectiveness • Stakeholder collaboration • Sustainability • Gender Stakeholder survey methods used key informant interviews, focus groups and a workshop to obtain info from stakeholders concerning their: • knowledge about publications on the SPEED website • use of the publications • perceptions about the publications related to quality and use by the stakeholder community Technical analysis was used to assess the sample of publications provided in terms of: • Writing Level • Language (English vs. Portuguese) • Alignment with SPEED indicators and actual activities carried out • Flow and organization in writing, proper use of grammar and formatting • Technical level of writing; appropriateness for the target population • Adequate level of detail in regards to stated topic and target audience • Use of Social Media and Networks • Website Functionality Due to time and budget constraints, the SPEED evaluation team was directed by USAID to select half of the publications from the initial list provided by USAID for review. The sampling was a balance of publication types and topics. The initial list of publications (Master List) provided by USAID to the SPEED evaluation team included: • Reports: 36 (20 English; 16 Portuguese) • Notes: 25 (8 English; 17 Portuguese) • Presentations: 20 (11 English; 9 Portuguese) • Blogs: 20 (3 English; 17 Portuguese) • Total: 101 (42 English; 59 Portuguese) 45 The sample of publications selected for review included: • Reports: 18 (10 English; 8 Portuguese) • Notes: 12 (4 English; 8 Portuguese) • Presentations: 10 (5 English; 5 Portuguese) • Blogs: 10 (2 English; 8 Portuguese) • Total: 50 (21 English; 29 Portuguese) All publications reviewed can be found on the SPEED website at http://www.speed￾program.com/ourwork/all-publications. Findings and Recommendations The Evaluation Team concluded that the SPEED publications and its website are generally effective based on the criteria assessed that were identified in the inception report. With regard to publication relevance to SPEED’s target audience and activity indicators, the Evaluation Team could not make a confident assessment due to inconsistencies found in the data and a lack of stated objectives related to publication relevance. A comparison was attempted using data found in the Project Monitoring Plan (PMP) worksheets between FY 2013 quarters 3 and 4 (Annex 17: Table 2B) related to assignment of FTF Type to initiatives (policies, regulations and administrative procedures), and use of a different classification system to categorize publications by topic. This comparison found that publications generally covered the same topics as SPEED initiatives, however, there was a disparity in the proportion of focus devoted to each topic. Because it is not known if SPEED intended its publication focus by sector to proportionately mirror its activity focus by sector, there was no basis for comparison with regard to effectiveness concerning relevance. Specific inconsistencies cited refer to data from the SPEED Performance Monitoring Plan (PMP) worksheet for FY 2013 Q3 (Annex 17: Table 1) that indicates that 81% of all policies, regulations and administrative procedures initiated relate to agriculture. Annex 17: Table 2B then shows the drastic change in categorization of initiatives in FY 2013 Q4, where only 23% of initiations are assigned to agriculture. In comparison, only 7% of SPEED’s publications directly address the agriculture sector, as shown in Annex 17: Table 2A, which contains statistics aggregated from the sample of SPEED publications reviewed by the Evaluation Team. The Team would have liked to include all SPEED activities in the comparison, however, a list of SPEED activities (seminars, workshops, forums, trainings, retreats, trips, etc.) categorized by sector or program indicator was not made available to the Evaluation Team for comparison. Consequently, a comparison of these activities and publications could not be made by sector. Regardless, SPEED should consider reconciling the differences discovered in assignment and/or categorization of its initiatives (policies, regulations, administrative procedures) and activities (seminars, workshops, trainings, etc.) versus publications on the website to make future analysis more effective. 46 Publications were found to be very effective when assessed for quality. This finding is supported by the fact that stakeholders overwhelmingly deemed the quality of publications to be very good, so quality of publications should not be an area of concern for SPEED. Regarding awareness and use of publications on the SPEED website, it is difficult to assess the effectiveness of this metric without having Specific, Measurable, Achievable, Relevant and Time-bound (SMART) target objectives for comparison. The team’s primary recommendation is for SPEED to firmly establish SMART objectives such as target numbers for levels of user awareness and use. Further, a deeper insight should be developed into how SPEED envisions use of its publications to produce or improve outcomes for its stakeholders. A well-defined vision for use and outcomes will improve topic selection and presentation of publications to the target audience. Findings from stakeholders suggest that awareness and use of SPEED publications among partners and direct beneficiaries is high. SPEED should focus on increasing awareness and use of publications among non-beneficiaries. Market segmentation should be used to identify and target non-beneficiaries that stand the most to gain from awareness and use of SPEED publications in line with desired program outcomes. Additionally, SPEED should take advantage of partner and direct beneficiary resources and cooperation to help develop a promotional strategy and implement a plan to increase awareness and use among targeted non-beneficiaries. Along these lines, it will be important to ensure partner and direct beneficiary buy-in in order to effectively promote the publications on the site. Finally, hiring a communication specialist to lead and support the efforts mentioned above was a good decision. With regard to language type, 10% of stakeholders prefer that the site offer all content and publications in Portuguese. Considering that Portuguese is the official language of Mozambique and approximately 50% of the population five years of age and older speaks Portuguese6 , ensuring that all web content and publications are in Portuguese would help to increase stakeholder accessibility, awareness and use of the publications on the site. Though budgetary and labor constraints present a challenge to maintaining a bilingual site, it is recommended that SPEED seek solutions through innovation and Partner and Direct Beneficiary cooperation to overcome this obstacle. Stakeholder collaboration related to publications on the website is an area of concern for the evaluation team. Survey method findings demonstrated that objectives and practices related to the selection of topics for publications and the posting of publications on the website were unclear. Information and practices as described by SPEED staff conflicted with perceptions of partners and direct beneficiaries. This seeming weakness presents an opportunity for improvement, in that improved collaboration with stakeholders would likely lead to increased traffic levels on the website and increased use of its publications. To improve stakeholder collaboration with regard to publications, SPEED needs to clarify its stakeholder collaboration intentions, messaging and practices internally. Then it should voice them consistently and loudly to stakeholders. It is recommended that SPEED encourage submission of topic ideas and input related to publications through conspicuous advertisement and invitation on its website. Currently, there is no mention or active call for stakeholder collaboration related to publications on the site. Nor is the posting of draft 6 Mozambique Data Portal; 2007 Census; http://www.ine.gov.mz/en 47 documents intended to elicit feedback from stakeholders outlined or explained on the website. With regard to sustainability, SPEED staff were uncertain and unclear about a plan to sustain the publication website through the gap period until the successor to SPEED emerges. Consequently, all parties responsible need to formulate a plan to ensure sustainability. Once formulated, the plan should be communicated clearly internally, and then the plan should be communicated to SPEED stakeholders. Many stakeholders are concerned and uninformed about the impending closure of SPEED and how it will affect them, including their access to publications on the website. The closing of SPEED also presents a great opportunity to build capacity in a partner or direct beneficiary organization and improve overall program sustainability by collaborating with a partner to keep the publication site up and running. The successor program to SPEED stands to benefit greatly from a well-managed publication site gap transition. Concerning gender, minimal coverage and consideration has been given to this topic by SPEED publications. In fact, of all the publications reviewed, only “Report 020 - Mozambique Civil Aviation Liberalization” and “Basic Food Basket & Transport Subsidies: What's the Future?” addressed gender concerns. In the latter publication, a program for vulnerable women is mentioned. Further, there is a general lack of understanding surrounding gender considerations in the stakeholder community. Thus, there is a great opportunity to include gender considerations in all aspects of speed activities and advocacy going forward. To this end, a concerted effort should be made to promote a dialog with and between stakeholders in order to raise awareness about gender and to empower members of the community to begin sharing their own experiences and advocating on behalf of gender considerations. Technically, SPEED publications received very positive valuations. As a result, there were few recommendations for improvement in this area. However, with regard to the use of social media, SPEED should take advantage of potential synergies between the publication site and its social media networks to increase traffic on its site and the use of its publications. It is important that promotional efforts related to the publication site and social media networks be coordinated in order to optimize effectiveness. Finally, the evaluation team tracked minor glitches and errors found on the website which it has explained in detail in the literature review and inventoried under ‘Comments’ in the ‘complete program inventory’ and ‘table of publications reviewed’ spreadsheets that can be found as annexes to the full literature review. It is recommended that these glitches be fixed and that the site be maintained at a high standard in order to ensure a positive user experience for stakeholders. 48 5.0 Additional Recommendations The following recommendations are presented for USAID/Mozambique to consider for the remaining period of SPEED and for follow-on programming as a result of interests expressed by USAID outside of the evaluation scope of work: a. Demand Driven versus Results-Based When the mission considers subsequent policy-focused programs after SPEED, the evaluation team recommends the format to be demand driven. We make this recommendation for the following reasons: • SPEED and its predecessor programs have successfully facilitated policy change in Mozambique • This genre of programs utilized a demand driven format that created responsiveness, flexibility and results that were not originally envisioned when designed • This legacy can be optimally leveraged and extended by continuing the same format • The Mission staff has accumulated experience and expertise in managing demand driven, policy oriented program. • Mozambican ministries and stakeholders are familiar with the demand driven approach of the Mission and US Government The evaluation team believes that SPEED has essentially become a model, or case study, worthy of further examination for future policy-focused programs in the region. b. Public versus Private Sector The new SPEED activity needs to strike a balance between its focus on public sector leaders influencing policy decisions and private sector stakeholders whose interests need to be reconciled and who need assistance in advocating for change and in seeing that past policy changes are truly implemented. SPEED should continue to focus on political leaders in the best position to advance policy and regulatory changes, most of them located in the executive branch of government (Ministries and the Presidency). Additional efforts with Parliament should focus on leaders in a position to help enact measures requiring parliamentary approval rather than broadly dispersed among members. Support to private sector business leaders and for consultations both in Maputo and elsewhere in the country should continue in order to build the constituency for change. Proportionally, SPEED impacted more public policy issues than private sector oriented issues. This maybe a misleading fact since a number of those public sectors interventions will directly or indirectly end up benefiting private businesses. Logically, while many readers will seek a quantitative balance or formula between private and public impact, it is clear that both The SPEED Model • Demand driven • Experienced Mission Staff • Growing National Awareness • Proven Policy Results 49 sectors have been supported and advanced by SPEED. Thus, subsequent programmatic activities should closely examine opportunities to leverage SPEED’s accomplishments in both sectors because ultimately many, if not most, public policies end up impacting businesses in some way. Measuring impact in the public policy arena is difficult for a number of reasons but when this analysis is combined with an assessment of the impact in the private sector, measuring the combined impact becomes easier. Changing public policies that do not ultimately improve the opportunities for businesses to operate more effectively and profitably will impede development and growth of the private sector. More specifically, micro, small and medium sized enterprises will stall because they are unable to grow their respective businesses. The evaluation team believes that SPEED’s successor activity should continue its focus on policy issues that impact MSMEs because ultimately these entities will prove a sustainable impact of future programmatic efforts. c. National versus Sub-National Policy in Mozambique continues to be very centralized and major decisions are still made largely in Maputo. SPEED’s successor activity and its partners will certainly understand this situation and will want to work to change it to allow businesses outside of Maputo greater representation in national associations carrying out advocacy on their behalf and a greater ability to carry out business operations without having to make multiple trips to Maputo. However, SPEED’s follow-on activity and USAID need also to understand the cost implications of changes needed in the activity in order to make it more geographically inclusive. Despite SPEED’s work on the Open Skies initiative, travel in Mozambique continues to be costly; land transport is increasingly difficult and dangerous because of the resurgence of conflict. To include business in places like Beira, Quelimane, Nampula/Nacala and Cabo Delgado in policy consultations and training will require substantially more resources and commitment of staff time than was available to SPEED. d. Careful Consideration of Inter and Intra-Sector Conflicts Future SPEED activities need to take into account conflicts between sectors (agriculture vs. mining and petroleum) and within sectors (large-scale commercial agribusiness vs. small holder farming) as development corridors are implemented and extractive industry projects are negotiated and brought on-stream. The effect of the resource boom on Mozambique’s overall competitiveness and in particular of agricultural sector project needs further work beyond that already done by SPEED on the Dutch Disease and on means for acquiring land for agribusiness enterprises. SPEED should make certain that policy and regulatory reforms it promotes lead to inclusive growth for the country and does not lead to growth in one sector or to parts of one sector at the expense of other sectors or other parts of the same sector. e. Recommendations for Follow-On Activity e.1. Extension of SPEED to February 2015 The evaluation team strongly supports this extension of SPEED to February 2015, and the decision to increase the burn rate so that much-needed policy research and other SPEED interventions can continue at a high level at this critical time. Usually, activities generally trail off at the end of projects: their directors attempt to finish activities which have been 50 started, limit new activities and face limitation as staff leaves to take new long-term jobs. SPEED will attempt to buck this trend. USAID needs to make it clear that its support for policy work is undiminished by the ending of the SPEED activity. This will give staff the confidence to stay on and join teams brought together to bid on the new activity. e.2. Potential Gap Period At this writing (March 2014), only 11 months remain before SPEED ends. In view of the work required to get a new RFP out, for proposals to be prepared, analyzed, a new contract to be awarded and a team mobilized to start the work, it is possible that there may be a gap between end of SPEED and the start of the new activity could emerge. It would be important 1) to minimize the gap period, 2) to assure a similar level of funding during the gap period to the time leading up to it and to the funding planned for the new activity and 3) to maintain similar staffing during the gap period (COP, technical and admin staff, etc.) as during the prior and future activity. Whatever mechanisms can be found (funding from other projects, interim funding from other USAID sources, etc.) should be tapped to assure that activities continue seamlessly during the gap period. e.3. Future Activity The future policy reform activity following in the footsteps of SPEED should continue to be demand-driven. Ideally, funding should be from sources without earmarks (i.e., “any policy use”). This being said, SPEED has managed so far with an increasing proportion of its funding coming from earmarked sources and has still managed to enact an impressive agenda of policy changes as well as providing support for business associations to grow and thrive. Suppliers of such funds need to be made aware that even changes occurring outside their sectors are likely to have positive repercussions within them and to facilitate future change when conditions in their particular sector are more conducive to change. e.4. Duration of the New Activity The new SPEED policy activity would be best served by the longest possible duration of the contract period because of the long-term nature of policy reform. Building rapport with policy-makers and then taking proposed changes through the various stages from initial identification through analysis to implementation takes time. Providing the activity with a four-year timeframe for the contract and the possibility of an extension year would be desirable. An even longer period could also be justified. e.5. Policy work at Inauguration Time The SPEED activity should take advantage of the change in Government to move its policy agenda forward. An excellent opportunity will emerge when the new Government takes power to provide a policy work agenda for those taking their seats in the Cabinet. Key ministries include: agriculture, trade, finance and tourism. The policy activity needs to be ready to meet this challenge and to take advantage of this opportunity. Even during the electoral period, business associations need to focus their advocacy work on politicians campaigning for election and to assure that their positions on policies important to business 51 and the development of the Mozambican economy are conducive to growth. Thus the election period is not a write-off but should be more correctly viewed as an opportunity to take advantage of. SPEED should take advantage of the opportunities which the political process offers for bringing policies conducive to a better business environment forward e.7. Follow-up on Past Policy Changes In its final months, SPEED and subsequently the new activity should review policies which have been changed to make sure that backsliding has not occurred or that implementation has really occurred as expected. It could very well be that policy changes which appeared at the time to have been done and over with could require some additional work (writing and getting an implementing rule enacted, fine-tuning text of an approved law to work out wrinkles which have emerged during implementation, etc.). Such follow-up work is high￾value and should not be undervalued by but rather supported by USAID to get the best possible value from the work SPEED has already accomplished. For example, a USAID agricultural policy reform project in Egypt showed that very small changes in regulations or rule often were the key to making much larger pieces of policy actually work for business. Other lessons learned from similar programs in other countries can be found in Annex 16. e.8. Marketing Policy Change SPEED has been successful and has garnered positive opinions of most of those who know it. The name recognition is valuable. It is worth considering if the follow-on activity should be called “SPEED.2” or “SPEED II” or some name that conjures up the image of what SPEED has done in the past. USAID should carefully consider what the best name for the new activity should be in addition to its content. SPEED should take advantage of communication channels (such as TV, radio and print media) to allow the general public to be better informed about policy changes which SPEED has been instrumental in helping to bring about. Such a communication will help the general public to be more receptive to additional policy change than might otherwise be the case. e.9. Location of the New Activity The evaluation team believes that having SPEED as an independent activity not housed within any specific ministry gave it greater ability to work over a broad front and to move forward policy changes where the opportunity arose irrespective of what Ministry was responsible for the specific policy area. Likewise, it allowed the activity more independence in terms of the institutions supported and the type of support provided. The new SPEED activity should be an independent one, operating in its own right and not housed within any specific Ministry. It may, however, be advantageous to have some staff that is embedded in specific Ministries where principal policy changes are likely to originate or which will have to give their approval for changes to be enacted or implemented. Increased SPEED geographical coverage outside of Maputo would be desirable, despite the fact that policy-making remains an activity which is very centralized. Expansion of coverage 52 would allow business leaders and regional government officials to provide input and to take part in the consultative process going into defining new policies supported by SPEED. e.10. Limitations on Activities Undertaken Even if it is not a major part of the new activity, SPEED’s ability to provide grants is worth retaining. Its retention will simply give the new activity one more tool to address its objectives and to retain the flexibility it needs to maximize its chances of successfully meeting challenges and taking advantage of opportunities. Grants can be particularly useful to smaller organizations, particularly to organizations located outside of the Maputo area, whose access to external resources is extremely limited. The extent to which grants are used or are made to organizations outside Maputo will depend in part on the level of funding available for the new activity. e.11. Recommendations Derived from Key Informant Interviews Key informants felt that the follow-on activity can address the following areas in the future: Key Informant Suggestions • Reform in infrastructure in the area of banking • Disseminate information on legislation and providing institutional support. • Prepare agents in terms of linkages to markets • There is a great need for education/awareness of the general population on basic banking practices, specifically with access to credit. This is not supposed to be a key responsibility of a bank to educate. • Reform of new policies approved by the Central Bank at all levels • SPEED should directly support associations because CTA does not reach its ultimate beneficiaries. 53 ANNEXES 54 ANNEX 1: Quantitative and Qualitative Data Collection Design Matrices Qualitative Data Collection Design Matrix Evaluation Questions Evaluation Sub Questions Illustrative Indicators Data Source Collection Methods Sampling Section Criteria Data Analysis Method Question Area 1: Effectiveness 1. How and to what extent has SPEEDs programming been effective in the following areas, and what were the contributing factors: a. Building Local capacity in policy dialog and policy implementation. b. Balancing between demand-driven approach (external USAID stakeholders) and other task order requirements (internal USAID stakeholders). c. How can SPEED indicators be improved d. How effective has SPEED been at the 1. Demand-driven (external USAID stakeholders) vs. task order (internal USAID stakeholders) requests addressed by SPEED. 2. Current SPEED indicators are SMART ● Specific ● Measurable ● Achievable ● Relevant ● Time-bound 3. Policies amended, developed or blocked due to SPEED involvement. 4. Positive vs. negative responses of training participants to how the knowledge/skills learned 1. Key Informant Interviews 2. Desk Review of SPEED internal/external documents 3. Review of relevant GoM economic data sources. 4. WB Doing Business ranking and Transparency International’s Corruption Index 5. One Workshop in Maputo. 6. Three Focus Groups with the implementing partner and stakeholders (1 Beira, 2 Maputo) 7. Training participant lists 1. Representative of SPEED beneficiary for each sector of focus in private and public sectors. 2. Representative of non-SPEED beneficiaries for comparison purposes. 3. Official documents submitted by SPEED to USAID. Mixed Method Parallel Qualitative and Quantitative 55 Macroeconomic level and sector specific work? at training have been applied to their current work. Question Area 2: Stakeholder Collaboration 1. How and to what extent has SPEED engaged, collaborated with, and leveraged stakeholders? a. How do stakeholders view SPEED’s work? b. Is SPEED engaging with appropriate key Government stakeholders 1. Financial and human resource amounts leveraged with partners. 2. Networks and partnerships built and strengthened as a result of SPEED activities. 3. SPEED stakeholder responses toward SPEED’s level and relevance of engagement. 4. Type of Government Ministries SPEED has engaged with vs. key development policy focus areas. 1. Key Informant Interviews 2. Desk Review of SPEED internal/external documents 3. One Workshop in Maputo. 4. Two Focus Groups with the implementing partner and stakeholders (1 Beira, 2 Maputo) 1. Representative of SPEED beneficiary for each sector of focus in private and public sectors. 2. All associations and formal organized groups that SPEED has engaged. 3. SPEED documents that show activity results that would encourage the building of partnerships and networking. Mixed Method Parallel Qualitative and Quantitative Question Area 3: Sustainability 1. What sustainable practices has SPEED implemented? a. How Sustainable is SPEED’s work on Strengthening 1. Processes/mechanisms established in support of policy lobbying efforts. 2. Training programs and/or methodologies institutionalized within 1. Key Informant Interviews 2. Desk Review of SPEED internal/external documents 3. One Workshop in Maputo. 1. Representative of SPEED beneficiary for each sector of focus in private and public sectors. 2. Representative of Mixed Method Parallel Qualitative and Quantitative 56 institutional capacity of private and government institutions, especially CTA. b. To what extent will the policy reforms; processes create long lasting economic benefits? SPEED beneficiary organizations. 3. Positions, programs or equipment that will be maintained after SPEED ends. 4. Level of capacity built within SPEED stakeholders. 4. Two Focus Groups with the implementing partner and stakeholders (1 Beira, 2 Maputo) 5. List of equipment donated through SPEED. non-SPEED beneficiaries for comparison purposes. 3. All official documents submitted by SPEED to USAID that USAID shares. Question Area 4: Gender 1. How has SPEED integrated Gender Issues with its work efforts? a. How has SPEED addressed the different Gender roles and needs when selecting policy areas, conducting policy analysis or advocating policy change? And how could any changes or improvements be made in this area? b. Are there particular policy areas that have to date or could in the 1. Number of hired women on the project Level of position; 1.a) Level of participation in decision making and management. 2. Level of gender equitable language in policy papers. 3. Equitable number of male/female participants in SPEED supported activities. 1. Key Informant Interviews 2. Desk Review of SPEED internal/external documents 3. One Workshop in Maputo. 4. Two Focus Groups with the implementing partner and stakeholders (1 Beira, 1 Maputo) 1. Women in Policy focused organizations. 2. Representative of SPEED beneficiaries for each sector of focus in private and public sectors that can speak to issues of gender. 3. All official documents submitted by SPEED to USAID that USAID shares. Mixed Method Parallel Qualitative and Quantitative 57 future especially benefit women traders or entrepreneurs? Quantitative Data Collection Design Matrix Evaluation Questions Evaluation Sub Questions Illustrative Indicators Data Source Collection Methods Sampling Section Criteria Data Analysis Method Initial Activity Design SOW, Strategic Results Framework 1. Number of USAID offices, GOM, CSOs and private sector entities involved in identification and original design and still involved in implementation 2. Targeted results to be achieved and compared 1. Interviews with internal USAID and SPEED staff. 2. Interviews with SPEED stakeholders. 3. Desk Review of documents relevant to design phase of SPEED 1. Interviews with selected key informants from GOM, CSOs and private sector entities; also with implementing companies involved in design 2. Documents that represent time period through to approved SPEED work plan Mixed Method Parallel Quantitative Management and Tracking of SPEED SITT Mechanisms Analysis and comparison of quantitative results 1. Consistency in defining Tracking tools 2. Consistency in targets and results throughout Tracking tools 1. Desk review Tracking tools 2. Interviews with Program Office and COR 3. Interviews with Implementer Teams 4. Review of Results Framework 1. Document is an internal SPEED tracking tool Mixed Method Parallel Qualitative and Quantitative Outcomes 1. GOM, CSOs and private sector entities sectors and institutions 1. Policies amended, developed or blocked due to SPEED’s involvement. 1. Desk review of tracking tools 1. Organizations received grants or other direct/indirect support Mixed Method Parallel Quantitative 58 assisted through SPEED Activity 2. List of institutions by sector, type and value of grant or other assistance 2. Key informant interviews 3. Specific sector (GOM) annual reports and databases 2. SPEED and key stakeholder reporting documents related to policy reform Literature Review 1. Notes 2. Reports 3. Presentations 4. Blogs 1. Traffic flow to publications section of SPEED website. 2. Positive responses from SPEED stakeholders to value of publications. 3. Awareness level of non￾SPEED beneficiaries about publications. 4. Level of correlation between publication and SPEED initiative at same time period. 1. Publications uploaded on SPEED website 1. Publications on SPEED website. Mixed Method Parallel Quantitative 59 Annex 2: Evaluation Participants Evaluation Participants disaggregated by Informant Type, Sector and Sex (49 Total) Category Number Percent of Total Informant Type Direct Beneficiary 22 45% Partner 9 18% Non-Beneficiary 10 21% USAID 5 10% SPEED Staff 3 6% Total 49 100% Sector GoM 19 39% Gov Int 3 6% USG 8 16% Private 13 27% Non-Profit 5 10% Academia 1 2% Total 49 100% Gender Male 35 71% Female 14 29% Total 49 100% 60 Annex 3: Key Informant Participants KI Participants disaggregated by Informant Type, Sector and Sex (30 total) Category Number Percent of Total Informant Type Direct Beneficiary 12 40% Partner 7 23% Non-Beneficiary 3 10% USAID 5 17% SPEED Staff 3 10% Total 30 100% Sector GoM 11 37% Gov Int 2 7% USG 8 26% Private 5 17% Non-Profit 3 10% Academia 1 3% Total 30 100% Gender Male 23 77% Female 7 23% Total 30 100% 61 ANNEX 4: Focus Group Participants Focus Group Participants disaggregated by Informant Type, Sector and Sex (8 total) Category Number Percent of Total Informant Type Direct Beneficiary 3 38% Partner 1 13% Non-Beneficiary 4 50% USAID 0 0% SPEED Staff 0 0% Total 8 100% Sector GoM 4 50% Gov Int 0 0% USG 0 0% Private 4 50% Non-Profit 0 0% Academia 0 0% Total 8 100% Gender Male 5 62.5% Female 3 37.5% Total 8 62 ANNEX 5: Workshop Participants Workshop Participants (Maputo) disaggregated by Informant Type, Sector and Sex (11 total) Category Number Percent of Total Informant Type Direct Beneficiary 7 64% Partner 1 9% Non-Beneficiary 3 27% USAID 0 0% SPEED Staff 0 0% Total 11 100% Sector GoM 4 36.5% Gov Int 1 9% USG 0 0% Private 4 36.5% Non-Profit 2 18% Academia 0 0% Total 11 100% Gender Male 7 64% Female 4 36% 63 ANNEX 6: Summary of SPEED Capacity-Building Activities Capacity Building Activities Year Event Private Sector / CSO Gov’t Total Org’s M F Sub￾total Org’s M F Sub￾total M F Sub￾total 2011 31 20 959 298 1257 8 159 38 197 1118 336 1454 2012 28 15 491 146 637 14 888 180 1068 1379 326 1705 2013 24 10 1632 271 1903 1632 271 3159 Total 83 45 3082 715 3797 22 1047 218 1265 4129 933 5062 Source: SPEED Annual Reports 64 ANNEX 7: Mozambican Macroeconomic Indicators 2010 – 2014 From IMF Country Report No.14/20: Mozambique: January 2014 Act. Act. Est. CR 13/200 CR 13/200 2010 2011 2012 2013 2014 Real GDP Growth (%) 7.1 7.3 7.2 7 7.1 Average CPI Inflation (%) 10.4 2.1 5.5 5.6 Total Revenue (% GDP) 19.6 20.8 23.3 24.5 23.5 Total Expenditure & Net Lending (% GDP) 32.9 33.7 32.6 36.3 35.5 Fiscal Deficit before grants (% GDP) -13.2 -13.1 -9.5 -11.8 -12 Overall Account Balance (Mil USD)* 608 323 236 100 199 Current Account Balance (Mil USD) -1113 -3059 -6426 -5854 -7364 Capital / Financial Account Balance (Mil USD) 1663 3364 6748 5954 7563 *Significant Trade deficit, but influx of FDI under Capital and financial account balance offsets Trade defict and makes Overall account balance positive. 65 ANNEX 8: Foreign Direct Investment in Mozambique by Sector 66 ANNEX 9: PMP Summary Table Quarter Three Fiscal Year 2013 PMP Summary Table – 2013 Quarter Three ATB Indicator Unit Disaggregation 2011 Target 2011 Actual 2012 Target 2012 Actual 2013 target 2013 Actual 2.Jobs Created # Male not set 16,000 10,000 Female not set 5,000 PPR Indicator Unit Disaggregation 2011 Target 2011 Actual 2012 Target 2012 Actual 2013 Target 4.2.2-5 Number of participants in USG supported trade and investment capacity building trainings # Men not set 1,118 1,200 dropped # Women not set 276 400 dropped PMP Summary Table – 2013 Quarter 4 ATB Indicator Unit Disaggregatio n 2011 Targe t 2011 Actua l 2012 Target 2012 Actual 2013 target 2013 Actual 2014 target 2.Jobs Created # Male not set 16,000 10,000 1,817 5,000 2,627 5,000 Female not set 5,000 PPR Indicator Unit Disaggregatio n 2011 Targe t 2011 Actua l 2012 Target 2012 Actual 2013 Target 2013 Actual 2014 Target 4.2.2-5Number of participants in USG supported trade and investment capacity building trainings # Men not set 1,118 1,200 dropped dropped dropped dropped # Women not set 276 400 67 ANNEX 10: Indicator Status USAID Results Framework Indicator Identification SPEED Performance Indicators Status Comments Assistance Objective: Inclusive growth of targeted economic sectors 2. Jobs Created Inconsistent Indicator only tracked in 2012. Data is missing for 2011 and 2013. Intermediate Result 2: Enabling Environment Improved 2.1. Reduction (in $) in costs resulting from reform Inconsistent Indicator tracked in 2011 with no target and targeted in 2012 with no tracking. 2.2. World Bank Doing Business Ranking Inconsistent Indicator does not appear 2.1.1. Status of targeted reforms following an improved policy process Consistent Indicator was tracked in 2011 with no target and tracked in 2012 and 2013 with a target Sub-Intermediate Result 2.1: Capacity for policy advocacy strengthened 2.1.2. Number of policy advocacy CSO’s strengthened Consistent Indicator was tracked in 2011 with no target and tracked in 2012 and 2013 with a target 2.1.3. Number of policies advanced through CSO’s Consistent Indicator was tracked in 2011 with no target and tracked in 2012 and 2013 with a target Sub-Intermediate Result 2.2: Implementation of policy enhanced 2.2.3. Number of policies the GoM reforms with USG support Consistent Indicator was tracked in 2011 with no target and tracked in 2012 and 2013 with a target 4.8.2-28: Number of laws, policies, strategies, plans, agreements or regulations addressing climate change and/or biodiversity conservation officially proposed, adopted, or Inconsistent Indicator was added and tracked in 2012 with a target and in 2013 with a target but without actual results reported. 68 implemented as a result of USG assistance Other Indicators 4.2.2-10: Person hours of training completed in trade and investment capacity building supported by USG assistance Inconsistent Indicator was added and tracked in 2012 and in 2013 with a target but without actual results reported. 4.2.2-5: Number of participants in USG supported trade and investment capacity building Inconsistent Indicator was dropped from 2012 onward 4.8.1-7: Number of policies promoting sustainable natural resource management and conservation that are implemented as a result of USG assistance Inconsistent Indicator was dropped from 2012 onward 69 ANNEX 11: SPEED Policy Stage Achieved Per Year Analyzed Presented for consultation Presented for legislation Passed Implementation begun Stage 1 Stage 2 Stage 3 Stage 4 Stage 5 TOTAL 2011 1 2 4 5 2 TOTAL 2012 16 18 8 5 6 SUBTOTAL per 2013 Quarter: 2013q1 0 0 1 1 2 2013q2 4 2 1 3 2 2013q3 3 4 3 2 5 2013q4 0 0 0 0 0 TOTAL 2013 7 6 5 6 9 Source: PMP Quarter Three Fiscal Year 2013 Stages for the completion of work on each policy, regulation or procedure are defined as follows: Stage 1 Analyzed: Completed review of existing or proposed policy regulation or administrative procedure and/or proposal of new policy, regulations or administrative procedures in the reporting period. Stage 2 Drafted and presented for public/stakeholder consultation: Completed public debate and/or consultation with stakeholders on the proposed new or revised policy, regulation or administrative procedure in the reporting period. Stage 3 Presented for legislation/decree: Revised policy/regulation/administrative procedure was presented for legislation/decree to improve private-sector led economic growth in the reporting period. Stage 4 Passed/approved: Official approval (legislation/decree) of new or revised policy, regulation or administrative procedure by relevant authority in the reporting period. Stage 5 Passed for which implementation has begun: Implementation of new or revised policy, regulation or administrative procedure by relevant authority in the reporting period. 70 ANNEX 12: List of Institutions by Sector, Type, and Form of Assistance # Institution Sector Type Value of Grant / Form of Assistance 1 Confederation of Business Association (CTA) Private CSO Grant - $16,000 2 Centra Terra Viva Private CSO Capacity Building 3 AgriFuturo Private Donor Capacity Building 3 Sal & Caldeira Private CSO Capacity Building 4 Ministry of Industry and Commerce (MIC) Private CSO Capacity Building 5 Obsevatorio Mocambicano de Governacao Private CSO Capacity Building 6 Association of Commerce and Industry (ACIS) Private CSO Capacity Building 7 Tax Authority Public Government Institution Capacity Building 8 Tiri Pamodzi Donor Foundation Technical Assistance, Capacity Building, Grant – IT Equipment $29,726.23 10 Maputo Muncipality Public Government Institution In-Kind Grant – IT Equipment $79,783.45 USD 11 Institute of Directors (IOD) Grant - $44,412.00 USD 12 Attorney General’s Office Public Government Institution Technical Assistance 13 Quelimane City Mayor’s Office Public Government Institution Technical Assistance 14 Arco Norte Tourism company Private Business Capacity Building 15 Department of Urbanization Public Government Institution In-Kind Grant – IT Equipment, amount unspecified 16 The Ministry of Agriculture Public Government Institution Technical Assistance 17 Ministry of Tourism Public Government Institution Technical Assistance 18 Ministry of Mineral Resources Public Government Institution Technical Assistance 19 Ministry of Justice Public Government Technical 71 Institution Assistance 20 Ministry of Transport and Communication Public Government Institution Technical Assistance 21 Treasury Public Government Institution Technical Assistance 22 Ministry of Finance Public Government Institution Technical Assistance 23 Tax Authority Public Government Institution Technical Assistance 24 Ministry of Industry and Commerce Public Government Institution Technical Assistance 25 Attorney General’s Office Public Government Institution Technical Assistance 26 Ministry of Tourism Public Government Institution Technical Assistance 27 Ministry of Mineral Resources Public Government Institution Technical Assistance 28 Institute for Promotion of SMEs (IPEME) Public Government Institution Capacity Building, Grant – amount unspecified 29 MISA Private CSO Capacity Building 30 Manica PG Public Government Institution Capacity Building 31 AMECON Private CSO Capacity Building 32 CMCM Public Government Institution Capacity Building 33 GIRBI / MIC Public Government Institution Capacity Building 34 CMCQ Public Government Institution Capacity Building 35 Institute for Promotion of Import/Export (IPEX) Public Government Institution Capacity Building 36 International Union for Conservation of Nature (IUCN) Private CSO Capacity Building 37 Ministry of Agriculture (MINAG) Public Government Institution Capacity Building 38 Business Registration Agency Public Government Institution In -Kind Grant – IT Equipment 39 Center for Arbitration, Conciliation, and Mediation (CACM) Private CSO Grant - $100,000 USD in 2013, $92,409.59 USD in 2012 40 CEMO Private CSO Capacity Building 41 Women’s Business Private CSO Capacity Building 72 Association (FEMME) 42 National Association of Agricultural Associations (FENAGRI) Private CSO Capacity Building 43 Economist National Association (AMECOM) Private CSO Capacity Building 73 ANNEX 13: Questionnaires Submitted separately 74 ANNEX 14: Main Database Submitted separately 75 ANNEX 15: Organizations Participating in Data Collection AMECON (Economist National Association) Agricultural Promotion Central (CEPAGRI) Alliance for Green Revolution in Africa (AGRA/SSTP) and former MoAG Arbitration, Conciliation and Mediation Center of Mozambique (CACM) Association of Commerce and Industry (ACIS) Association of National Youth Entrepreneurs (ANJE) AT/Customs Attorney General Office Bank ABC Beira Links Beira Provinical Court CE Confederation of Business Association (CTA) FAN (Danish Embassy) Gil da Conceicao Bires (GAZEDA) Institute for Promotion of Import/Export (IPEX) Institute for Promotion of SMEs (IPEME) Institute of Directors (IOD) Instituto Nacional das Comunicacoes de Mocambique (INCM) Maputo Municipality (CMCM) Maputo Municipality (CMCM) Media Institute of Southern Africa (MISA) Michigan State University Ministry of Agriculture Ministry of Industry and Commerce Ministry of Justice Ministry of Tourism (MITUR) Mozambique Celular (mCel) National Directorate of Land and Forests (DNTF) NORAD Private Sector Working Group (PSWG), GIZ Member Provincial Court SCA Consultants Sofala Governor SPEED TIri Pamodzi USAID-Agriculture, Trade and Business (ATB) USAID-Democracy and Governance (DG) USAID-Feed The Future (FTF) World Bank-International Finance Corporation (IFC) 76 ANNEX 16: Lessons Learned from Other Countries Evaluation team members have experience with agricultural policy change in many countries. We have decided to limit ourselves to two countries and two projects, both financed by USAID which were found by independent evaluations to be highly successful in achieving sustainable policy changes. These projects are the Agricultural Policy Reform Project (APRP) in Egypt managed by Abt Associates and RDI and the SPEED project in Uganda managed by Chemonics. Agricultural Policy Reform Project (APRP) – Egypt The Agricultural Policy Reform Project (APRP) project operated in Egypt from 1997 to 2001 and was seen as having achieved major accomplishments in agricultural policy reform. Many of these have been due in large measure to the project’s approach of creating consensus on a particular policy reform and then taking the essential steps toward implementation which make broad policies into realities improving the lives of individual stakeholders. The program not only covered policies but got involved in implementing regulations and procedures in close collaboration with producers and their associations who were directly and in some cases affected by seemingly minor rules. Its consensus building activities are similar to SPEEDs consultative process for its only work on policy change. APRP also understood the important of providing accurate data for businessmen and women to use in order to make reasonable decisions on their agricultural-related businesses. APRP has a much more benchmarked approach than does SPEED but also took advantage of opportunities as they emerged, particularly after main constraints to doing business which the project targeted had met. Like SPEED, the project earned respect by Government which sought APRP’s assistance in dealing with policy and procedural issues as they emerged. In the final years of the project, attention was given to avoid backsliding on policy and procedural changes which had already been made with the help of the APRP project. Support for Private Enterprise Expansion and Development in Uganda (SPEED) Support for Private Enterprise Expansion and Development in Uganda (SPEED) project aimed to fill the gap between SMEs and their access to financial services, the gap separating microfinance institutions (MFIs) from the commercial banking system, the gap between the business development service requirements of target clients (MSMEs, MFIs and financial institutions) and the gaps in the policy, legal, and regulatory environment. So it was a project which provided BDS support to businesses and financial institutions and to assure the finance that they needed while also supporting policy, legal and regulatory change needed to improve the business environment. Its work with enterprises illuminated its policy work, providing it the means to identify needed interventions and to take advantage of opportunities to improve the business environment as they emerged and in relation to the needs of assisted businesses and financial institutions. SPEED also provides BDS and other types of support which was critical to the development and growing capacity of private sector associations and foundations. SPEED Uganda participated in commercial justice reform including mediation and arbitration. It provided assistance to improving both land and companies registries. It helped with improvements to the leasing law and did an assessment of tax treatment of leasing businesses. It also supported improved data collection on businesses for the Ministry of Finance. These actions taken in consultation with private sector businesses and their associations and organizations and with the support of Government led to major improvements in the business environment in Uganda and have had long-lasting effects. In Mozambique, the question has arisen over whether or not to include policy functions in 77 projects designed mainly to support various types of business. This approach worked for SPEED Uganda, however, it took place when the country was at a lower stage of development of private sector business than is the case in Mozambique. Nevertheless its experience bears looking at even if USAID/Mozambique chooses to continue a different approach. 78 ANNEX 17: Literature Review Tables 1, 2A, and 2B Table 1: Statistics calculated from PMP SPEED FY2013 Q3 Report Table 2A: Statistics calculated from the list of publications reviewed 79 Table 2B: PMP Worksheet Comparison – Initiative counts by FTF Type FTF Type 31012013 PMP - SPEED FY2013 Q1.xlsx PMP - SPEED FY2013 Q2.xlsx PMP SPEED FY2013 Q3.xlsx 22112013 SPEED Indicators tracking Table FY2013 Q4 final HM 28Nov SM NG Dec 04.xlsx Agricultural sector wide 33 37 48 13 Climate change adaptation/NRM 1 1 1 4 Food security/vulnerable 2 2 2 2 Macroeconomic 5 5 6 37 Other 1 1 1 0 Outputs 1 1 1 1 Total # of policies, regulations, admin prodcedures acted upon 43 47 59 57 PMP Worksheet File Names 80 81