FINAL PERFORMANCE EVALUATION OF THE AZERBAIJAN COMPETITIVENESS AND TRADE (ACT) PROJECT FINAL EVALUATION REPORT [September 2013] This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Mendez, England & Associates. FINAL PERFORMANCE EVALUATION OF THE AZERBAIJAN COMPETITIVENESS AND TRADE (ACT) PROJECT Final Evaluation Report September 2013 Prepared under RFTOP No. SOL-112-13-000005 Submitted to: USAID/Azerbaijan Submitted by: Colin Maclean, Team Leader Danil Samoilenko Fagan Aghayev Ibrahim Mammadzadeh Contractor: Mendez England & Associates 4300 Montgomery Avenue, Suite 103 Bethesda, MD 20814 Tel: 301- 652 -4334 www.mendezengland.com DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government ii | Page CONTENTS EXECUTIVE SUMMARY ................................................................................................................. ii Evaluation Purpose and Evaluation Questions ................................................................................................... ii Project Background .................................................................................................................................................. ii Evaluation Design, Methods and Limitations ...................................................................................................... ii A. Meeting Targets/Reporting Progress ..................................................................................................... iv Findings ...................................................................................................................................... iv Conclusions ............................................................................................................................... iv Recommendations ..................................................................................................................... iv B. Progress in Key Intervention Areas – Results and Impacts .............................................................. v Findings ....................................................................................................................................... v Conclusions ............................................................................................................................... vi Recommendations .................................................................................................................... vii C. Evaluating the Effectiveness of Technical Assistance ......................................................................... vii Findings ................................................................................................................................................... vii Conclusions ............................................................................................................................. viii Recommendations ................................................................................................................... viii D. Evaluating the Sustainability of ACT Project ........................................................................................ ix Findings ..................................................................................................................................... ix Conclusions ............................................................................................................................... x Recommendations ..................................................................................................................... x E. Lessons learned .......................................................................................................................................... xi 1.0 EVALUATION PURPOSE & EVALUATION QUESTIONS ............................................ 1 1.1 Evaluation Purpose ...................................................................................................................................... 1 1.2 Evaluation Objectives ................................................................................................................................. 1 1.3 Evaluation Questions .................................................................................................................................. 1 2.0 PROJECT BACKGROUND ................................................................................................ 2 3.0 EVALUATION METHODS & LIMITATIONS ................................................................. 4 3.1 Evaluation Methodology ............................................................................................................................. 4 3.1.1 Quantitative Research and Analysis .................................................................................... 5 3.1.2 Qualitative Research and Analysis ....................................................................................... 5 3.2 Evaluation Limitations ................................................................................................................................. 6 4.0 FINDINGS, CONCLUSIONS & RECOMMENDATIONS ............................................... 6 iii | Page 4.1 MEETING TARGETS / REPORTING PROGRESS .............................................................................. 6 4.1.1 Question 1: How effective was the Contractor in working towards meeting USAID and GOAJ goals and objectives in ACT Project? Comment on critical gaps and project assumptions? ............................................................................................. 6 4.1.2 Question 2: Was the data collected sufficient to measure project progress and results in all three project components and in particular with respect to intermediate results? .............................................................................................................. 9 4.1.3 Question 3: Did the project design take into consideration gender issues such as gender division of labor, time use, control of resources, access to finance, etc.? .......................................................................................................................... 12 4.1.4 Question 4: Are various results reported accurate and verifiable? Have the results achieved by the project been cost-effective and sustainable? ................................... 15 4.2 KEY INTERVENTION AREAS AND RESULTS AND IMPACTS ................................................. 17 4.2.1 Question 1. Assess the progress and impacts of the value chain operations under ACT Project in terms of: i) technology transfer and the relevance of technologies transferred; ii) strengthening farming advisory and input delivery services; iii) capacity building of pot-harvest and processing facilities; and iv) linking producers with markets. ................................................................................. 17 4.2.2 Question 2. Assess the range of policy reform efforts supported, progress and the status (e.g. legislative status for each reform supported) and priority reforms to be supported in the future. .......................................................................... 22 4.2.3 Question 3: Evaluate the progress made in Azerbaijan’s accession to WTO and further support to be extended. .............................................................................. 24 4.3 EFFECTIVENESS OF TECHNICAL ASSISTANCE ........................................................................... 25 4.3.1 Question 1: How effective was STTA provided by Contractor in meeting the needs of the project clientele? Assess the impact of the STTA on the beneficiary in terms of relevance of STTA recommendations and their adoption. ................................................................................................................................ 25 4.3.2 Question 2: How effectively has ACT created awareness about the issues it worked on among the private scetor, donors, civil society and government agencies in Azerbaijan? ........................................................................................................ 28 4.4 SUSTAINABILITY OF THE ACT PROJECT ...................................................................................... 29 4.4.1 Question 1: Assess the extent of stakeholder ownership of interventions undertaken by ACT project. ............................................................................................. 29 4.4.2 Question 2: What processes, systems and management were put in place to ensure that the results and impact of ACT Project will be sustainable? ........... 31 4.4.3 Question 3: What were the obstacles to sustainability and what measures should be taken to increase sustainability in future projects? ................ 33 iv | Page 4.4.4 Question 4: Appraise the role of cost-sharing approach to Short-Term Technical Assistance in terms of creating ownership of the beneficiaries and sustainability. ......................................................................................................................... 35 5.0 LESSONS LEARNED ...................................................................................................... 38 ANNEXES A. Statement of Work B. Evaluation Work Plan C. Evaluation Design & Methodology D. Overall Monitoring and Evaluation Matrix E. List of Interviewees F. List of Documents Reviewed G. Illustrative Interviewee Questionnaires H. Summary of BDS Responses I. ACT Spending per Component through June 2013 J. ACT STTA Days by Component K. ACT Cost Breakdown of Experts and Staff L. Conflict of Interest Forms FINAL PERFORMANCE EVALUATION OF ACT i | Page ACRONYMS ACT AMCHAM ASIP Azerbaijan Competitiveness and Trade Project American Chamber of Commerce Agriculture Sustainability Investment Project ATLAS BEEPS Azerbaijan Trade Linkages and Agribusiness Strengthening Activity Business Environment and Enterprise Performance Survey BSP CBA CEDAW Business Service Provider Central Bank of Azerbaijan Committee on the Elimination of Discrimination against Women COP Chief of Party COR DCED Contracting Officer’s Representative Donor Committee for Enterprise Development DCOP ERC Deputy Chief of Party Economic Research Centre EU European Union FMS Financial Monitoring Service GOAJ HACCP ICT ISPM IPPC Government of Azerbaijan Hazard Analysis and Critical Control Points Information and Communication Technology International Standards for Phytosanitary Measures International Plant Protection Convention M&E MCIT Monitoring and Evaluation Ministry of Communications and Information Technology ME&A MoA MoED MoF MoFA MoH M4P Mendez England & Associates Ministry of Agriculture Ministry of Economic Development Ministry of Finance Ministry of Foreign Affairs Ministry of Health Markets for the Poor MONEYVAL Evaluation of Anti-Money Laundering Measures & the Financing if Terrorism (Council of Europe) MOU Memorandum of Understanding NGO PSCEP Non-Governmental Organization Private Sector Competitiveness Enhancement Project PMP PPD Performance Management Plan Public Private Dialogue PSD RIA SCC SCSMP SDC SECO Private Sector Development Regulatory Impact Analysis State Customs Committee State Committee on Standardization, Metrology and Patent Swiss Agency for Development and Co-operation State Secretariat for Economic Affairs SME Small and Medium Enterprise SOW SPA SPS STTA TCN TVET TIRSP Statement of Work State Procurement Agency State Phytosanitary Service Short Term Technical Assistance (Expert) Third Country Nationals Technical and Vocational Education and Training Trade and Investment Reform Support Program TOR Terms of Reference UN United Nations USAID WARD United States Agency for International Development Women’s Association for Rational Development WB World Bank FINAL PERFORMANCE EVALUATION OF ACT ii | Page EXECUTIVE SUMMARY EVALUATION PURPOSE AND EVALUATION QUESTIONS This is a report on the final performance evaluation of the Azerbaijan Competitiveness and Trade (ACT) Project, funded by the United States Agency for International Development (USAID) Mission in Azerbaijan and implemented by Sibley International during the period October 2010 - September 2013. The evaluation of ACT was conducted from July to August, 2013, by a team assembled by Mendez England & Associates (ME&A) with headquarters in Bethesda, Maryland. The team consisted of four experts - two international and two local – with significant experience in Azerbaijan and in the region. The main purpose of the evaluation was to determine the extent of the contribution made by ACT in supporting the Government of Azerbaijan (GOAJ) make progress towards regulatory reform in strengthening and improving agricultural value chains and advisory support services, as well as enhancing Azerbaijan’s trade environment with particular emphasis on accession to the World Trade Organization (WTO). The main thrust of the evaluation was to: 1) determine the extent to which ACT was successful in influencing a change in the mindset of key decision-makers in GOAJ towards improving the business enabling environment in Azerbaijan; and 2) assess the effect of measures taken by the project to improve the economic and trading prospects of the farming community nationwide, including farmers and local business service providers (BSPs). An integral part of the evaluation mission was to answer a set of pre-determined questions in order for USAID/Azerbaijan to obtain a greater understanding of the effect that ACT had on intended beneficiaries. In addition, the evaluation was to provide lessons learned for USAID/Azerbaijan regarding the design and implementation of ACT, and offer recommendations for consideration regarding the implementation of the follow-on Azerbaijan Trade Linkages and Agribusiness Strengthening (ATLAS) Activity. The evaluation questions were sub-divided into four specific categories: 1) Meeting Targets/Reporting Progress; 2) Progress in Key Intervention Areas – Results and Impacts; 3) Evaluating the Effectiveness of Technical Assistance; and 4) Evaluating the Sustainability of the ACT Project. The Evaluation Team’s response to each of the questions within the above categories is addressed within the main body of this report, both in tabular format as well as text. PROJECT BACKGROUND ACT was designed to help eliminate or mitigate technical and administrative barriers that were deemed to be hindering economic progress in Azerbaijan with respect to private sector development. ACT would therefore facilitate or act as a catalyst for change through supportive measures aimed at improving the country’s business enabling and trade environment, while at the same time implementing initiatives designed to enhance the competitiveness of those involved in the agriculture sector. The ultimate result of ACT’s interventions would lead to specific indicator targets being met across three component areas: 1) Domestic Business Environment Improved; 2) Liberalized, more Transparent Trade Environment Developed; and 3) Targeted Agricultural Value Chains Improved. The first two components were largely focused on processes and steps taken towards achievement of an ultimate objective, such as WTO accession, while the third component concentrated on improving the business and trading performance of farmers and food processors through the introduction of productivity improvement measures and training programs. Capacity building of BSPs would also feature as an important ACT intervention. In summary, as a direct result of ACT’s interventions, key decision-makers in GOAJ would be better informed regarding how to improve the governance of Azerbaijan, while farmers and food processors would become more productive, knowledgeable, and better-off after improving their working practices. EVALUATION DESIGN, METHODS AND LIMITATIONS The evaluation of the ACT Project was conducted during the period July – August, 2013, prior to the end of the project (September 2013). The Evaluation Team took the timing of the evaluation into account when preparing its evaluation design and methodology, as it was apparent that ACT’s final reports would not be available for review during the course of the evaluation. Accordingly, reported results highlighted in Annex D – Overall Monitoring and Evaluation Matrix, reflect performance only from data and information gathered during the evaluation mission. However, where results are anticipated by the end of the project, the Evaluation Team has FINAL PERFORMANCE EVALUATION OF ACT iii | Page commented upon in the relevant section. In addition, in May 2012, the project was subject to a significant revision of its original Statement of Work (SOW). In agreement with USAID/Azerbaijan, this revision was also taken into consideration with respect to the actual work the Evaluation Team was tasked to carry out. The methodology that was ultimately adopted by the Evaluation Team to secure the required information and data that would support its findings, centered on conducting face-to-face interviews with pre-identified stakeholders with first-hand knowledge of ACT (mainly in Baku), and with key respondents within the agricultural sector, including farmers, food processors, BSPs and relevant local authorities (mainly countrywide). Interviewees were selected following discussions between the Evaluation Team and ACT’s Chief of Party and Component Leaders. Agreement was then reached on who should be interviewed based on a list provided by the ACT team from which a final decision was then made and a visit schedule established. Geographic location was also an influencing factor in drawing up the final list due to the distances involved in terms of travel. An important aspect of the latter was the decision to include focus group discussions (FGDs) with farmers in the regions visited by the Evaluation Team, in order to obtain their feedback regarding ACT’s interventions in the various agriculture sub-sectors that the project supported. At the same time, the Evaluation Team reviewed ACT-related documents and materials, such as annual work plans and quarterly reports, monitoring and evaluation (M&E) plans, surveys, budgets, and other sources of reference that would validate its findings and conclusions. After desk review of materials, the Evaluation Team concluded that Components 1 and 2 of the project would require more focus on qualitative aspects regarding performance, while Component 3 would be subjected mainly to a quantitative analysis. The Evaluation Team encountered few limitations during its fieldwork in Azerbaijan. Some of the more relevant are listed below: 1. The evaluation took place during summer holiday season in Azerbaijan, with an additional two-day official public holiday to celebrate the end of Ramadan (8th and 9th August, 2013). This impacted the Evaluation Team’s ability to arrange interviews because a number of targeted respondents were not available. In addition, the Evaluation Team experienced some difficulty in organizing meetings in the regions as this required central government’s approval at short notice. 2. As the issue of “sustainability” constituted one of the four categories of questions to be answered, the Evaluation Team felt that is was important to define “sustainability” in order not to unnecessarily limit the scope of its investigations and/or findings. To that end, different interpretations were allocated, first to BSPs, and second to government bodies. For BSPs “sustainability” would focus on the extent to which they experienced/were experiencing business growth, while for official parties “sustainability” would focus on the degree to which they now felt better equipped to implement actions without the need for continued external support. 3. The Evaluation Team considered the possibility of recall bias with regard to interviewee responses given that the project started in October 2010 and that some respondents could find it difficult to accurately compare organizational arrangements/access to services three years from prior to date. 4. As some anticipated results in Components 1 and 2 would not occur till after the project ends, the Evaluation Team would only be able to comment on the likelihood of results being achieved post-project which, in turn, could limit meaningful commentary on the likely sustainability of some of those activities (as per the definition mentioned above). In summary, while important, the above limitations did not prevent the Evaluation Team from gathering sufficient information and data needed to draw conclusions and make recommendations for similar, USAID-funded projects in the future, including the anticipated ATLAS project. Following is a summary of the main findings, conclusions and recommendations within each of the four main categories. FINAL PERFORMANCE EVALUATION OF ACT iv | Page A. MEETING TARGETS/REPORTING PROGRESS Findings 1. The Evaluation Team was required (as per the SOW) to assess whether there were any “significant or critical gaps” in the ACT Project’s implementation occurring in the period 2010-2013. The Evaluation Team used the following definition of gaps: “any delay or breach (against PMP) in service delivery as a result of: 1) unavailability or non-performance of key personnel; 2) delays in submission of progress reports; and 3) inability to respond to USAID specific requests.” The Evaluation Team found no evidence of such gaps occurring throughout the implementation of the ACT Project. For example, Amendment 31 altered the SOW of ACT and introduced a revised SOW. Amendment 4 made effective2 adjustments to the grant element. While these and previous modifications took place, it appears that they did not cause any gaps in the implementation of the project. 2. Following a thorough review and forensic examination of the ACT-related documentation provided to the Evaluation Team prior to mobilization for Azerbaijan and during the mission itself, it became evident that reporting formats and structures employed to track performance appeared to have no common thread linking one with the other in terms of recording actual results against plan over time. 3. The SOW of ACT Project3 did not include gender as a specific cross-cutting theme or a distinguished component. There were no gender specific objectives, activities, outputs, and outcomes assigned and embedded in the design of ACT components. The design of Component 3 (Activity 3.B) did include a performance indicator: Number of women and/or women-owned and managed businesses benefitting from project assistance. The project’s M&E plan and reports were expected to track and report this indicator. 4. A forensic analysis of the data examined in Annexes I, J and K, appears to highlight discrepancies in the allocation of funds throughout the years of ACT’s implementation. The Evaluation Team found that although Amendment 3 of the original SOW had tasked the project with realigning activities towards support for agriculture within Component 3 via providing support to BSPs, this appears not to have occurred in practice. Conclusions 1. The Evaluation Team noted that, overall, the Contractor has ensured that there were no gaps in the implementation of the project and has responded in an efficient way to various modifications issued by USAID as evident by lack of delays or breaches in service delivery that might have been caused by such modifications. 2. ACT’s data gathering procedures appear more complex than necessary for tracking and measuring performance. This was confirmed by at least one component team leader whose views suggested a lack of direction in the project which had “no tangible goals except Component 3” resulting in an inability to accurately measure performance. A further comment concluded that due to “internal project resistance” towards employing M&E professional software, few in the project understood the purpose or value of trying to measure performance. 3. ACT’s design did not take into consideration gender issues, such as gender division of labor, time use, control of resources, or access to finance. If benchmarked against the USAID Gender Equality and Empowerment Policy and the USAID Policy Framework, the design of ACT Project has failed to respond to requirements and guiding principles featured in these documents. 4. The evidence appears to suggest that ACT did not take adequate account of the requirement in Amendment 3 to reduce the ‘use of international consultants’ in favor of support to BSPs in order to increase their capacity to effectively service the project’s ultimate beneficiaries – farmers and food processors. Therefore, cost￾effectiveness remains questionable. Recommendations 1. It is recommended that USAID sets more robust requirements in terms of the design and introduction of M&E plan earlier in the project implementation. This should include a requirement to conduct baseline surveys during the first few months of project life.                                                              1 05/03/2012. 2 09/20/2012. 3 ACT Project, amendment/modification No. 3, 05/03/2012. FINAL PERFORMANCE EVALUATION OF ACT v | Page 2. USAID should ensure that projects of this nature employ readily accessible, professional project management IT software, based on ‘interactive’ reporting, i.e. logging and analysing of data by project team members on an on-going basis. Such a system should be installed at the start of every long-term technical assistance project. 3. Future USAID interventions must be designed in accordance with the USAID Gender Equality and Empowerment Policy and the principles set out in the USAID Policy Framework, 2011-2015. Design of future interventions should be guided by some or all of the 7 guiding principles that underpin this policy and the parameters of the USAID Forward. 4.  USAID oversight of the project’s budget allocation needs to be sufficiently rigorous and robust enough to track performance, identify anomalies in a timely manner, and advise project management if remedial action is required. There is an overwhelming need for a professional M&E reporting mechanism to be an integral part of a project’s implementation modus operandi. B. PROGRESS IN KEY INTERVENTION AREAS – RESULTS AND IMPACTS Findings 1. Technology transfer: During the regional visits, the Evaluation Team found that the majority of respondents among farmers/groups received assistance from international STTA mainly in the form of 1-3 days classroom-based seminars, complemented with a study tour to Georgia. The majority of respondents among farmers/groups rated the relevance and outcome4 of international STTA as average or low but have assessed the relevance, quality, and impact of the study tour and ACT local consultants as high. Strengthening farm advisory and input delivery services: In consideration of cost-effectiveness of inputs vis a vis envisaged SOW, the SOW required the Contractor to deliver its assistance primarily through local BSPs. This appears to not have been the case necessarily, as the majority of farmer training also included (to a variable extent) direct inputs from international STTA and ACT staff. Capacity building of post-harvest and processing facilities: The Contractor has extended highly visible support in the target regions to post-harvest and processing facilities in the hazelnut sub-sector. While visiting these regions, the Evaluation Team noted several examples of cost-sharing approach, leading to various investments by processors in better hygiene, food quality, and infrastructure. Linking producers with markets: In hazelnut and fish farming sub-sectors, the Evaluation Team recorded favorable responses of clients. Respondents at Aznar pointed out several tangible outcomes of STTA provided by US Nationals and ACT local consultants. The management of the enterprise has confirmed that ACT Project has been instrumental in assisting the enterprise with widening international export linkages and securing new contracts for export of pomegranate and other fruit juices. 2. Policy Reform – Relevance: According to the responses received by the Evaluation Team, the overall picture appears quite positive:  On the issue of whether interventions were demand driven (i.e. reflected expressed needs of beneficiaries), most of respondents stated that the government beneficiaries of the project’s activities were consulted with regard to the need for the project, its content, and its delivery.  With regard to the economic policy reform emphasis of the ACT Project, most respondents were in agreement with the statement “The work of the project has included substantive interaction/ collaboration with similar projects/activities of World Bank and GIZ in Azerbaijan.”                                                              4 Outcomes are the changes, benefits, learning or other effects that happen as a result of USAID work. Impact is the broader or longer￾term effects of the ACT project or USAID outputs, outcomes and activities. For example, these are effects on people other than the direct users of ACT, or on a broader field such as GOAJ. FINAL PERFORMANCE EVALUATION OF ACT vi | Page  Most government organizations (ministries, state committees, agencies, etc.) mentioned that having proposed amendments and changes to existing legislation conducted through open public-private dialogue, was extremely productive. Policy Reforms – Sustainability: In addition to the networking opportunities afforded in the range of conferences and other events, project participants from different organizations confirmed that there has been quite an active engagement among project participants, and (to some extent) across projects. Furthermore, project participants and ACT staff reported a good deal of contact and discussion regarding project work and findings. 3. WTO Accession Feedback received from meetings with governmental organizations suggests an overall perception that the ACT Project has achieved its aim of assisting GOAJ adapt and adopt legislation that satisfies WTO requirements. However, this is an on-going process. Conclusions 1. Technology transfer: While STTA and ACT local staff technical assistance was designed to be focused on increasing productivity, yields and sales at farm level, the Evaluation Team found mixed or fractured evidence to robustly gauge the extent to which international STTA in itself was able to deliver relevant technologies or was successful in facilitating actual technology transfer. The Evaluation Team did find clearer evidence that overall, in the dairy and livestock sub-sectors, STTA by US Nationals was less tailored to the needs of clients than that in hazelnuts and fish farming sub-sectors. Strengthening farm advisory and input delivery services: The Evaluation Team concluded that there is no clear and robust evidence suggesting that ACT delivered its assistance primarily through local BSPs, as was required by the SOW. The Evaluation Team also concluded that the outcomes of ACT’s support to BSPs were uneven across the board. Mixed results pointed out that an extensive program of support extended to BSPs by international STTA and longer term technical assistance by ACT consultants, might have not fully achieved intended outcomes. Capacity building of post-harvest and processing facilities: The Evaluation Team concluded that given observed investments by enterprises, there is clear evidence of growing appreciation among processors of the importance of Hazard Analysis and Critical Control Points (HACCP), particularly in connection with export of produce and as means of boosting competitiveness. Linking producers with markets: There is also clear evidence available that as a result of ACT’S support to pomegranate and hazelnut sectors, processors directly, and farmers/groups indirectly, benefitted from an improved access to international markets. 2. Policy Reform – Relevance: The overall positive response regarding progress towards policy reform supported by the ACT Project could have been achieved fully if these efforts were implemented in a more consistent manner in order to maintain a balance of interests between key stakeholders and their coalitions (State Custom Committee, Ministry of Finance, Ministry of Agriculture, State Committee for Standardization, Metrology and Patent, Ministry of Economic Development, and Ministry of Health among others). Policy Reforms – Sustainability: The networking that has been achieved by the ACT Project is a valuable asset for sustainability of future USAID interventions. 3. WTO Accession ACT appears to have succeeded for the most part in influencing GOAJ towards making significant progress completing the required package of reforms demanded by the WTO. FINAL PERFORMANCE EVALUATION OF ACT vii | Page Recommendations 1. Technology transfer: STTA assistance at farm level should ideally be complemented by cost-shared, matching grants, or fully subsidized pilot projects (depending on what is most practical) aimed to enhance theory-based training with practical adaptation of technologies and practices. Strengthening farm advisory and input delivery services: Farm advisory and input delivery services fall within the merits of the National Extension Service of the Ministry of Agriculture. Agro-Scientific Centers are the appropriate units for extending farm advisory and input delivery assistance to farmers. It is recommended to review US Government (USG) support to BSPs as primary vehicle of delivering assistance to farmers. It is considered that partnerships between leading Agro￾Scientific Centers, BSPs, farmer associations, and processors are likely to build more sustained results. Capacity building of post-harvest and processing facilities: Given perceived success of HACCP interventions, it is recommended that future USAID Azerbaijan projects continue extending related support to processors. Such assistance should be subject to a thorough assessment of their commitment to HACCP adherence and investment potential. Linking producers with markets: It is recommended that future USAID projects continue to build on the success of ACT’s assistance to pomegranate and hazelnut sectors in terms of further technology transfer at farm and processor level, and further extension of HACCP reforms. Official registration of the Hazelnut Association will lead to consolidation of hazelnut processors around Zagatala, reinforcing the importance of comprehensive support to hazelnut farmers in sustaining equitable yields, productivity, and sales. 2. Policy Reform – Relevance: The future ATLAS project should continue coordination with the World Bank-funded ASIP project, as well as projects funded by other international donors. The balance of coalitions’ interests should be considered to encourage inter-ministerial cooperation and coordination. Policy Reforms – Sustainability: The anticipated ATLAS project should continue with workshops by using the already existing network. 3. WTO Accession: There are no specific recommendations required regarding further support to the WTO from ACT given the above-mentioned input to date. C. EVALUATING THE EFFECTIVENESS OF TECHNICAL ASSISTANCE Findings 1. Regarding Components 1 and 2, there is clear evidence based on feedback from key beneficiaries such as MoED, Financial Monitoring Service (FMS) of Central Bank of Azerbaijan (CBA), and the Ministry of Foreign Affairs (MoFA), among others, that STTA input was valued and appreciated. More specifically, STTA support contributed towards assisting GOAJ regarding WTO accession, developing an e-learning education system for FMS, and supporting the State Procurement Agency in organizing round tables to discuss legislation and proposed law amendments. In addition, STTA input was prevalent in assisting the MoFA in preparing for Working Party meetings with respect to WTO accession. 2. With respect to Component 3, analysis of the data in Annex J indicates that, to date, a total of 4,842 days have been allocated to STTAs (3,949 international and 893 local). STTA input for agriculture activities (demonstration farms, farmer training and direct extension support, direct value chain assistance and market analysis) accounted for 1,541 days or 32% of the total STTA number of days allocated. Furthermore, this STTA input only occurred in Years 2 and 3 as a result of Amendment 3. The level of input from international STTAs remained the same in Years 2 and 3, although Amendment 3 clearly directed that this should be reduced in order to support local BSPs. From the field trips conducted by the Evaluation Team, which FINAL PERFORMANCE EVALUATION OF ACT viii | Page covered all regions supported by the project, there is little evidence to suggest that all this effort by STTAs, in real terms, was effective either relative to reported M&E indicator achievements, or what the beneficiaries themselves advised was the actual impact on their activities. Finally, in considering that, collectively, international and local STTAs accounted for nearly $10million over the three years (roughly 44% of the entire ACT budget) it is questionable as whether STTA input has been effective to any great degree. 3. In general, the Evaluation Team found that most interviewees were aware of the existence of the ACT Project, but not necessarily of its main objectives (comments from EU Delegation and Swiss Co-operation Office, Swiss Embassy, among others). However, it is important to note that actual collaboration did take place, for example, between the ACT Project and the Swiss, both acting as co-leaders of the semi-annual Trade and Investment of the Donors Coordination Meetings initiated by the European Union (EU). In addition, the Swiss and the EU participated in the USAID-led Agricultural Donors Coordination Working Group. In fact, during meetings that the Evaluation Team had with respondents from the government and the private sector, it became clear that some of them confused the ACT Project with one or other similar on-going programs with which they were cooperating. Conclusions 1. STTA input within Components 1 and 2 from both a qualitative point of view and a quantitative perspective (most indicators to date have been achieved or exceeded in both components5 ), have for the most part been in line with the project’s expectations. STTA contribution towards helping respective ministries and departments move forward relative to agreed support initiatives and action plans has also largely been successful and, in overall terms, effective. Feedback from GOAJ beneficiaries confirms each of these to be the case. 2. Within Component 3, however, to some extent it is difficult to be conclusive about the effectiveness of STTA input as their involvement largely occurred following Amendment 3. The data contained in Annex J prepared by the Evaluation Team provides evidence but not a verdict. One of the problems encountered is that much of the commentary obtained from the field is either anecdotal or not supported by documented evidence, i.e. sales figures, productivity analysis, etc. Although the project conducted farmer and processor surveys, these (as discussed elsewhere in the report) cannot necessarily be relied upon either due to the subjective nature of the data provided or subsequent calculations that have to be made where percentages are involved. A further point in question stems from the ability of local experts to offer advice and training to ACT clients. Evidence from the field appears to indicate that they did not have the requisite skills or know-how to either train the BSPs or advise farmers practically or professionally. Therefore, the best that can be said, is that STTAs in some cases provided useful input to some clients, while in others their contribution was not deemed to be of value or relevant. 3. The main conclusion reached regarding overall awareness is that ACT was not active enough in establishing closer links with major stakeholders at the start of the project, or creating a sufficiently high enough profile aimed at informing the donor community in particular. Recommendations 1. There are no specific recommendations suggested here regarding STTA input with respect to activities within Components 1 and 2 for the ACT Project, which is near completion. 2. From the above findings and conclusions, there is a strong case for serious consideration to be given by USAID to the way in which budget allocations for short-term experts - whether local, international or third country nationals - are calculated. Furthermore, there must be some logic as to the purpose of their planned interventions in terms of how this relates to project indicators, outcomes, and impacts. The result of their input should be measurable and directly related to the specific needs of target audiences. In practical terms, experts should be knowledgeable and have the necessary motivational as well as technical skills to make things happen.                                                              5 Some of the indicator targets not yet achieved are anticipated to be achieved by the end of the project. FINAL PERFORMANCE EVALUATION OF ACT ix | Page 3. One of the most effective means to ensure close cooperation between technical assistance projects such as ACT and key stakeholders in a host country is an information gathering survey that needs to be designed, planned, organized and conducted (particularly aimed at donors) immediately following the start of a project. This should serve to ensure that common goals are agreed upon regarding the implementation of respective projects, with a view to avoiding duplication of activities and/or confusion among targeted beneficiaries receiving support. It should be noted here though that a consultation event was conducted with all donors at the design stage of the ACT Project to determine the area of their engagement and lessons learned. Goals of the project were also in compliance with the new USAID Country Development Strategy. However, from comments made by some stakeholders, more might possibly need to be done (by future similar projects) to secure their commitment to contributing towards agreed goals. D. EVALUATING THE SUSTAINABILITY OF THE ACT PROJECT Findings 1. There is clear evidence that ACT Project’s STTAs worked closely with internal ministerial staff involved in the drafting of the legislation. This is supported by comments made by most interviewees that given the centralized governance of Azerbaijan, no legislative drafts or amendments would be considered without the active involvement of GOAJ in-house experts. In reviewing ACT’s Results Framework tables in each of the project’s yearly Work Plans, it is apparent that a significant effort has gone into the development of draft laws, guidelines, preparation of seminar materials, and organization of roundtables by the project team, particularly within Component 1. 2. The Contractor has rigorously pursued the approach of putting in place processes, systems, and management arrangements in partner and beneficiary organizations. The Evaluation Team noted that several examples of this approach have been applied, as reported by the Contractor in the project progress reports. During the fact-finding meetings in Baku and other regional trips, the Evaluation Team made every effort to verify the ACT Project data directly with partner and beneficiary organizations and farmers/groups. The Evaluation Team’s interpretation of processes, systems, and management did not include training activities themselves but rather evidence of change that occurred as a result of training and technical assistance (TA). 3. ACT interventions have supported key elements of GOAJ reforms for improving the business and trade environment in Azerbaijan, including changes in legislation, support to local financial institutions, and technical assistance to improve agribusiness knowledge and skills, among others. However, there remain serious obstacles to sustainability, as evidenced by the significant gap existing between economic growth and institutional development. This issue presents significant difficulties for Azerbaijan’s transition from quantitative indicators to qualitative indicators, and from a supply-based economy to demand-driven economy. 4. With regard to ACT’s task to improve targeted agricultural value chains within Component 3, the Evaluation Team found that some progress was made in strengthening the technical and managerial capacity of Business Development Services (BDSs), supporting the preparation of demonstration plots6 , contributing towards the establishment of business linkages/relationships especially in pomegranate and hazelnut sectors and developing marketing plans for a dozen companies with export potential. The project also worked with financial institutions (FIs), including four commercial banks (Demir Bank, Turan Bank, Bank Republika and AG Bank), a non￾banking financial institution (AzerCredit), and two financial sector associations (AMFA and ARPA) to improve and expand lending to agriculture. ACT also provided HACCP support to six enterprises via intermediary service provider Azsertifika. 5. Cost sharing – Banking Sector Reform: AgBank has made a contribution of 10,000 AZN towards the cost of STTA provided by the Contractor to the bank. The ACT Project has extended support in the analysis of bank procedures and practices, development of strategic directions, and the design of a new micro-finance product. The ACT Project extended wider analysis and strategic advice to the bank over the course of six months in 20127. The bank considered that a                                                              6 For example a pomegranate GAP demonstration plot in the Bigir village in Goychay region. 7 The assistance extended by U.S. National Professionals was considered less relevant and effective than that of Non-U.S. Expatriates/Third Country National (TCN) Professionals. FINAL PERFORMANCE EVALUATION OF ACT x | Page more focused STTA on the subject of micro-finance product development was better tailored to specific needs of the bank at that time. Cost sharing – HACCP: The Evaluation Team noted growing appreciation among processors of the importance of HACCP, particularly for export operations and as means of boosting competitiveness. For example, Russia’s recent accession to the WTO, demands that Azerbaijani exporters adhere to HACCP and ISO standards closer than ever before. Conclusions 1. Available evidence suggests that the ACT Project has successfully contributed towards key staff of beneficiaries/implementing partners, playing a significant role in their respective positions with regard to the preparation of draft laws and amendments in collaboration with project experts. While it is not possible to conclude that every such initiative has resulted in 100% ‘ownership’, it is certainly safe to say that progress has been made in building the capacity of GOAJ staff and in-house legal experts regarding processes and procedures. 2. While it is premature to assess the outcomes or impacts of ACT operations, the findings indicate that the Contractors’ effort aimed at delivering a robust approach to ensure sustainability has started yielding some intermediate results. However, given the content of data gathered, the Evaluation Team concluded that the results are more evident in the private banking sector, as well as hazelnut, pomegranate, and aquaculture sectors. 3. The large gap between country economic growth indicators and quality of institutional changes remains the very serious obstacle for USAID’s on-going and future efforts for increasing the sustainability of projects. 4. ACT Project played an active role in supporting initiatives in the field geared towards future sustainability. Various examples of processes launched, systems designed and installed, and management practices adapted, tend to give confidence that such tools and mechanisms may be applied in the future without external support of the USG. Furthermore, it is possible to conclude that the employment of a cost-sharing approach might have also contributed to sustainability. 5. Cost sharing – Banking Sector Reform The outcomes of the cost-sharing approach, combined with provision of tailored STTA, are more likely to lead to systemic changes in strategic and managerial behavior of clients. The banking sector clients have augmented outdated financial products with new offers that are based on modern risk assessment methodologies relatively new to Azerbaijan. AgBank, for example, is keen to build on the success of the new financial product designed, launched and piloted with ACT’s support in Barda, Yevlakh, and Shamaki, by rolling out the product to several other regions without external assistance. Cost sharing - HACCP Evidence gathered by the Evaluation Team leads to the conclusion that the cost-sharing approach for STTA applied by the Contractor has been effective. In particular, cost-sharing with partners and clients of the ACT Project represented added value to USG assistance. Recommendations 1. The Evaluation Team does not have specific recommendations in this area since feedback from targeted beneficiaries/implementing partners was largely positive and supportive of ACT’s initiatives. However, some commentators suggested that any follow-on USAID project might focus some attention on legislative reform, specifically aimed at small, medium, enterprises (SMEs). 2. It is recommended that future USAID projects put emphasis of combining TA with efforts to address systemic measures following USG support rather than at intermediate level. 3. Continued support for institutional changes in existing economic policy-making processes and procedures at national and local levels are core for increasing the sustainability of international support. At the local level, support could be given to establish farmers and rural SMEs business associations or specialized agricultural producers cooperatives. At the regional level, research centers could be supported together with consulting FINAL PERFORMANCE EVALUATION OF ACT xi | Page organizations. At the national level, support could be envisaged to help restructure the Ministry of Agriculture (MoA) and its regional service units. 4. TA at bank institution and farm level should ideally be complemented by a cost-sharing or matching grant approach (depending on what is most practical) aimed at enhancing theory-based training and a TA with practical adaptation of new products, technologies and practices. Feedback from project beneficiaries suggests that this collaborative approach yields results and serves to ensure ‘ownership’ of initiatives and enhance future sustainability of those beneficiary recipients of financial support. 5. Cost sharing – Banking Sector Reform & HACCP The Evaluation Team recommends that the cost-sharing approach to STTA is applied as standard measure in projects where it is considered relevant and practical to achieve greater level of client commitment and local ownership. Effective management of expectations and clients’ ability to sustain STTA are equally important to the success of a cost-sharing approach. This recommendation is primarily based on the Evaluation Team’s conclusion that, in some cases (banking and financial reform, hazelnuts, dairy, and livestock), ACT clients’ have given preference to the assistance extended by non-U.S. Expatriates/TCN Professionals, which they found more relevant and effective than that offered by U.S. National Professionals. E. LESSONS LEARNED 1. Given the timeframe, budget originally envisaged, as well as restrictive socio‐political environment in Azerbaijan, the targets set seem too ambitious. Therefore, USAID should seriously consider the efficacy and value of designing a project that can realistically be implemented relative to the anticipated results as defined by pre-determined indicators in RFTOPs and SOWs. 2. Changing the SOW in the middle of the project, as defined by Amendment 3 of the SOW in May 2013 agreed between USAID/Azerbaijan and the Contractor, does not necessarily lend itself to a smooth transition towards implementing the envisaged changes in direction. In this case, there was clear confusion among members of the ACT Project team as to what really initiated such a significant realignment in project activities and objectives. Neither is there any official explanation in project documents or reports regarding such a change, which arguably adds to the confusion. Perhaps USAID should consider developing a set of criteria, which would need to be satisfied before SOWs can be amended. 3. International donors such as USAID often draw the assumption that by simply implementing a long-term technical assistance program in a host country means that beneficiaries and key stakeholders have ‘bought into’ the rationale and purpose of the program. This in fact is often not the case as more than one donor can be advising or working with those same beneficiaries or implementing partners at the same time. Evidence in this case appears to suggest that, to some extent, this has indeed occurred (GIZ, EC and others). Perhaps USAID might consider a more robust stance in SOWs regarding donor coordination to avoid duplication and to ensure that direct beneficiaries understand and appreciate the differences in assistance between USAID projects and others. This would best be achieved at the start of a project. 4. One of the issues arising from this evaluation was the fact that there is no guarantee that by simply advising the GOAJ on the preparation of legislation and regulations, this support will necessarily lead to their implementation. In this respect, it may be more beneficial for USAID to employ a broader interpretation of business enabling environment rather than limit itself to working towards improving the legislative and/or regulatory framework of a host country. This includes also working towards supporting diversification of the actors in the society dealing with economic policy issues and who could take the quality professional discourse on the economic issues further on their own. Furthermore, USAID should consider what it is in position to accomplish in the short- and long-run rather than undertake the whole menu of the necessary economic reforms, which will not fly given established political structures in the country. FINAL PERFORMANCE EVALUATION OF ACT 1 | Page 1.0 EVALUATION PURPOSE & EVALUATION QUESTIONS 1.1 EVALUATION PURPOSE This is a report on the final performance evaluation of the Azerbaijan Competitiveness and Trade (ACT) Project funded by the United States Agency for International Development (USAID) Mission in Azerbaijan. The project was implemented by Sibley International between October 2010 and September 2013. According to the Statement of Work (SOW) for this assignment (see Annex A), the purpose of the evaluation was to: 1. Determine the accomplishments (results) and impacts of ACT interventions and their sustainability; and 2. Provide lessons learned and recommendations from ACT’s design and implementation to inform the implementation of the follow-on Azerbaijan Trade Linkages and Agribusiness Strengthening (ATLAS) Activity. In addition, the evaluation needed to determine: 1. The progress in regulatory reforms achieved in Government of Azerbaijan (GOAJ) institutions that have participated/benefited from the ACT Project. 2. The ACT Project’s success in strengthening the value chains (production, processing-storage, and market linkages) in which it worked. 3. The level of development of farmer advisory and agribusiness services. 4. The project’s effectiveness in coordinating and collaborating with stakeholders (GOAJ, other donors, beneficiaries). 5. The various factors and conditions in the country that have enhanced or limited the effectiveness and results of the ACT Project’s assistance in Azerbaijan. 6. Whether or not deliverables outlined in the Work Plans have been achieved. The evaluation was commissioned by USAID/Azerbaijan Mission, which will use the evaluation’s findings and recommendations to guide the Mission’s future designs and implementation. The evaluation was conducted by a team assembled by Mendez England & Associates that consisted of four experts: international expert Mr. Colin Maclean (Team Leader); Mr. Danil Samoilenko (Trade/Economic Policy Reform Specialist); Dr. Fagan Aghayev (Local Agriculture Specialist); and Dr. Ibrahim Mammadzadeh (Local Trade and Economic Growth Specialist). 1.2 EVALUATION OBJECTIVES Per the SOW, the objectives of the evaluation were to: 1. Evaluate the progress made by the ACT Project under each component based on established targets; and 2. Provide recommendations for follow-on programming for USAID interventions in value chain development, business enabling environment (in the agriculture sector, competitiveness and, in particular, further assistance for World Trade Organization (WTO) accession). 1.3 EVALUATION QUESTIONS The evaluation questions included: A. Meeting Targets/Reporting Progress: 1. How effective was the Contractor in working towards meeting USAID and GOAJ goals and objectives in all three years of the ACT Project? a. Have there been any significant or critical gaps in ACT’s implementation? b. Were the assumptions established at the beginning of the project realistic throughout the period of implementation? Identify the assumptions changed and their impact on project progress and results? FINAL PERFORMANCE EVALUATION OF ACT 2 | Page 2. Was the data collected sufficient to measure program progress and results in all three program components? Were performance reports (weekly, monthly, quarterly, other) sufficient to monitor progress or identify problems in implementation and progress toward results? 3. Assess whether the data collected by the ACT Project provides evidence of progress towards accomplishment of intermediate results in the three areas. 4. Did the project design take into consideration gender issues such as gender division of labor, time use, control of resources, access to finance, etc.? 5. Are various results reported accurate and verifiable? Have the results achieved by the project been cost effective and sustainable? B. Progress in Key Interventions Areas and Results and Impacts: 1. Assess the progress and impacts of the value chain interventions under the ACT Project in terms of: i) technology transfer and the relevance of technologies transferred; ii) strengthening farm advisory and input delivery services; iii) capacity building of post-harvest and processing facilities; and iv) linking producers with markets. 2. Assess the range of policy reform efforts supported, progress and the status (e.g., legislative status of each reform supported) and priority reforms to be supported in future. 3. Evaluate the progress made in Azerbaijan’s accession to WTO and further support to be extended. C. Effectiveness of Technical Assistance: 1. Short term technical assistance (STTA): How effective was the STTA in meeting the needs of the ACT Project clientele? Assess the impact of STTA on the beneficiaries in terms of relevance of STTA recommendations and their adoption. 2. How effectively has the ACT Project created awareness about the issues it worked on among the private sector, donors, civil society, and Government agencies in Azerbaijan? D. Sustainability of the ACT Project: 1. Assess the extent of stakeholder ownership of interventions undertaken by the ACT Project. 2. What processes, systems and management were put in place to ensure that the results and impact of the ACT Project will be sustainable? 3. What were the obstacles to sustainability and what measures should be taken to increase sustainability in future projects? 4. Appraise the role of cost-sharing approach to Short-Term Technical Assistance (STTA) in terms of creating ownership of the beneficiaries and sustainability. 2.0PROJECT BACKGROUND The ACT Project was effectively designed to continue the work of earlier USAID-funded initiatives in Azerbaijan, in particular the Trade and Investment Reform Support Project (TIRSP)8 and the Private Sector Competitiveness Enhancement Project (PSCEP),9 in order to improve the business enabling environment, enhance competitiveness, and facilitate the elimination of barriers to trade and investment in Azerbaijan. The underlying rationale for such an initiative was that by supporting the GOAJ embrace the need for change in working practices with regard to legal and regulatory reform, creating a more open and transparent trade environment, and strengthening value chains in agriculture/agribusiness, the project would ultimately assist to facilitate business growth and improve foreign direct investment. In its recent “Doing Business 2013 – Azerbaijan” report, the World Bank ranked Azerbaijan 67th out of 185 countries; neighboring country Georgia by comparison ranks 9th, while Armenia ranks 32nd. While these ease of doing business rankings do not necessarily reflect ongoing initiatives aimed at improving economic competitiveness, a high ranking does reflect the extent to which a government has created a regulatory environment conducive to operating a business. In this respect, Azerbaijan still has some way to go towards creating a level playing field for domestic and international businesses that is more open, transparent, and free from the influences of local dominant private                                                              8 August 2007 – September 2010. 9 September 2008 – December 2010. FINAL PERFORMANCE EVALUATION OF ACT 3 | Page sector oligarchs. At the time of writing (August 2013), it was clear that the GOAJ has made some progress in recent years in addressing some of the above concerns, i.e., towards WTO accession, although this process has been slow and in fact started in 1997. To some extent, these developments were addressed by the GOAJ in its Presidential Decree “State Program on Socio-Economic Development of Regions of the Republic of Azerbaijan, 2009-2013,” particularly with regard to planned support for the non-oil sector. A brief review of the Action Plan contained within this document indicates the government’s intention to pursue six initiatives related to Tax and Monetary Credit Policy reform, and seventeen initiatives with regard to Measures on Expansion of Industrial and Agricultural Production. In addition, there are several hundred other actions involving cross-ministerial support. However, while this appears to suggest the government’s acceptance of the need for change in key areas of the economy, there is little available evidence about the extent of these planned interventions today, not least as it is unlikely that ministries have the capacity to carry through these measures. Indeed, this aspect was confirmed by several government bodies10 interviewed during the evaluation mission. Against this background, and given that much still remains to be done on the road to creating a market-based, competitive economy free from the constraints that have hindered progress in recent years in Azerbaijan, it appears evident that the decision by USAID to initiate the ACT Project in October 2010 was justified and necessary. In short, ACT dealt with a gap that needed to be addressed on this road to progress, namely, to provide the GOAJ with the skills and know-how needed to drive through legislative and regulatory reform. By acting as a facilitator and as a catalyst for change, ACT would be tasked with helping build the capacity of government ministries and agencies to actually implement identified reforms. At the same time, the project would strive to ensure that key personnel within ministries and other government agencies were provided with the necessary skills and expertise to be able to train their own people in the reform process so that they no longer need to rely on external consultants. With regard to ACT’s assistance to the business community, including farmers, this would be achieved through identification and capacity building of selected Business Service Providers (BSPs) enhancing the likelihood of sustained benefit after the project ended. In brief, ACT was a $21,997,184 initiative that started in October 2010 and is due to end in September 2013. Following a revision of the original SOW between USAID and Sibley International, as per Amendment 3 of Task Order Number: AID-112-TO-10-00002, and Indefinite Quantity Contract Number: AID –EEM-I-00-07- 00003, effective date 05/03/2012, the following components replaced the original ones together with a realignment of key result areas: 1) Domestic Business Environment Improved; 2) Liberalized, More Transparent Trade Environment Developed; and 3) Targeted Agricultural Value Chains Improved. In terms of this evaluation, and in agreement with USAID/Azerbaijan, the main focus of the Evaluation Team’s review would therefore be on the ACT’s activities following that amendment. Figure 1: Map of Azerbaijan In specific terms, the three components of ACT involved two distinct types of intervention: 1) Systematic Transformation, intended to change the nature of economic relationships; and 2) Enterprise Transactions, targeted to individual farmers and businesses with a view to enhancing productivity and incomes. Components 1 and 2 largely involved systematic interventions, while Component 3 mainly focused on transactional interventions. With the ACT Project nearing completion in September 2013,                                                              10 In particular the Ministry of Foreign Affairs and the State Procurement Agency. FINAL PERFORMANCE EVALUATION OF ACT 4 | Page USAID/Azerbaijan decided to conduct an end-of-project performance evaluation11, which is the subject of this evaluation report. The findings, conclusions and recommendations, described in detail both in the body of the report and its annexes, are the work of the team that was tasked with carrying out the evaluation. The result of the evaluation should allow USAID/Azerbaijan to identify lessons learned, assess strengths and weaknesses associated with the project, and refer to suggested recommendations for future project planning purposes. 3.0 EVALUATION METHODS & LIMITATIONS 3.1 EVALUATION METHODOLOGY To design the methodology for this evaluation, the Evaluation Team took into account a number of key factors, including quantitative and qualitative aspects of the mission, which could have influenced or placed intended/unintended limitations on its strategy for answering the pre-determined evaluation questions12. The more important of these factors are outlined below: 1. Key Factor: An important aspect of ACT’s mission was to meet a number of indicators and quantitative targets against which the project’s performance would be measured. Issue: As ACT was approaching completion (30 September 2013) at the time of the evaluation, the Evaluation Team concluded that the project’s results depicted in the various annual and quarterly reports13 to date relating to quantitative targets had already been approved by USAID and, therefore, there would be no need to verify the accuracy of the data reported in project documentation, including whether or not quantitative results had actually been achieved. Outcome: Given the above, the only requirement incumbent on the Evaluation Team was to report on ACT’s data gathering and reporting methodology in order to determine its efficiency and relevance from an operational point of view and thus, the extent to which it contributed to the effectiveness, or otherwise, of the project’s internal monitoring and evaluation (M&E) system. 2. Key Factor: The gathering and analysis of data and information from key informants during the evaluation took place in the middle of the summer holiday season in Azerbaijan, as well as during Ramadan, with local public holidays on the 8th and 9th August in celebration of Bayram. The Evaluation Team was also made aware that meetings with officials and/or the organization of focus group sessions with farmers outside of Baku needed government approval14 in the form of advanced letters of introduction. Issue: With reference to the above, the Evaluation Team had little option but to consider the possibility that targeted interviewees may not in fact be available to meet any of the team members during the evaluation mission. Outcome: Notwithstanding the above potential restrictions, the Evaluation Team was ultimately able to meet with 95 individuals from a wide cross-section of institutions, associations, government bodies, and private sector businesses and organizations. 3. Key Factor: One of the Evaluation Team’s tasks was to research and comment on the sustainability of the ACT Project in terms of stakeholder ownership of interventions and the impact of project results on their respective organizations and activities. Issue: In order to address this issue, it was important for the Evaluation Team to identify a number of meaningful criteria against which the sustainability of beneficiary institutions, agencies, and bodies could be assessed. Outcome: After serious deliberation, it was decided that there should be two categories of beneficiary:                                                              11 Final ACT reporting documentation would not be available for review by the Evaluation Team as they would not be ready at the time of the evaluation i.e. Final Report and Final Quarterly Report. 12 Annex C – Evaluation Design – highlights some of the anticipated limitiations that could impact on the evaluation. 13 Although the ACT 3rd Quarterly Report for 2013 was made available for review it had not yet been officially signed off by USAID/Azerbaijan and thus cannot be quoted as de facto. 14 On request from the Evaluation Team USAID/Azerbaijan prepared and dispatched offcial letters of introduction to the Executive Committee in Lankaran and to the Deputy Head of the Presidential Administration of Republic of Azerbaijan. FINAL PERFORMANCE EVALUATION OF ACT 5 | Page 1) BSPs15; and 2) official bodies including government ministries and/or their agencies. For the BSPs, sustainability would be defined in terms of business growth, including number of clients and number and type of new services offered. Performance would then be measured relative to the position they were in prior to ACT’s assistance and their anticipated position post-project intervention. In addition, consideration would also be given to any cost-sharing activities that occurred regarding project – client collaboration where client ‘buy-in’ of a jointly funded initiative could be viewed as a commitment towards sustainability. For the official bodies, sustainability would be defined in terms of the extent to which beneficiaries felt that following ACT’s assistance, they would subsequently be in a position to implement initiatives and deliver services themselves without further external expert advice. 3.1.1 Quantitative Research and Analysis The Evaluation Team reviewed a wide range of project-related documentation provided from several sources, including USAID/Azerbaijan (during pre-mobilization) and Sibley International (upon arrival in Azerbaijan). In particular, the Evaluation Team reviewed ACT’s M&E plans and related documents, such as Performance Management Plan (PMP) excel spreadsheets, as the main reference point for statistical data throughout the lifetime of the project. The team’s analysis of this material formed the basis of observations regarding the methodology employed by ACT to track its performance relative to pre-determined indicator targets (see Annex D – Overall M&E Matrix). The verification of statistical results actually achieved by ACT at the time of the evaluation, however, did not fall within the remit of this evaluation. The impact of ACT’s interventions on intended beneficiaries was reviewed by cross-referencing achievements (or otherwise) against the Evaluation Team’s findings from evidence gathered from its qualitative research approach. This included findings from interviews with pre-identified respondents with direct knowledge of ACT, as well as from feedback gained from focus group sessions with farmers. Given the nature of interventions, the Evaluation Team concluded that emphasis on Components 1 and 2 would largely relate to qualitative analysis, while Component 3 would focus mainly on quantitative analysis. The importance and relevance of this distinction lies within the revised SOW agreement between USAID and Sibley International16 in Amendment 3, May 2012, which highlighted the differentiation between activities/result areas that were effectively ‘in process17,’ i.e. Components 1 and 2 and those that could be quantitatively measured as per Component 3. Quantitative evaluation consisted of analyzing numeric data sourced from PMPs, M&E plan, project and progress reports, and performance indicators. This data was verified through semi-structured focus and discussion groups with BSPs and farmers that participated in or benefitted from the ACT Project. Annex D of this report depicts quantitative data indicators and targets, level of achievement, and comments on any deviation from the plan. 3.1.2 Qualitative Research and Analysis As already commented upon, the Evaluation Team focused on qualitative research and analysis for Components 1 and 2. Here, emphasis was much more on the effects of capacity building activities implemented by ACT, i.e. conducting of training sessions, facilitating events, acting as a catalyst between various ‘stakeholder’ entities, delivering seminars, and providing guidance on public awareness initiatives, among others. In addition, the team reviewed ACT’s contribution towards drafting briefing papers, legal amendments, and the project’s role in driving forward the WTO accession process. During its interviews with key informants, the Evaluation Team strived to elicit the views and opinions of the recipients of ACT’s support, and subsequently provide comments regarding ACT’s ability to essentially change mindsets and entrenched opinions at the decision-making levels of government. While this approach had to rely on subjective opinion and comment from interviewees, it did not diminish the quality of the Evaluation Team’s findings, since responses to questions were often verified though triangulation methods.                                                              15 Business Development Services (BDS) Agencies and HACCP Consulting Companies were identified as those organizations that would deliver ACT assistance to farmers not only to reduce use of inernational experts but to enhance sustained benefits post￾project. 16 This amended SOW identifies the need to employ internediate indicators to gauge the success of ACT interventions, as systematic change (behavioural change in attitude of beneficiaries) cannot solely be attributed to the project during its lifetime due to other possible influences. 17 Project activities within Components 1 and 2 mainly involved ACT interventions aimed at supporting progress towards an end result i.e WTO accession and would therefore be the subject of qualitative analysis rather than quantitative. FINAL PERFORMANCE EVALUATION OF ACT 6 | Page Although the focus of the evaluation centered on eliciting feedback from ACT’s direct beneficiaries, such as government ministries and agencies, BSPs and farming community, it was also important to ascertain the views and opinions of third party stakeholders, such as business and farming associations, financial institutions, and the international community, including the World Bank and European Union Delegation. The Evaluation Team met with representatives from each of these groups (see Annex E - List of Interviews - for a complete list). The primary purpose of these meetings was to ascertain the perceptions of individuals or organizations about ACT, with the purpose of enhancing the quality of recommendations for any future planned interventions by USAID/Azerbaijan. 3.2 EVALUATION LIMITATIONS The Evaluation Team encountered few major limitations during its fieldwork in Azerbaijan. Some of the more relevant are listed below: 1. The evaluation took place during summer holiday season in Azerbaijan, with an additional two-day official public holiday to celebrate the end of Ramadan (8th and 9th August, 2013). This impacted the Evaluation Team’s ability to arrange interviews because a number of targeted respondents were not available. In addition, the Evaluation Team experienced some difficulty in organizing meetings in the regions as this required central government’s approval at short notice. 2. The ACT Project was still ongoing at the time of the evaluation18. Therefore, the project’s final quarterly report and end-of-project report were not available for review at the time of the evaluation. As a result, the Evaluation Team did not have access to ACT’s reported final results, in particular with regard to quantitative analysis. Comments on this aspect have therefore been limited to ACT documents made available to the team while in-country. Similarly, as expected results in Components 1 and 2 would not occur until after the project has ended, the Evaluation Team was only able to comment on the likelihood of results being achieved post-project, which has an impact in some cases on the sustainability of activities and actions by beneficiaries. 3. The Evaluation Team considered the possibility of recall bias with regard to interviewee responses given that the project started in October 2010 and that some respondents may find it difficult to accurately compare organizational arrangements/access to services three years from prior to date. 4. The Evaluation Team assumed that all results reported to date in ACT progress reports (with the exception of the latest 3rd Quarterly Report, 2013) have been signed off by USAID/ Azerbaijan. Therefore, the team has not commented on the achievement of performance indicator targets but instead it has limited its remarks to ACT’s data gathering and reporting procedures, as well as the extent to which they contributed (or otherwise) to effective self-monitoring and evaluation. 5. The ACT Project employed large numbers of teams of international and local staff, most of whom were not available to answer evaluators’ questions. Given the high turnover of project staff over the course of two years, continuity and intellectual memory were additional challenges faced by the evaluation. In summary, while important, the above limitations did not prevent the Evaluation Team from gathering the information and data needed to draw conclusions and make recommendations for similar USAID projects funded in the future, including the anticipated ATLAS project. 4.0 FINDINGS, CONCLUSIONS & RECOMMENDATIONS 4.1 MEETING TARGETS / REPORTING PROGRESS 4.1.1 Question 1: How effective was the Contractor in working towards meeting USAID and GOAJ goals and objectives in ACT Project? Comment on critical gaps and                                                              18 ACT is due to end September 30, 2013. FINAL PERFORMANCE EVALUATION OF ACT 7 | Page project assumptions? Figure 2: QUESTION 1 - SUMMARY OF EVALUATION DESIGN AND METHODS Evaluation Question 1 Type of Analysis Conducted Data Sources and Methods Used Type and Size Sample Limitations 1. How effective was Contractor in working towards meeting USAID & GOAJ goals and objectives in ACT project? Comment on critical gaps and project assumptions. Description – based on content analysis of expert opinions Key informant interviews with ACT project personnel; ACT Project partners; ACT Project reports and M&E Plan. Interviewees identified by Evaluation Team and USAID. Components 1,2,3 None perceived a. Have there been any significant or critical gaps in the ACT Project’s implementation? Description – based on content analysis of expert opinions Key informant interviews with ACT project personnel; Key ACT Project partners. Interviewees identified by Evaluation Team and USAID. ACT Project reports and contract amendments. Components 1,2,3 None perceived b. Were the assumptions established at the beginning of the project realistic throughout the period of implementation? Identify the assumptions changed and their impact on project progress and results. Description – based on content analysis of expert opinions Key informant interviews with ACT project personnel and partners; USAID/Azerbaijan staff; ACT Project reports and contract amendments Analysis of validity of hypothesis behind ACT project. Interviewees identified by Evaluation Team and USAID. Components 1,2,3 None perceived 4.1.1.1. Have there been any significant or critical gaps in ACT’s project implementation? Findings 1. The Evaluation Team was required (as per SOW) to assess whether there were any ‘’significant or critical gaps’’ in the ACT Project’s implementation occurring in the period 2010-2013. The Evaluation Team used the following definition of gaps: “any delay, breach or seizure in service delivery as a result of: 1) unavailability or non-performance of key personnel; 2) delays with submission of progress reports; and 3) inability to respond to USAID’ specific requests.” The Evaluation Team found no evidence of such gaps occurring throughout the implementation of ACT. Amendment 319 altered the SOW of ACT and introduced a revised SOW. Amendment 4 made effective20 adjustments to the grant element. While these and previous modifications took place, it appears that they did not cause any gaps in the project implementation. 2. The Evaluation Team noted that there was a noticeable break with the design and submission of the M&E plan to USAID in the first year of project implementation. Order for Services was issued on the 30th of September 2010. The original SOW envisaged that the final Annual Work Plan needed to be in place within eleven weeks of the award start date and that the Contractor had to submit an M&E Plan in draft form within one month of the approval of the Work Plan (i.e. by the end of January 2011). ACT Year 1                                                              19 05/03/2012. 20 09/20/2012. FINAL PERFORMANCE EVALUATION OF ACT 8 | Page Annual Work Plan21 says that the Contractor will be developing a comprehensive M&E Plan to ensure timely data collection and reporting. However, the baseline survey22 with the purpose to establish the original conditions, or reference point, for the relevant PMP indicators was deployed only later in the fall of 2011. The M&E Plan was eventually submitted to USAID in September 2011, which is about 8 months later that envisaged by the original SOW, or 1 year since the start of the ACT in September 2010. Conclusions 1. The Evaluation Team noted that overall the Contractor has been efficient in ensuring that there were no significant or critical gaps in the ACT Project’ implementation regardless of various modifications issued by USAID. 2. Given that the Contractor utilized 37% of the overall project funding in Year 123, the Evaluation Team concluded that ACT invested substantial amount of US Government (USG) funding while the M&E plan was not fully operational as it was still under discussion with several generations of CORs. Nevertheless, these resources were utilized prior to Amendment No. 3 that introduced new SOW with revised objectives, targets, and performance indicators to those present in the original SOW. This means that the Contractor utilized international STTA at a cost of $948,327 while working towards targets in the original SOW that later were partially or fully revised in the new SOW. The significance of this comment is that irrespective of how expenditures were distributed at the start of the project no funds should have been spent at all in the absence of an agreed and signed-off M&E plan. Recommendations 1. It is recommended that USAID set more robust requirements with regard to the design and introduction of M&E plan earlier in the project implementation. This should include a requirement to conduct baseline surveys during the first few months of project life. 2. While Amendment 3 was desired by USAID, fundamentally altering the course and scope of ACT in the middle of implementation is by any standard a dramatic change to enforce on a project. Introduction of such modifications puts pressure on the Contractor to revise its existing approach, methodologies and planned inputs, targets, outcomes and outputs in a short period of time. It also requires the Contractor to re-evaluate the assistance it has extended earlier with the next targets in hand. Revision of previously set targets to the extent enforced by Amendment 3 inevitably raises concerns as to what extent resources utilized prior to that modification ultimately contributed to the achievement of newly agreed targets. 4.1.1.2. Were the assumptions established at the beginning of the project realistic throughout the period of implementation? Identify the assumptions changed and their impact on project progress and results. Findings In responding to this question it is worthwhile noting the Evaluation Team’s important observation, following a review of the ACT Project documentation, that the original SOW24 signed between Sibley International and USAID/Caucasus regarding the ACT Project, rather unusually, made no reference whatsoever to any assumptions that might need to be considered with regard to its implementation. Therefore, what follows in this section refers only to the assumptions that were mentioned in Amendment 3 of this SOW signed in May 2012. 1. With respect to Components 1 and 2, where interventions regarding ‘Systematic Transformation’ focused on ‘changing the nature of economic relationships across a wide range of participants, actions and activities’, the underlying assumption is that the performance of the ACT Project should not be measured against specific result indicators. Instead, interventions would be based on the hypothesis that the result(s) of ACT interventions might only occur post-project and therefore could not be directly attributable to                                                              21 October 1, 2010 – September 30, 2011. 22 ASR Consulting and Research Group. 23 $7,057,409.14 (or 37%) as per ACT Spending per Component report through June 2013. 24 SOW as part of Contract No. AID-EEM-I-00-07-00003 dated 09/30/2010. FINAL PERFORMANCE EVALUATION OF ACT 9 | Page project activities during the lifetime of the project. Against this background, the available evidence suggests that ACT largely fulfilled its brief to implement actions and initiatives that contributed towards progress in reaching a goal i.e. WTO accession, adoption of Hazard Analysis and Critical Control Points (HACCP) standards and increased export opportunities at participating hazelnut and pomegranate enterprises. 2. With respect to Component 3, which focused on ‘Enterprise Transactions’ where project interventions were expected to result in specific indicators being met following direct assistance to beneficiaries such as BSPs, farmers, farmer associations, and food processors, the underlying assumption was that in pursuing this agenda, real and lasting benefits would accrue to the recipients of project support, i.e. with regard to productivity improvement and income generation. Here, it can be argued that ACT’s performance has had mixed results, some important aspects of which are discussed in more detail in the following sections of this report, especially those concerning value chains and short-term technical assistance inputs. In particular, some of the indicators employed to measure performance required the project to make calculations in order to quantify percentage changes in that indicator over time, which were essentially based on data provided by farmers. In any event, reliance on such data (acquired mainly through surveys) appears to have been unsound, mainly due to the fact that the farmers in question effectively do not keep any record of their operating activities let alone make calculations. The information provided according to some of those who participated in the survey was based mainly on an educated guess or rough estimate regarding a particular activity. Conclusions 1. In terms of project design assumptions, those highlighted in Amendment 3 with respect to Components 1 and 2, remain valid and for the most part realistic. However, an important observation that needs to be made is that while intermediate result indicators were to gauge the success of systematic change within the project, this proved difficult to evaluate due to the lack of a functioning tracking mechanism (See 4.1.4 below for further elaboration). 2. Regarding the assumptions related to Component 3 where specific interventions aimed at supporting beneficiaries, both BSPs and farmers, were to result in strengthening the capacity of the former and improving the livelihoods of the latter, it is possible to conclude that such assumptions may not have been as robust as anticipated. This conclusion appears to be supported by feedback from those interviewed during the Evaluation Team’s field trips around the country where several BSPs indicated that following the ACT’s assistance (through capacity building measures) they did not feel better prepared or equipped to be able to advise clients (farmers or food processors) (see Annex H – Summary of BDS providers’ Responses). Recommendations 1. None specific other than the need to have a performance tracking mechanism in place, which can measure to some extent the progress of systematic change. Any assumption in an SOW should comment on the importance of this. 2. Assumptions related to quantitative performance should be certain that specific sets of indicators can in fact yield verifiable results that can realistically be measured. For example, a simple comparative analysis of the yield in a particular crop before and after specific project interventions can be fairly easily calculated. Similarly, for processors it would be relatively straightforward to ascertain progress or growth over time by determining the value and volume of exports before and after technical assistance. 4.1.2 Question 2: Was the data collected sufficient to measure project progress and results in all three project components and in particular with respect to intermediate results?25 Figure 3: QUESTION 2- SUMMARY OF EVALUATION DESIGN AND METHODS Evaluation Question Type of Analysis Conducted Data Sources and Methods Used Type and Size Sample Limitations                                                              25 Following a pre-mobilization enquiry by the Team Leader to USAID regarding similarities in Questions 2 and 3 in the SOW it was subsequently agreed that Question 3 could form an integral part of Question 2. FINAL PERFORMANCE EVALUATION OF ACT 10 | Page Figure 3: QUESTION 2- SUMMARY OF EVALUATION DESIGN AND METHODS Evaluation Question Type of Analysis Conducted Data Sources and Methods Used Type and Size Sample Limitations 2. Was the data collected sufficient to measure progress and results in all three project components and in particular with reference to intermediate results? Description – based on content analysis of expert opinions PMP; M&E Plan; Project reports; ACT project staff; Key informant interviews with ACT Project partners; ACT Project reports and contract amendments ACT project documentation; Interviewees identified by Evaluation Team and USAID including direct beneficiaries in Baku and nationwide Farmer focus groups Components 1,2,3 Process based project design; Limited perceived robustness of the initial baseline survey; unavailability of the end line survey; Lack of pre-determined quantitative targets (C3 – sales; productivity etc. benchmarks). Findings 1. Following a thorough review and examination of ACT-related documentation provided to the Evaluation Team prior to mobilization for Azerbaijan and during the mission itself, it became evident that reporting formats and structures employed to track performance appeared to have no common thread linking one with the other in terms of recording actual results against plan over time. For example, the first occasion on which any M&E reporting is mentioned, is in the 4th Quarterly Report of Year 1 (integrated with the Annual Report for that year) under Cross Cutting Activities. Here, the indicator table is referred to as PMP. At the end of Year 2, in the equivalent report, this becomes a Monitoring Report. At the start of Year 3, a separate M&E plan appears, which describes how indicators have been arrived at26 and how within each of the three project components results are measured as output (e,g. number of people trained) or outcomes (e.g. extent of behavioural change(s) in beneficiary attitude or performance). Indicators in the PMP within the M&E plan have apparently been selected to establish “critical links” and/or “causal chains.” More significantly, the same report outlines that some indicators are sub-divided into those whose results can be collected through ACT’s management information system (MIS) or from implementing partners (see Finding 2. below), while other indicators can be measured through results obtained from farmer or food processor surveys. However, the Evaluation Team finds it puzzling that although Amendment 3 of the original SOW occurred in May 2012, it took so long to develop an M&E plan that would only be relevant to the final year of activities. Furthermore, it is not clear on what basis these final indicators were decided. In short, the entire project monitoring and reporting system appears confused and unstructured, a finding confirmed to some degree by ACT’s component leaders. When asked to explain how indicators were arrived at and how they were to be measured, some respondents appeared confused and seemed to lack understanding as to what was expected. Further explanation included comments suggesting Amendment 3 changed everything and that this was a primary cause for confusion regarding indicators and data gathering activity. 2. An important feature related to ACT’s data gathering activities concerned the project’s use of baseline and mid-term farmer surveys27 to assess performance relative to specific PMP indicators, such as:  Percentage change in yield of targeted commodities as a result of USG assistance.  Percentage change in the value of purchases from smallholders of targeted commodities as a result of USG assistance.  Percentage of farmers, processors and others who have adopted new technologies or management practices as a result of USG assistance.  Number of micro-enterprises receiving business development services (BDS) from USG assisted sources.                                                              26 Apparently from two specific sources: 1) Standard Foreign Assistance Indicators as defined by USAID and 2) custom indicators developed by USAID/Azerbaijan. 27 Source of reference: 1) Azerbaijan Farmer Survey Report (no date) and 2) ACT Mid-Term Farmer Survey Report, January 2013. FINAL PERFORMANCE EVALUATION OF ACT 11 | Page In short, the purpose is to measure changes in performance over time by using a “panel survey” methodology where, in this case, the same groups of dairy, pomegranate and fish farmers are interviewed – 359 farmers in 21 regions28. This exercise requires an end-line survey in order to be able to make a final judgement regarding performance, which at the time of this evaluation had not been accomplished thus not allowing the Evaluation Team to offer any meaningful comment on the overall performance of the various indicators in question. As several of the indicators29 (in Component 3) require percentage calculations to be made based on data provided by farmers (who notoriously rarely maintain any business￾related records on issues of productivity), there is always a question as to the accuracy of such data. The Contractor informed the Evaluation Team that the data gathering process towards the end-line survey was in the process of being completed in July-August but the analysis and synthesis were yet pending. The end-line survey data was not made available to the Evaluation Team during the evaluation mission. In examining the results to date in ACT reports regarding percentage changes, the value of these results is unclear without the benefit of an end-line survey. Furthermore, the issue of how percentage targets were originally arrived has not been defined in any ACT documentation. Were these targets agreed with the beneficiaries? If not, how were they calculated and what purpose would they serve? These issues were raised with ACT but with no definitive response. Indeed, from feedback from farmer focus group sessions around the country, there is no evidence that farmers had any idea on how to record or report on measures of productivity or any percentage change in anything. Similarly, there remains a question as to the accuracy of figures quoted regarding job creation, as this requires a definition of what constitutes a “job” – short-term, seasonal or long-term employment contract? 3. Regarding the issue of Intermediate Results, the Evaluation Team found some confusion as to what constitutes an Intermediate Result relative to a Key Result30 and what relation they have to PMP indicators and or activities. Furthermore, the issue as to how Intermediate Results can be measured and thus commented upon, remains problematic as there are 13 Intermediate Results and 35 Key Results in the Work Plans with no direct link to any M&E reporting structure of PMPs. Since there are no specific indicators for scrutiny of Intermediate Results - which essentially only have Outputs/Deliverables – and there are no actual versus planned parameters to consider, it is not possible to offer further comment in this report. With regard to whether this issue is not specifically related to Intermediate Results, as described above, but refers more to the “big picture” in terms of progress to date, then the results indicated in Annex D – Overall Monitoring and Evaluation Matrix, offer an overview of achievements against plan. Where deviations appear significant, explanation is depicted in the relevant Comments column. For example, progress with the Central Bank of Azerbaijan (CBA) and Financial Monitoring Service (FMS) experienced a number of difficulties, in particular the risk assessment system. The CBA offered no specific explanation on this issue so no further comment is possible. Conclusions 1. The ACT Project’s data gathering procedures appear muddled and more complex than necessary for tracking and measuring performance. This was confirmed by at least one Component team leader whose views suggested a lack of direction in the project, which had “no tangible goals except Component 3,” resulting in an inability to accurately measure performance. A further comment concluded that due to “internal project resistance” towards employing M&E technology (professional software), few in the project understood the purpose or value of trying to measure performance. It is not possible for the Evaluation Team to conclude that the absence of professional project management software resulted in targets not being addressed or met; it simply means that the entire data gathering process seems to have developed by default over time rather than as the result of any meaningful pre-determined M&E system. The Evaluation Team has taken into account the ramifications of the changes in indicators in Amendment 3 of the original SOW, which required a re-focusing of activities, especially with regard to Component 3.                                                              28 ASR Consulting and Research Group was commisioned to conduct the ACT baseline survey. 29 There are 28 in total across all 3 Components. 30 Intermediate Results and Key Results appear in ACT’s three Yearly Work Plans in a Key Results Framework Table. FINAL PERFORMANCE EVALUATION OF ACT 12 | Page 2. With respect to the issue regarding the use of baseline or mid-term surveys to gather data and information from intended direct / indirect beneficiaries, it is difficult to draw any meaningful conclusion as to the purpose of such surveys, given the inherent difficulties mentioned above. The absence of an end￾line survey further detracts from being able to draw conclusions. In terms of the overall value of such surveys, it is questionable as to whether the data contained has any value to the project in terms of managerial decision-making purposes. 3. Regarding Intermediate Results, the Evaluation Team can only conclude that in the absence of any reporting structure in place to actually track their performance, no further comment is possible. The only conclusion that can be drawn is that some indicators were achieved, others are likely to be achieved by the end of the project, and some others are not likely to be achieved (mostly those in Component 3). Furthermore, it is also possible to conclude in light of this finding that the data collected was insufficient to measure any meaningful progress towards results simply based on the lack of available reporting evidence. Recommendations 1. USAID should ensure that projects of this nature employ readily accessible, professional project management software that is based on interactive reporting, i.e. logging and analysing of data by project team members on an on-going basis. Such a system should be installed at the start of every long-term technical assistance project. Furthermore, by adopting this methodology, which can calculate and extract data for monitoring purpose, reporting to USAID will be simplified and constructive, with information that is effectively available in real time and that can be presented in a format agreed between the project and USAID, i.e. weekly, monthly, quarterly, etc. 2. USAID might wish to consider the value or usefulness of employing surveys to gather data on intended beneficiaries, especially where calculations are required to determine the percentage change of an indicator. Alternatively, USAID may consider the option of not including this type of indicator in SOWs, instead focusing on indicators which can realistically be measured over time based on verifiable inputs (i.e. before intervention a client prepared no meaningful financial statements – after training the client is able to prepare a basic cash flow or income statement). 3. At the project design stage, it is highly recommended that indicators be clearly defined quantitatively and/or qualitatively and separated from actions, activities, or other expected results. In adopting this approach, the design and preparation of a properly functioning M&E system (as outlined above) assumes more purpose in terms of project reporting. This will also help track on-going performance so that results can be noted at any particular point in time. 4.1.3 Question 3: Did the project design take into consideration gender issues such as gender division of labor, time use, control of resources, access to finance, etc.? Figure 4: QUESTION 3 -SUMMARY OF EVALUATION DESIGN AND METHODS Evaluation Question Type of Analysis Conducted Data Sources and Methods Used Type and Size Sample Limitations 3. Did the project design take into consideration gender issues such as gender division of labor, time use, control of resources, access to finance, etc.? Description – based on content analysis of expert opinions Review of documents a. ACT SOW (Amendment №3); PMP; M&E plan and progress reports; b. Gender Equality and Empowerment Policy, USAID, 2012; c. Assessment of the Impact of Trade Policy Reform in Countries Acceding to the World Trade Organisation: The Gender Dimension, UN, 2010; d. Guidelines for Incorporating Organized by Evaluation Team in sample regions – target = 4-8 per focus group session including representatives from all identified participants. Components 1,2 (Gender mainstreaming in legislation and regulatory framework). Component 3 Limited availability of sex disaggregated data in ACT project reports; Unavailability of participants due to summer/religious holiday period: i. Public Union for Gender Equality and Women’s Initiatives; ii. “Women of the XXI FINAL PERFORMANCE EVALUATION OF ACT 13 | Page Figure 4: QUESTION 3 -SUMMARY OF EVALUATION DESIGN AND METHODS Evaluation Question Type of Analysis Conducted Data Sources and Methods Used Type and Size Sample Limitations Women’s Economic Empowerment into M4P programmes; DFID, 2012. Gender Focus Group Discussion e. Women’s Association for Rational Development (WARD). f. SDC/SECO. Meetings with gender organisations/female workers in selected regions (Value Chains: (1) pomegranate, (2) dairy, (3) aquaculture; (4) dairy, (5) hazelnut, (6) cold store (Value chains as regards gender division of labour, time use, control of resources, access to finance). Century” Public Union. Findings Review of Documents 1. ACT’s SOW31 did not include gender as a specific cross-cutting theme or a distinguished component. There were no gender specific objectives, activities, outputs and outcomes assigned and embedded in the design of the project’s components. The design of Component 3 (Activity 3.B) did include a performance indicator: Number of women and/or women-owned and managed businesses benefitting from project assistance. ACT’s M&E plan and reports were expected to track and report this indicator. The Contractor responded to this requirement by tracking progress and providing sex-disaggregated data for some of the indicators under Component 1, 2 and 3. ACT’s M&E report for Years 1-3 includes a number of quantitative indicators that measure cumulative expected targets versus cumulative actual targets, including:  Number of direct exposures of public education/advocacy.  Number of people trained with the goal of improving or working more effectively with the business enabling environment.  Number of participants in trade and investment environment trainings. These indicators provide evidence only regarding the number of male and female participants, which were exposed to public education/advocacy actions or were trained by the ACT project. Data provided for all other indicators across Components 1, 2 and 3 was not disaggregated by gender. 2. USAID Gender Equality and Empowerment Policy and USAID Policy Framework, 2011-2015, features seven guiding principles that underpin this policy along with the parameters of the USAID Forward reform agenda. Under this policy, USAID investments are aimed at three overarching outcomes, two of which are directly relevant to the design and scope of the ACT:  Reduce gender disparities in access to, control over and benefit from resources, wealth, opportunities and services (i.e. economic).  Increase capability of women and girls to realize their rights, determine their life outcomes, and influence decision making in households, communities, and societies.                                                              31 ACT Project, amendment/modification No 3, 05/03/2012. FINAL PERFORMANCE EVALUATION OF ACT 14 | Page In strategic planning at the country or project level, USAID expects these outcomes to be adapted and translated into specific results with associated targets and indicators. Gender Focus Group Discussion 3. The Evaluation Team conducted a focus group discussion with the Women’s Association for Rational Development (WARD). Key observations made were the lack of meaningful streamlining of gender in the legislative and regulatory framework, for example with regard to legal empowerment, training and operational engagement of Gender Focal Points. 4. It appears that several recommendations of the Committee on the Elimination of Discrimination Against Women (CEDAW) given to Azerbaijan in 2005 have not been fully addressed, particularly those related to introduction of comprehensive gender policies.32 Other relevant matters still to be addressed relate to the Law on Gender Equality (2006), which still appears not to have been underpinned by an Action Plan. 5. The Evaluation Team also conducted a focus group discussion with the Swiss Agency for Development and Cooperation (SDC) and State Secretariat for Economic Affairs (SECO). SDC/SECO project portfolio includes many thematic features, projects and activities that are similar to the scope of ACT in all of its components and some target regions. The purpose of the discussion was to gauge to what extent gender is mainstreamed in the design of SDC/SECO projects. The Evaluation Team found that SDC/SECO project designs are based on the Guidelines for Incorporating Women’s Economic Empowerment (WEE) into Markets for the Poor (M4P) programs. Meetings with Gender Organizations/Female Workers in Selected Regions 6. The Evaluation Team conducted several focus group discussions with farmers and farmer associations. They included ACT beneficiaries in Goychay (pomegranates), Barda (dairy), Agjabedi (livestock), Sheki (aquaculture), and Zagatala (hazelnuts and aquaculture). A questionnaire was prepared for distribution to both female and male participants. Female farmers or entrepreneurs were mostly unavailable to meet with the Evaluation Team; therefore, the team is unable to offer meaningful comment on disaggregation regarding women. Conclusions 1. ACT project design did not take into consideration gender issues such as gender division of labor, time use, control of resources, or access to finance. If benchmarked against the USAID Gender Equality and Empowerment Policy and USAID Policy Framework, the design of ACT has failed to respond to requirements and guiding principles featured in these documents. The Evaluation Team concluded that ACT project design did not have any specific gender thematic focus regarding gender equality, female empowerment and gender integration objectives across Components 1, 2 or 3. This indicates that the design of the ACT project, despite a review that led to the introduction of new SOW in 2011, had not responded to key requirements of the USAID Gender Equality and Female Empowerment Policy and principles outlined in the USAID Policy Framework, which was adopted in March 2012. The Evaluation Team found evidence though that gender mainstreaming and integration into programs was central to USAID's development approach in 2010 with the overarching objective of promoting gender equality as per USAID and WFP Gender Partnership33. 2. ACT used a narrow interpretation of the “Number of women and/or woman-owned and managed businesses benefitting from project assistance” indicator: sex-disaggregated data was quantitative and examined numbers of female and male participants exposed to project actions or trained. USAID Gender Equality and Empowerment Policy features a more comprehensive definition of what benefits are or may be that is beyond the scope of measure of numbers of participants trained. 3. The Contractor’s PMP and M&E plan were not fit for the purpose of achieving gender specific quantitative or qualitative targets and monitoring sex-disaggregated data as regards the ACT project results and impacts on women and women owned/run businesses in terms of:  Improvements in their management practice  Compliance with international quality control                                                              32 Consideration of reports submitted by States parties under article 18 of the Convention, CEDAW/C/AZE/2‐3, 2005. 33 http://home.wfp.org/stellent/groups/public/documents/resources/wfp222249.p FINAL PERFORMANCE EVALUATION OF ACT 15 | Page  Adaptation of new technologies  Increase in yield  Value of purchases and sales  Value of international exports  Number of jobs  Number of agricultural loans Given lack of gender focus in ACT design, and lack of sex-disaggregated data for these parameters, the Evaluation Team concluded that it was not possible to gauge how or to what extent ACT achieved intermediate results when supporting women or women-owned/run businesses. For the same reason, it was not possible to robustly assess in which ways and to what extent women benefitted from ACT interventions. 4. At the M&E level, the Contractor has attempted to gather some sex-disaggregated data across the three components in its baseline survey. Undisclosed numbers of women were included in the surveys’ respondent group in 201134. The Contractor reported that “women who participated in the surveys did not significantly differ from their male counterparts in age, nor did they differ in terms of experience.’’ It appears that other common criteria (i.e. equality in the work place, access to collateral, access to loans and financial resources, access to education, access to land and machinery, etc.) may however have not been taken into account. The Evaluation Team, therefore, concluded that the representative sample used by ACT was not, for the most part, gender specific. Recommendations 1. Future USAID interventions must be designed in accordance with the USAID Gender Equality and Female Empowerment Policy and the principles set out in the USAID Policy Framework, 2011-2015. Design of future interventions should be guided by some or all of the 7 guiding principles that underpin this policy and the parameters of the USAID Forward featured under Findings above. 2. The WEE, M&E and impact assessment systems must be integrated into a project’s cycle from the very start. This approach should ideally be supported by a strategic gender specific framework and log frame for a project with predetermined and agreed targets. If possible, these documents should be based on a baseline survey conducted during the first few months of the project life, as well as set explicit WEE objectives within the broader context of what the gender specific objectives are, and whether gender specific constraints will be addressed. Question 4.1.4: Are various results reported accurate and verifiable? Have the results achieved by the project been cost-effective and sustainable? Figure 5: QUESTION 4 - SUMMARY OF EVALUATION DESIGN AND METHODS Evaluation Question Type of Analysis Conducted Data Sources and Methods Used Type and Size Sample Limitations Are the various results reported accurate and verifiable? Have the results achieved by the project been cost effective and sustainable? Description – based on content analysis of expert opinions PMP; M&E Plan; ACT progress reports; Institute for Scientific Research on Economic Development (under MoED Interviewees identified by Evaluation Team and USAID. (WTO: cost benefit analysis and WTO related activities). Components 1,2,3 A robust cross￾component cost effectiveness analysis may not be readily available in the project Findings The rationale and purpose of this particular question stems from USAID’s concerns regarding its previous Private Sector Competitiveness Enhancement Project (PSCEP) in Azerbaijan, which started in September 2008 and ended in December 2010 (ten months earlier than planned). The issue at hand is whether or not ACT utilized its resources in a cost-effective manner relative to project goals. The Evaluation Team’s findings have                                                              34 ACT Project Azerbaijan Farmer Survey Report-6-28 Baseline. FINAL PERFORMANCE EVALUATION OF ACT 16 | Page therefore focused on close scrutiny of the allocation of budget funds year by year and by component. This involved a substantial amount of ACT document review including “ACT Spending per Component Through June 2013,” which was made available to the evaluation experts and is attached to this report (see Annex H). The result of this document review and analysis was the preparation of two important charts based on information and data extracted from the documents in question: 1) STTA Days by Component (Annex J); and 2) Cost Breakdown of Experts and Staff (Annex K). All these documents, together with feedback from relevant interviewees, provided the evidence needed which resulted in the following findings. It should therefore be noted here that these findings necessarily include both quantitative as well as qualitative observations although the focus of the project was on the latter. Finding 1 deals with qualitative observations with respect to Components 1 and 2, while Findings 2 and 3 focus on quantitative commentary: 1. Components 1 and 2 focused mainly on processes or steps taken by beneficiaries and implementing partners towards achievement of an end result, i.e. WTO accession, laws passed or amended. Scrutiny of ACT reports and, in particular, the Key Results Framework in annual Work Plans, clearly highlights the extent of the project’s involvement in these areas of intervention. Although actual targets have not necessarily been met, i.e. with regard to business enabling environment legislation, the project anticipates that more progress will be made by the time it is completed (see Annex D, ACT Overall Monitoring and Evaluation Matrix for a detailed analysis of indicator performance). However, as this part of the evaluation was tasked with focusing on a qualitative review rather than a quantitative analysis, ACT can be judged to have been largely successful in having a positive impact on GOAJ’s decision-making apparatus and working practices regarding legislation. ACT’s contribution towards assisting GOAJ with WTO accession support is an example of a successful initiative verified by a number of interviewees but in particular the Ministry of Foreign Affairs (MoFA). On the other hand, a not so positive development was the project’s lack of progress regarding risk management in the Central Bank of Azerbaijan (CBA), although other forms of ACT support were appreciated by the Financial Monitoring Service (FMS), which is part of CBA. 2. Component 3 focused more on a quantitative review of results – there are 11 numeric indicators defined in ACT’s M&E plan. In summary, results vary from “full achievement to date” to “in the process of achievement in the final quarter of the project." However, it is important to note here, that some of the reported results, which have relied on calculations made by the project based on information provided by participants in surveys (farmers and processors), are difficult to verify due mainly to the fact that farmers rarely keep records of their working practices, especially with regard to yields and/or productivity. The initial ACT baseline survey was conducted by ASR Consulting and Research Group. Their final report was subsequently submitted to ACT. That report contains no information on the methodology used to gather the survey data other than the fact that it was a ‘panel survey’ where the same respondents would be targeted in follow-up surveys. No further comment on the methodology employed can therefore be discussed in this evaluation report. However, what is known is that the baseline farmer survey report was designed to establish the initial values of selected PMP indicators for farmers receiving assistance from ACT Component 3 value chain promotion activities in the dairy, pomegranate and aquaculture sectors. Similarly, attempting to verify the number of jobs created as a result of the project’s interventions, is problematic for the following reasons: 1) It is not clear what constitutes a job – full-time or part-time or seasonal?; and 2) A job can be created at a particular point in time but then become redundant – does this count as a job created? In short, job figures cannot realistically be verified and feedback from farmers during the Evaluation Team’s field trips supports the difficulty having this as a measurable parameter. In the absence of the methodology used (see above) in gathering data for the initial baseline farmer survey, it is not possible to comment further on whether Performance Indicator Reference Sheets were used and, if so, whether they were sufficiently descriptive in nature. 3. An analysis of the data examined in Annexes I, J and K appears to highlight discrepancies in the allocation of funds throughout the years of ACT. The Evaluation Team found that although Amendment 3 of the original SOW had tasked the project with realigning activities towards support for agriculture within Component 3 via providing support to BSPs, this appears not to have occurred in practice. Furthermore, Amendment 3 specifically required a significant reduction in the deployment of STTAs in favor of supporting BSPs (in this case local BDS agencies). Again, the project’s own statistics contradict this in reality. Between Year 2 and Year 3, STTA input in terms of person-days remained the same. At the same time, budget expenditure during the same period fell marginally with regard to agriculture activities. In overall terms, by the end of Year 2, nearly $14m of the total allocated of $19m (through June 2103) had already been spent on all components, leaving 27% remaining for Year 3. However, as a significant amount of work supporting agriculture would take place in the final year, it is questionable whether one can conclude that resources were used in a cost-effective way by ACT. The Evaluation Team’s findings with FINAL PERFORMANCE EVALUATION OF ACT 17 | Page respect to the question on STTAs (4.3.1 below) further elaborate on the issue of effectiveness of resources. Conclusions 1. The ACT Project has arguably met its obligations with respect to anticipated deliverables within Components 1 and 2, which for the most part did not require achievement of quantitative indicators (although 14 such indicators constituted part of M&E reporting). The progress made by the project is therefore broadly in line with Amendment 3 guidelines regarding “Systematic Transformation,” i.e. that project interventions are intended to “change the nature of relationships…. through removing barriers and frictions that impede economic activity.” 2. The conclusion reached here is that although Component 3 of the project addressed various numeric indicators as required by Amendment 3 regarding “Enterprise Transaction,” i.e. interventions aimed at individuals or businesses to “directly enhance productivity and income,” there have been mixed results to date in terms of performance and there is little certainty that the reported results are in fact accurate or verifiable, due to the subjective nature of the source. 3. The evidence appears to suggest that ACT did not take adequate account of the requirement in Amendment 3 to reduce the “use of international consultants” in favor of BSPs support in order to increase their capacity to effectively service the project’s ultimate beneficiaries – farmers and food processors. Therefore, cost-effectiveness here remains questionable. Recommendations 1. No specific recommendations suggested other than for USAID to consider the value and potential sustainability of allocating resources to non-measurable project indicators. 2. At project design stage, USAID should consider whether or not indicators can realistically be measured before establishing them. If based on data from secondary sources (i.e. beneficiary records, calculations or recall) usually acquired through surveys, does this help or hinder implementation? 3. USAID oversight of project budget allocation needs to be sufficiently rigorous and robust enough to track performance, identify anomalies in a timely manner, and to be able to advise project management if remedial action is required. There is an overwhelming need for a professional M&E reporting mechanism to be an integral part of a project’s modus operandi. Data gathering and analysis should be in real-time and on-going throughout the lifetime of the project so that performance at any particular point in time can be analyzed and reported upon. In adopting this approach, the cost-effectiveness issue will become self￾evident if deviation from plan occurs. 4.2 KEY INTERVENTION AREAS AND RESULTS AND IMPACTS 4.2.1 Question 1: Assess the progress and impacts of the value chain operations under the ACT Project in terms of: i) technology transfer and the relevance of technologies transferred; ii) strengthening farming advisory and input delivery services; iii) capacity building of post-harvest and processing facilities; and iv) linking producers with markets. Figure 6: QUESTION 1- SUMMARY OF EVALUATION DESIGN AND METHODS Evaluation Question Type of Analysis Conducted Data Sources and Methods Used Type and Size Sample Limitations Assess the progress and impacts of the value chain operations under the ACT Project in terms of : Description – based on content analysis of expert opinions Focus group sessions with farmers in selected regions; ACT PMP; M&E Plan To be organized by Evaluation Team in sample regions – target = 8-10 per focus group session including representatives from all identified participants. Unavailability of participants due to holiday period. FINAL PERFORMANCE EVALUATION OF ACT 18 | Page Figure 6: QUESTION 1- SUMMARY OF EVALUATION DESIGN AND METHODS Evaluation Question Type of Analysis Conducted Data Sources and Methods Used Type and Size Sample Limitations i. Technology transfer and the relevance of technologies transferred; ii. Strengthening farm advisory and input delivery services; iii. Capacity building of postharvest and processing facilities; iv. Linking producers with markets. (Records and evidence of new technology/ies introduced; Financial and sales records; Loans received; new supply orders and contracts signed). (Value chains: (1) pomegranate, (2) dairy, (3) aquaculture; (4) hazelnut, (5) cold storage) Component 3 1. % of farmers, processors, and others who have adopted new technologies or management practices 2. % change in the value of overall portfolios of agricultural loans or leasing agreements made by financial institutions receiving project assistance; 3. No. of new agricultural loans or leasing agreements made by financial institutions receiving project assistance; 4. % change in the value of international exports of the following targeted agricultural commodities as a result of USG assistance etc.) Impact is to be gauged in terms of outcomes and intermediate results. Robust and clearly defined counterfactual is not available to conduct an impact assessment and is outside of the scope of this evaluation Findings The SOW of ACT stated that the overall success of ACT was to be judged according to the degree of enterprise/farm growth achieved, and the improvements in the ability of BSPs to deliver assistance to farms and enterprises in the future. The design of questionnaires distributed to focus groups conducted with farmers and BSPs reflected this requirement. The Evaluation Team reviewed ACT progress reports for the period 2010-2013 (complete list provided in Annex E) and gathered findings during the meetings in Baku and the regions. The focus of research concerned each of the sub-sectors identified by the Contractor35 as exhibiting potential for competitive growth: dairy; hazelnuts; pomegranate; aquaculture and cold storages subject to availability of respondents, documented records and data. General Findings 1. The SOW36 contained a series of performance indicators, which were discussed and agreed in specific terms between the Contractor and USAID as part of the ACT PMP, M&E plan and progress reports. Annex D contains indicators which were included by the Contractor with USAID’s approval in the Overall Monitoring and Evaluation Matrix. Annex D also provides records of the level of achievement against the targets derived by the Evaluation Team from the PMP, M&E plan and progress reports. Due to a number of limitations, the Evaluation Team presumed actual consolidated records provided by ACT to be accurate. The Evaluation Team was not given access to the end-line survey, updated PMP (second half of 2013), and the final ACT report, as all three were work in progress during the evaluation. The analysis presented in the matrix therefore does not include full data for 2013. Furthermore, indicator of percentage change in value of purchases from smallholders was not disaggregated by sector. 2. The SOW noted that only in limited and specialized circumstances should international consultants have been used to deliver assistance to individual or very small groups of businesses. While the Contractor was expected to limit the use of international consultants, the Evaluation Team found no specific guidance in SOW or a plan in PMP or ACT progress reports showing to what extent precisely such limits were to be applied by the Contractor. However, the areas in which such assistance was considered relevant were                                                              35 Included the analysis on 37 sectors concluded by USAID's Private Sector Competitiveness Enhancement Project (PSCEP), in 2008 and 2009. 36 ACT Contract Amendment/Modification No 3, 05/03/2012 (containing revised SOW).) FINAL PERFORMANCE EVALUATION OF ACT 19 | Page left rather broad in the SOW, indicating that the emphasis should have been placed on transferring knowledge and skills of ACT consultants to a wider group of local stakeholders (extension agents, BDS providers, farmers, SMEs, and other public and sector members) across several sub-sectors. 3. The Evaluation Team found that ACT Key Results Framework37 indicates that BDS providers, local STTA and local Long-Term Experts were engaged to deliver the majority of intermediate results in Value Chain development by providing direct assistance to farmers/groups. The framework gives no indication as to what extent international STTA (US Nationals or/and TCN Professionals) were engaged in the provision of direct (to farmers/groups) or indirect (through BSPs) assistance. Specific Findings Technology transfer and the relevance of technologies transferred 4. During the regional visits, the Evaluation Team found that the majority of respondents among farmers/groups received assistance from international STTA mainly in the form of 1-3 days classroom￾based seminars, complemented with a study tour to Georgia. The majority of respondents among farmers/groups rated the relevance and outcome of international STTA as average or low but assessed the relevance and quality of the study tour and ACT local consultants as high. 5. Average assessment was given by respondents to ACT’s assistance in such areas as artificial insemination and calf hatches. This was explained by the fact that farmer training on artificial insemination was largely theoretical, while wider adaptation and installation of calf hatches represented relatively high costs for farmers against intended outcomes. For example, out of 80-90 farmers engaged in calf hatches program, only 6 were able to install the units. E-shepherds38 appeared to have received much greater application in Barda and Agjabedi (15 farmers) and other target areas. The majority of respondents indicated that adaptation of e-shepherd technology led to an average of 180 AZN savings in costs per farmer per month. 6. ACT has been able to provide grants to fish processors in Sheki and Neftchala, helping them to upgrade infrastructure and operational facilities. Girkhbulag trout farm received a grant towards installation of feed processing unit. Wider adaptation of modern trout farming technologies and practices appeared to have led to significant growth of the farm’s yield (100 tonnes in 2012 versus 2-3 tones prior to start of the ACT support) and the reduction of costs. 7. With regard to the use of TCN professionals’ assistance to processors, all respondents among processors (HACCP) rated STTA TCN as highly relevant. STTA TCN professionals and HACCP consulting companies have been instrumental in facilitating acceptance of HACCP importance by enterprises. The respondents also confirmed that STTA and the cost-sharing approach (see Question D4) have led to significant outcomes at enterprises in terms of investment into HACCP adherence. Strengthening farm advisory and input delivery services 8. In consideration of cost-effectiveness of inputs vis a vis envisaged SOW, the SOW required the Contractor to deliver its assistance primarily through local BDS providers. This appeared not to be the case since the majority of farmer training has also included (to a variable extent) direct inputs of international STTA and ACT staff. Assistance provided by BDS providers was also evidently limited to provision of theoretical training of farmers. 9. Component 3 transactional interventions were meant to strengthen BSPs, as well as to enhance the likelihood of sustained benefit after the completion of the project along with improvements to farms and enterprises. The Evaluation Team has recorded variable responses from respondents as regards the relevance and quality of STTA and ACT local consultants’ training provided to BDS providers. Significant variance was observed in responses assessing strength of internal BDS Training-of-Trainers (ToT) and outcomes with regard to competitiveness of BDS to provide better services to farmers. The Evaluation Team has also noted that over 50% of BDS providers’ reports submitted to the ACT Project appeared to be of basic or low quality. The responses are shown in Annex H. Capacity building of post-harvest and processing facilities                                                              37 Year 3 FY2013 October 1, 2012 – September 30, 2013. 38 A system used by farmers to track animals based on electronic technology i.e. using GPS. FINAL PERFORMANCE EVALUATION OF ACT 20 | Page 10. The Contractor has extended highly visible support in the target regions to post-harvest and processing facilities in the hazelnut sub-sector. The Evaluation Team noted during its visit several examples of the cost-sharing approach leading to various investments by processors in better hygiene, food quality and infrastructure. The findings of the Evaluation Team in this area are described in more detail under Question 4.4.4 below. Linking producers with markets 11. In hazelnut and fish farming sub-sectors, the Evaluation Team recorded favorable responses of clients. Respondents at Aznar pointed out several tangible outcomes of STTA provided by US Nationals and ACT local consultants. The management of the enterprise confirmed that ACT had been instrumental in assisting the enterprise with widening international export linkages and securing new contracts for export of pomegranate and other fruit juices. 12. Girkhbulag trout farm reported no change due to ACT’s assistance as regards expansion of links with markets. The farm enjoys active demand for fresh fish locally and continues to supply mostly local clients maintaining steady dynamics of sales. Expanding supplies to a more remote market is currently constrained by lack of HACCP adherence at the farm level, as well as lack of fish processing equipment and weak cold storage, packaging and transportation capacity. 13. The Evaluation Team found little or no evidence of dairy farmers benefiting from better linkages with markets that can add value to their product. While milk yields increased reportedly by 20-25% as a result of the ACT Project’s support, increase in the value of sales currently represents 2-3 qepik per litre for milk collected at farm gate. Conclusions General Conclusions 1. Indicator analysis presented in Annex D - Overall Monitoring and Evaluation Matrix for Component 3, does not allow for a robust independent assessment of targets against level of achievement. Actual parameters indicate that targets were exceeded or under-exceeded but the data provided was not yet fully updated at the time of evaluation. The scope of this evaluation does not allow for an independent and robust verification of reported results. 2. The SOW is vague on the subject of use of direct specialized assistance by international consultants. The Evaluation Team found no evidence as to what extent a portion of STTA dedicated to this Component contributed to the actual results shown in Annex D. The Evaluation Team also found that given such extensive use of STTA prior to the introduction of M&E plan and indicators, the SOW remained vague in terms of criteria for the use of international STTA in Year 2 and Year 3. 3. The Evaluation Team found that the Contractor has continued to extend direct international STTA and ACT consultants’ support to the majority of on-farm activities in Years 2 and 3 regardless of the SOW requirement that such assistance should have been primarily delivered through BDS providers. Specific Conclusions Technology transfer and the relevance of technologies transferred 4. While STTA and ACT local staff TA was designed to be focused on increasing productivity, yields, and sales at farm level, the Evaluation Team found mixed or fractured evidence to be able to robustly gauge the extent to which international STTA in itself was able to deliver relevant technologies or was successful in facilitating actual technology transfer. The Evaluation Team did find clearer evidence that overall in the dairy and livestock sub-sectors STTA by US Nationals was less tailored to the needs of clients than that in the hazelnuts and fish farming sub-sectors. 5. On the basis of the analysis of respondents’ views, the Evaluation Team concluded that the training provided on artificial insemination had achieved little outcome with regard to technological transfer. It appears that the majority of farmers lacked the resources and inputs to apply the knowledge gained in practice. As indicated by all participating farmers, a complementary or follow-on assistance with the establishment of an insemination bank or a pilot insemination program would have been much desirable. No such type of technological support was extended to respondents engaged in dairy/livestock operations in Barda and Agjabedi, which led to the situation where technological transfers have taken place on a minor scale or not at all. FINAL PERFORMANCE EVALUATION OF ACT 21 | Page 6. The Evaluation Team concluded that combination of TA with matching grants enhanced the outcomes of STTA and ACT local staff TA in terms of technological transfers taking place. There was also clearer evidence available that as a result of ACT’s support, fish farmers have recorded an increase in yield and sales. 7. The Evaluation Team concluded that there is clear, although intermediate, evidence in place suggesting that HACCP reforms at enterprises directly contributed to sustaining growth of international exports and increased competitiveness. Strengthening farm advisory and input delivery services 8. The Evaluation Team concluded that there is no clear and robust evidence suggesting that ACT delivered its assistance primarily through local BDS providers as was required by the SOW. In addition, the team concluded that the outcomes of ACT’s support to BDS providers were uneven across the board. Mixed results pointed out that an extensive program of support extended to BDS providers by international STTA and longer term TA by ACT consultants might have not fully achieved intended outcomes. Capacity building of post-harvest and processing facilities 9. Given observed investments by enterprises, there is clear evidence of growing appreciation among processors of the importance of HACCP, particularly in connection with export of produce and as means of boosting competitiveness. Linking producers with markets 10. There is clear evidence available that as a result of ACT’s support, pomegranate and hazelnut processors directly, and farmers/groups indirectly, benefitted from an improved access to international markets. 11. The Evaluation Team concluded that there is no existing evidence to suggest that as a result of the ACT Project, support links to markets has improved or, if yes to what extent, for fish, dairy and livestock farmers. Recommendations In connection with the above findings, the Evaluation Team recommends that: 1. Given the significant scope of ACT and the importance of assessing value for money of assistance, future evaluations should be conducted once final documented evidence (end-line surveys, final report, PMP) has become available. 2,3,4 The SOW and associated budgets should provide clear guidance on the use of STTA. The Contractor should be required to submit forward looking plans, clearly defining areas where STTA is needed and the associated expected outputs. The M&E plan should provide a clear connection between planned inputs, desired outputs, and expected outputs to facilitate monitoring of the Contractor’s performance at all critical milestones against the SOW. The design of future projects should not be process-driven but results-driven. Additionally, Contractor payments should be remitted after the achievement of Key Performance Indicators (KPIs) and subject to the actual delivery of pre￾determined outputs. Specific Recommendations Technology transfer and the relevance of technologies transferred 5,6 Farm-level STTA assistance should ideally be complemented by cost-shared matching grants or fully subsidized pilot projects (depending on practicality) aimed to enhance theory-based training and the practical adaptation of technologies and practices. 7,9 Given the perceived success of HACCP interventions, future USAID/Azerbaijan projects should continue to extend related support to processors, subject to a thorough assessment of their commitment to HACCP adherence and investment potential. Strengthening farm advisory and input delivery services 8. Farm advisory and input delivery services fall within the merits of MoA. National Extension Service and agro-scientific centers provide the appropriate vehicle for extending such assistance to farmers. USG support to BDS providers, the primary vehicle of delivering assistance to farmers, should be FINAL PERFORMANCE EVALUATION OF ACT 22 | Page reviewed because partnerships between leading agro-scientific centers, BDS providers, farmer associations, and processors are likely to build more sustained results. Capacity building of post-harvest and processing facilities This is covered by recommendation No.7 above. Linking producers with markets 10. Future USAID projects should continue to build on ACT’s successful assistance to the pomegranate and hazelnut sectors through further farm- and processor-level technology transfers and further HACCP reform extension. Registering the Hazelnut Association officially will consolidate hazelnut processors around Zagatala, thus reinforcing the importance of comprehensive support to farmers for sustaining equitable yields, productivity and sales. 12,13 Any future support to dairy and livestock farmers, including the use of international STTA, must be thoroughly reviewed in favor of TCN and local professional assistance. Future interventions should aim to surpass simply the provision of generic farm trainings in order to achieve equitable value added outcomes for participating farmers. 4.2.2 Question 2: Assess the range of policy reform efforts supported, progress and the status (e.g. legislative status for each reform supported) and priority reforms to be supported in the future. Figure 7: QUESTION 2 - SUMMARY OF EVALUATION DESIGN AND METHODS Evaluation Question Type of Analysis Conducted Data Sources and Methods Used Type and Size Sample Limitations 2. Assess the range of policy reform efforts supported, progress and the status (e.g. legislative status for each reform supported) and priority reforms to be supported in the future Description – based on content analysis of expert opinions Key informant interviews with key personnel in relevant government ministries; ACT progress reports; PMP and M&E plan. (President Office – Legal Department; Office for Cooperation in South Caucasus. SDC, SECO; American Chamber of Commerce etc. and WTO – as per Question 4.2.3 Interviewees identified by Evaluation Team and USAID. Components 1,2 Overall legislative and regulatory reform (Business Enabling Environment; Trade and Agriculture etc.) and inter-governmental policy reform cooperation. None perceived Findings To adequately address this question responses are sub-divided under two headings – Relevance and Sustainability. Relevance During meetings and interviews with the Evaluation Team, ACT stakeholders raised a number of issues related to relevance, including the extent to which USAID has responded to the needs of its intended beneficiary institutions. More specifically, the Evaluation Team assessed the extent to which ACT relied on the contribution of its implementing partners to ensure the successful implementation of project goals. 1. The Evaluation Team noted the following as a result of interview responses:  Most respondents agreed with the statements that the intervention “matches closely the needs of the government stakeholders in which it is implemented” and that “the project matches closely the needs of the country as a whole” – (see Annex E for full list of interviewees).  Most respondents stated that interventions were demand driven (i.e. reflected expressed needs of beneficiaries), meaning that government beneficiaries (MoED, MoF, CBA, SCSMP, MoH) were consulted on the need for the project, its content, and delivery. FINAL PERFORMANCE EVALUATION OF ACT 23 | Page  Most respondents agreed that “the work of the project has included substantive interaction/ collaboration with similar projects/activities of World Bank, GIZ in Azerbaijan.” 2. Regarding issues raised in meetings with USAID/Azerbaijan staff, the Evaluation Team elicited commentary from representatives of the GOAJ (CBA, MoF, MoED, SCSMP, respectively) that highlighted the issue of the visibility of the ACT Project among beneficiary government agencies. The senior official from SCSMP indicated that their office (which is the primary interface in the GOAJ dealing with the ACT Project’s work on the WTO) had very little information about the ACT Project conference on WTO trade barriers that was planned for July 5, 2013. Furthermore, his department never received any project reports. In the case of MoF, officials stated that some in the Ministry were upset regarding the model for calculating macroeconomic indicators for forecasting purposes, which the ACT Project gave to MoED but did not share with the MoF. This was considered as a lack of coordination in this particular project activity. 3. The positive responses by the WB and GIZ regarding donor cooperation with the ACT Project, especially in the context of WTO accession, confirms the project’s achievement in making progress towards improving the legislative process. As a result, Azerbaijan is now far closer to WTO accession than would otherwise have been the case (stated by a senior official of MoED). However, at the highest levels of decision-making, political will is still necessary to complete outstanding issues (political and legal reforms, and to finalize bilateral negotiations mentioned by MoED) which directly relate to carrying out the radical institutional reforms that have still not started in the country. In short, the GOAJ still appears to have some way to go in initiating institutional changes at all levels of administration and management. This remains a significant obstacle towards adopting the Competitiveness Code, Tax Code, Custom Code, Technical Regulations Law, and Food Products Law that were modified by the ACT Project team but still not approved by GOAJ, and developing new regulatory frameworks with clear mechanisms for implementing the legal reforms. 4. ACT is stimulating the knowledge, skills and development of capacity in a number of research institutions, such as the Economic Research Center (ERC) which focuses on the cost-benefit analysis methodology and models, as witnessed through the growth in the number of researchers specializing fully or partly in this field. However, as a MoED senior official indicated, these knowledge and skills are not required by the government and almost no key stakeholders are interested in applying project analysis in their day-by￾day activities. Moreover, there are no special regulations which provide a mandate for the stakeholders to assess an investment project through cost-benefit analysis. 5. Finally, the success of reform depends on the quality and availability of human capital. However, there is no correlation between outcomes of the education sector and the labor market in Azerbaijan. There are sectors of the economy which require new specializations, such as procurement specialists. Sustainability The ACT Project’s training programs – national and regional programs which bring together officials from almost all four mainline ministries, as well as local agricultural entrepreneurs and farmers specialized in dairy, pomegranate and fruits/hazelnuts production – form a core part of its technical assistance in several ways. The programs are related to key ACT Project aims of promoting the domestic business enabling environment and developing a liberalized and more transparent trade environment. According to interviewed stakeholders, these training programs transferred necessary knowledge and skills for improving trade and business environments (ACT reports that trainings increased agricultural productivity by 20-25%) and program results reflected effective coordination of stakeholder involvement in trainings. A significant weakness, however, appears to have been these programs’ lack of outreach, which in turn appears to have limited their effectiveness. 6. In addition to the networking opportunities afforded in the range of conferences and other events, project participants from different organizations confirmed that there has been quite an active engagement among project participants and, to some extent, across projects. Furthermore, project participants and ACT Project staff reported a good deal of contact and discussion regarding project work and findings. Conclusions 1. The overall positive response regarding progress towards policy reform supported by the ACT Project could be achieved fully if these efforts are implemented in a more consistent manner in order to maintain a balance of interests between key stakeholders and their coalitions and are focused on rationalizing the FINAL PERFORMANCE EVALUATION OF ACT 24 | Page proportion of interventions at the national and local levels. From the feedback received, and from reviewing intervention documentation, it is clear that the issue of relevance is afforded a great deal of emphasis through comprehensive needs assessments and consultation procedures, and has helped to ensure a close match between the priorities of the target beneficiaries and the local areas in which projects are implemented. 2. ACT has provided little information for beneficiaries about their work plans; some stakeholders, such as SCSMP, never received ACT Project reports. 3. Political will is necessary to complete outstanding issues (political and legal reforms to finalize bilateral negotiations mentioned by MoED) which are directly related to carrying out institutional reforms and developing new regulatory frameworks. 4. Since 2005, the GOAJ has spent huge investments in financing infrastructure projects without economic/financial expertise. There is no regulation for applying cost-effectiveness techniques, including methodologies like cost-benefit analysis. 5. The education sector could be supported to prepare necessary specialities such as public procurement, privatization, securities, investment, etc. 6. The networking that has been achieved by the ACT Project is a valuable asset for the sustainability of future USAID interventions. Recommendations 1. The future ATLAS project should continue coordination with the World Bank ASIP project and other international donors. The balance of coalitions’ interests should be considered to encourage inter￾ministerial cooperation and coordination. 2. Transparency, through information sharing and reporting, is the most important for any Contractor and should be supervised by USAID through permanent meetings with stakeholders. 3. None specific as USAID can have little or no influence over ‘political will’. 4. Software, models, and methodologies for project expertise provided by USAID funds are very useful and practical. However, involvement of these innovations should be requested by GOAJ. 5. No specific recommendation for the ATLAS project regarding the issue of education unless it is considering such an intervention. 6. The ATLAS project should continue with workshops by using the network which already exists. 4.2.3 Question 3: Evaluate the progress made in Azerbaijan’s accession to WTO and further support to be extended. Figure 8: QUESTION 3 - SUMMARY OF EVALUATION DESIGN AND METHODS Evaluation Question Type of Analysis Conducted Data Sources and Methods Used Type and Size Sample Limitations 3. Evaluate the progress made in Azerbaijan’s accession to WTO accession and further support to be extended. Description – based on content analysis of expert opinions Key informant interviews with key personnel in relevant government ministries. (WTO Department, MOED; WTO Accession Commission, MFA; Copyright Agency; Ministry of Agriculture; Ministry of Finance State; Ministry of Health; Tariff Council; Committee on Standardization, Metrology and Patent; American Interviewees identified by Evaluation Team and USAID. Component 2 (Consumer seal program, HACCP; policy reform; Sanitary and phyto sanitary control; TBT/SPS; Tariffs Council; Policy Seljan Verdiyeva – Attaché (currently in Geneva) FINAL PERFORMANCE EVALUATION OF ACT 25 | Page Figure 8: QUESTION 3 - SUMMARY OF EVALUATION DESIGN AND METHODS Evaluation Question Type of Analysis Conducted Data Sources and Methods Used Type and Size Sample Limitations Chamber of Commerce; Independent Consumers Union). WTO Working Group notes and records reform Findings 1. Feedback received from meetings with governmental organizations suggests an overall perception that the ACT Project has achieved its aim of assisting GOAJ adapt and adopt legislation that satisfies WTO requirements. This is an on-going process. 2. Although beneficiaries expressed satisfaction at receiving draft laws, such as the Custom Code, Tax Code, Technical Regulations, Accreditation, Standardization and Food Products, which were prepared and/or changed through the support of the ACT Project, the GOAJ has not adopted these particular laws to date. Furthermore, there are additional actions that could help accelerate the WTO accession process such as a technical conformity assessment, export-import trade off, food safety control, and certification of food products, but ACT has not been actively involved. Conclusions 1. The ACT Project appears to have succeeded for the most part in influencing GOAJ towards making significant progress in completing the required package of reforms demanded by the WTO. 2. The technical measures achieved are not fully complete, especially those related to food safety, which require improvements in the government control system from customs to internal markets. Recommendations 1. Mainline ministries and other government bodies involved in the WTO accession process should be reorganized and institutionally strengthened. Whether this can be included in the ATLAS project is a matter for USAID. 2. Having basic laws already drafted by the ACT Project, the GOAJ now needs the specific regulatory framework for WTO accession. In the remaining time left for ACT, it would be useful if the project considered adding a list of key regulations for inclusion in the draft legislation (i.e. government roadmap). 4.3 EFFECTIVENESS OF TECHNICAL ASSISTANCE 4.3.1 Question 1: How effective was STTA provided by Contractor in meeting the needs of the project clientele? Assess the impact of the STTA on the beneficiary in terms of relevance of STTA recommendations and their adoption. Figure 9: QUESTION 1- SUMMARY OF EVALUATION DESIGN AND METHODS Evaluation Question Type of Analysis Conducted Data Sources and Methods Used Type and Size Sample Limitations 1. How effective was STTA provided by Contractor in meeting the needs of the project clientele? Assess the impact of the STTA on Description – based on content analysis of expert opinions Key informant interviews with key personnel in relevant government ministries (MOED; MFA; MoF; MoA, other partners, BDS, CSOs and Interviewees identified by Evaluation Team and USAID Components 1,2,3 Opinion bias of interviewees as regards semi-structured and open ended interviews. Impact is to be gauged in terms of outcomes and intermediate results. Robust counterfactual is not available and outside the scope of this evaluation. Relevance of STTA to the needs is to be assessed in this case. FINAL PERFORMANCE EVALUATION OF ACT 26 | Page Figure 9: QUESTION 1- SUMMARY OF EVALUATION DESIGN AND METHODS Evaluation Question Type of Analysis Conducted Data Sources and Methods Used Type and Size Sample Limitations the beneficiary in terms of relevance of STTA recommendations and their adoption. farmers (Value Chains). Output and outcomes is to be assessed in terms of (i.e. Legislation amended; Value Chains adoption of technologies, methods, practices etc.) and to gauge outcome of the STTA in real terms. Findings In order to meaningfully evaluate the contribution made by STTA experts (local and international) with respect to the impact they had on the ACT Project’s beneficiaries, a useful starting point would be to acquire an understanding of where, when and how STTAs were allocated throughout the lifetime of the project. The Evaluation Team, therefore, prepared a detailed summary of such data (see Annex J – ACT-STTA Days by Component) which provides a clear picture of the distribution of STTA days by component over the three years of implementation. Through analysis of this information, it was possible, to some extent, to provide a correlation between STTA input and the impact of their activities on clientele. From this, it was then possible to comment on the STTAs’ effectiveness based on the information gathered from direct one-to-one discussions with beneficiaries and via farmer group focus sessions. This calculation is easier to comment upon with respect to Component 3, where feedback from BDS providers, farmers, and food processing companies can confirm whether or not STTA activities (actions plus actual time spent with clients) made any significant or practical difference to their operations resulting in improved performance. It is also possible to comment on STTA activities within Components 1 and 2, although effectiveness will be harder to assess as the end result of their input might only transpire post-project, i.e. WTO accession or draft legislation. Furthermore, Components 1 and 2 were mainly the focus of a qualitative review, given that emphasis was on processes and procedures leading to changes in working practices and mind-sets. The following findings are therefore more aligned to Component 3 and specifically only to STTA impact which is the underlying theme of this question. 1. Analysis of the data in Annex J indicates that, to date, a total of 4,842 days have been allocated to STTAs (3,949 international and 893 local). In addition, STTA input for agricultural activities accounted for 1,541 days, or 32% of the total STTA days allocated. Furthermore, this STTA input only occurred in Years 2 and 3 as a result of Amendment 3. The level of input from international STTAs remained the same in Years 2 and 3, although Amendment 3 clearly directed that this should be reduced in order to support local BSPs. From the field trips conducted by the Evaluation Team, which covered all regions supported by the project, there is little evidence to suggest that STTA effort was effective, in real terms, in assisting with achieving M&E indicators or having an actual impact on the beneficiaries’ activities. Finally, in considering that, collectively, international and local STTAs accounted for nearly $10m over the three years (roughly 44% of the entire ACT Project budget), it is questionable as to whether STTA input has been effective to any great degree. This finding is based on feedback from various beneficiaries mentioned above during interview sessions. However, it is possible that the results of ACT’s anticipated endline survey (not available to the Evaluation Team at the time of the evaluation mission) may redefine this finding based on the data provided in that survey (notwithstanding the team’s earlier observation regarding the veracity of such data). 2. It seems that the shift in focus towards supporting BSP’s (in this case BDS’s) as directed in Amendment 3, has not in fact resulted in their businesses becoming sufficiently sustainable for them to be in a position to effectively service their clients (farmers and food processors). This finding is supported by the following issues raised by BDS providers interviewed:  Technical equipment provided was in some cases second hand and, in others, did not properly function (computers and printers).  Information provided was often not up-to-date and/or irrelevant (particularly related to dairy production).  Materials provided, such as printed brochures, were often difficult to understand due to their highly technical nature. FINAL PERFORMANCE EVALUATION OF ACT 27 | Page  The amount of time STTAs spent in the field working with clients was too short (1-3 days) which reduced the effectiveness of their work in some cases (e.g. dairy and hazelnut producers). Conclusions 1. To some extent, it is difficult to be conclusive about the effectiveness of STTA input as their involvement largely occurred following Amendment 3. The data contained in Annex J prepared by the Evaluation Team provides evidence but not a verdict. One of the problems encountered is that much of the commentary obtained from the field is either anecdotal or not supported by documented evidence i.e. sales figures, productivity analysis, etc. Although the project conducted farmer and processor surveys, these (as discussed elsewhere in the report) cannot necessarily be relied upon either due to the subjective nature of the data provided, or subsequent calculations that have to be made where percentages are involved. A further point in question stems from the ability of the local experts to offer advice and training to ACT clients. Evidence from the field appears to indicate that they did not have the requisite skills or know-how to either train the BDS providers or practically and professionally advise farmers. Therefore, the best that can be said is that STTAs in some cases provided useful input to some clients, while in other cases their contribution was not deemed to be valuable or relevant. 2. Another conclusion reached by the Evaluation Team concerns the management of services provided by the project regarding STTAs. There appears to be a lack of any coherent plan regarding the allocation of STTAs. Experts were allocated on a case-by-case basis but, for the most part, were based in Baku and not in the regions, which seems to have been a point of contention among beneficiaries regarding their availability. This seems puzzling given the resources allocated to STTAs from the ACT Project’s budget. It is therefore possible to conclude that this lack of planning, together with a lack of guidance for local experts, has led to their interventions not being as productive or effective as they could have been. While training workshops and seminars in the field were generally appreciated, they appear not to have been as practical as clients anticipated, forcing the conclusion that STTA input was not necessarily based on the needs or requirements of the intended audiences. Recommendations 1. From the above findings and conclusions, there is a strong case for USAID to seriously consider the way in which budget allocations for short-term experts – whether local, international or third country nationals – are calculated. Furthermore, there must be some logic as to the purpose of their planned interventions in terms of how they relate to project indicators, outcomes, and impacts. The results of their input should be measurable and directly related to the specific needs of target audiences. In practical terms, experts should be knowledgeable and have the necessary motivational and technical skills to make things happen. Hosting seminars and workshops for farmers, while useful, can often lead to “training fatigue” as participants in many occasions also attend similar such sessions with other donor-funded projects (USAID is not the only player in the field). This is highly relevant for the ATLAS project as the EU has commissioned two further projects in support of agriculture.39 Although not exclusively focusing on agriculture, they will both support rural businesses and entrepreneurs, and also the capacity of local authorities and service providers. 2. Projects of this nature should, at the design stage, have a clear strategy for the selection and deployment of STTA consultants, including budget allocation and number of work-days relative to planned results and impact. STTA services should be provided based on clear terms of reference and/or the development of action plans and follow-up procedures. Technical assistance to be provided should match the real needs of beneficiaries, materials provided should be presented in a format that is easily understood, and suggested assistance should be relevant and practical to adopt. STTA time-management should also be looked at more carefully, together with a review of expected outcomes and impacts on clients as a result of their interventions. In short, STTA should be provided following clarification with the respective GOAJ bodies via mail, phone communication or personal contact, after which, if the TA requests are consistent with project goals, TOR’s with specific deliverables and period of performance would be developed. End-of￾mission interviews would then be conducted with STTA’s and their field trips filed.                                                              39 1) Rural Development Support Programme - EUR 19.5m and 2) Support to Regional and Rural Development in Azerbaijan – EUR10m FINAL PERFORMANCE EVALUATION OF ACT 28 | Page 4.3.2 Question 2: How effectively has ACT created awareness about the issues it worked on among the private scetor, donors, civil society and government agencies in Azerbaijan? Figure 10: QUESTION 2- SUMMARY OF EVALUATION DESIGN AND METHODS Evaluation Question Type of Analysis Conducted Data Sources and Methods Used Type and Size Sample Limitations 2.How effectively has the ACT created awareness about the issues it worked on among the private sector, donors, civil society and government agencies in Azerbaijan? Description – based on content analysis of expert opinions Key informant interviews with key personnel in relevant government ministries, CSOs, private sector partners and farmers. M&E reports (PR; visibility; media) & records Interviewees identified by Evaluation Team and USAID. Components 1,2,3 Opinion bias of interviewees as regards semi￾structured and open ended interviews Findings 1. In general, the Evaluation Team found that most interviewees were aware of the existence of the ACT Project but not necessarily of its main objectives (comments from EU Delegation and Swiss Cooperation Office, Swiss Embassy, among others). During meetings with respondents from the government and private sector, it also became clear that some, in fact, confused the ACT Project with one or more other similar on-going programs with whom they were cooperating. 2. Approximately half of BDS interviewees advised that they had not been equally informed about the project’s activities or had participated from the beginning on planning, implementation, monitoring or follow-up of those activities. Comments by some of the BDS providers suggested that their basic needs and/or suggestions had not been taken into consideration by the project, including, for example, their requirements regarding agricultural equipment, seeds for demo plots, etc. 3. In terms of meeting its indicator targets apropos awareness raising activities, the project appears to have succeeded already or is on course to achieve them during the last quarter (see Annex D – ACT Overall Monitoring and Evaluation Matrix). To date, these activities have included media campaigns, various bilateral, pluri-lateral and multilateral meetings with GOAJ and international organizations, and USG￾supported trainings for improving trade and investment, productivity, and management practices. Conclusions 1. The main conclusions reached regarding overall awareness is that the ACT Project was not active enough in establishing closer links with major stakeholders at the start of the project, or creating a sufficiently high enough profile aimed at informing the donor community, in particular. These appear to have resulted in a lack of cooperation and information sharing among key players. In short, it seems reasonable to conclude that the ACT Project did not have a clear communication strategy in place for awareness raising initiatives. 2. There is sufficient evidence to conclude that as a result of miscommunication between the project and BDSs regarding the precise nature of the project’s activities, the ultimate beneficiaries of ACT Project support were not duly informed of the type of support that would be available and when. This was particularly evident in the regions where farmers, for example, were not fully aware of the project’s selection criteria for applicable interventions. This lack of awareness of what the ACT Project was offering the BDS providers, farmers, or food processors means that, in all likelihood, local experts employed by the project were not sufficiently informed of what the project could deliver. 3. From the data gathered from project reporting documentation, it appears that the ACT Project has satisfactorily met its contractual obligations although, given the above findings and conclusions in some instances, it is difficult to reconcile awareness raising activities with impacts on intended beneficiaries, especially in the regions. FINAL PERFORMANCE EVALUATION OF ACT 29 | Page Recommendations 1. One of the most effective means to ensuring close cooperation between technical assistance projects such as ACT and key stakeholders in a host country is to design, plan and conduct an information gathering survey (particularly aimed at donors) immediately following the start of a project. This should serve to ensure that common goals are agreed upon regarding the implementation of respective projects with a view to avoiding duplication of activities and/or confusion among targeted beneficiaries receiving support. 2. In any future similar project, where intermediaries such as BDS providers are supported through capacity building measures in order for them to become sustainable and able to service clients such as farmers and food processors, it is vital that they, together with experts engaged by the project, develop and initiate effective awareness raising campaigns aimed at all stakeholders and beneficiaries. This is important not least to avoid unnecessary confusion among intended beneficiaries regarding what kind of support is available and for whom. This process should begin when implementation commences so that “voices are heard” and effective communication is established. 3. By working more closely with the media, government agencies, and other stakeholders, projects such as ACT could spread their messages either by holding regular press briefings, issuing printed press releases, or setting up online databases that could be used as information centers. An integral part of any future project design should include a range of targeted awareness raising campaigns aimed at specific audiences with a view to establishing clear and open channels of communication between project and audience. 4.4 SUSTAINABILITY OF THE ACT PROJECT 4.4.1 Question 1: Assess the extent of stakeholder ownership of interventions undertaken by ACT project (provide examples) Figure 11: QUESTION 1- SUMMARY OF EVALUATION DESIGN AND METHODS Evaluation Question Type of Analysis Conducted Data Sources and Methods Used Type and Size Sample Limitations 1. Assess the extent of stakeholder ownership of interventions undertaken by ACT project (provide examples) Description – based on content analysis of expert opinions Key informant interviews with key personnel in relevant government ministries, partners and farmers Interviewees identified by Evaluation Team and USAID Components 1, 2, 3 On-gong legislative and regulatory reform for Business Enabling Environment; on-going WTO accession; inter￾governmental cooperation; Public Private Dialogue; Public Consultation; M&E Framework for Policy Implementation in place and functioning etc.); Government support to Value Chain development. Limited effective means of verifying responses Findings Following discussions with USAID/Azerbaijan, it was agreed that “ownership of intervention by the stakeholders” should refer to the level of involvement of local experts in GOAJ towards achieving outcomes and goals of the ACT Project. More specifically, the evaluation needed to consider the extent to which such experts were involved in drafting legislation in collaboration with the project’s STTAs, as well as whether the organizations to which they belong would take the lead in continuing reforms post-ACT Project. In the course of reflecting on this issue, the Evaluation Team found the following as a result of feedback from beneficiaries and implementing partners involved in the preparation and drafting of legislation or amendments to existing laws. 1. There is clear evidence that ACT Project STTAs worked closely with internal ministerial staff involved in the drafting of legislation. This is supported by comments made by most interviewees, according to whom no legislative drafts or amendments would be considered without the active involvement of GOAJ in-house experts, given the centralized governance of Azerbaijan. In reviewing the ACT Results Framework tables in each of the project’s yearly Work Plans, it is apparent that a significant effort has gone into the development of draft laws, guidelines, preparation of seminar materials, and organization of roundtables by FINAL PERFORMANCE EVALUATION OF ACT 30 | Page the project team, particularly within Component 1. In many instances, the implementing partner initiated the request for assistance from the project particularly with regard to providing experts (local and international) with relevant experience on specific issues (notably with regard to WTO accession, see Finding 2, below). Comments from the State Procurement Agency and the MoED confirm the importance and relevance of ACT’s support in the legislative reform process. Furthermore, the State Veterinary Service (HACCP guidelines), State Phytosanitary Control Service, MoH (guidelines on food sampling) and MoF (budget revenue model to improve forecasting), have all benefited from joint collaboration with ACT. 2. ACT also worked closely with the MoFA, State Procurement Agency (SPA), and MoED on WTO accession issues. SPA was particularly impressed with the ACT Project experts’ input on assisting key personnel on answering the WTO questionnaire for submission to Geneva. Positive comments were also made regarding the experts’ work in acting as catalysts between SPA and other ministries to approve the legal text of amendments. E-procurement was provided as an example of this collaboration. The ultimate result of this on-going collaboration between the ACT Project and beneficiaries and implementing partners appears to be that the latter have been sufficiently motivated by the project’s assistance to have ‘bought into’ the need and rationale for change in existing working practices. Furthermore, most interviewees agree that, to some extent, internal staff and experts in GOAJ departments now feel better equipped to address issues related to the preparation of drafts and amendments and, in many cases, possess the necessary tools and know-how to follow through initiatives post-project. 3. According to the Cabinet of Ministers, its collaboration with the ACT Project was mixed. They were happy with the project’s work on the Law on Technical Regulations, which was professionally prepared and ready for forwarding for its first reading. On the other hand, they were not very impressed with the ACT Project’s work on food safety controls and a certification system for food products legislation, which they felt had not been adequately prepared or presented. In addition, it was felt that the ACT Project had not involved key ministries such as MoED, MoH, or SCSMP during its deliberations. According to interviewees, this resulted in GOAJ not being able to agree with ACT’s proposals which were rejected. At the time of writing, an advertisement was circulating in international circles for two experts (funded by the EU) to support the legislative process in Azerbaijan with respect to food safety control and plant origin. The importance of this discovery lies in the fact that the Cabinet of Ministers is the highest executive authority in the country through which all legislation must pass. In terms of taking a lead, the Cabinet of Ministers does not prepare legislation – it only reviews what it received from line ministries and provides feedback following review. Conclusions 1. Available evidence suggests that ACT has successfully contributed towards key staff of beneficiaries and implementing partners playing a significant role in their respective positions with regard to the preparation of draft laws and amendments in collaboration with project experts. While it is not possible to conclude that every such initiative has resulted in 100% “ownership,” it is certainly safe to say that progress has been made in building the capacity of GOAJ staff and in-house legal experts regarding processes and procedures. In some cases, it is clear that further capacity building will be necessary in order to ensure continued progress in legislation preparation, as indicated by the MoFA. 2. ACT’s support for WTO accession was clearly acknowledged by all concerned parties as a successful series of necessary and important interventions. The project has therefore fulfilled its brief to enhance ‘progress towards advancing WTO accession’ although this will not happen until after the end of the project. 3. It is possible to conclude that insufficient research went into some of the legislative proposals submitted by ACT that came under the scrutiny of the Cabinet of Ministers. Recommendations 1. No specific recommendations as feedback from targeted beneficiaries/implementing partners was largely positive and supportive of ACT’s initiatives. However, some commentators suggested that any follow-on USAID project should focus some attention on legislative reform specifically aimed at SMEs. 2. If applicable to the planned USAID/ATLAS project in Azerbaijan, continuing the work towards WTO accession would be worthwhile to include. In addition, further work on agricultural subsidies might be useful – further research is needed according to MoFA. FINAL PERFORMANCE EVALUATION OF ACT 31 | Page 3. When preparing draft legislation, consider (and ensure) the need to involve all concerned state authorities with an interest in the proposed legislation. If not, it may not pass the scrutiny of the legal department within the Cabinet of Ministers. 4.4.2 Question 2: What processes, systems and management were put in place to ensure that the results and impact of ACT Project will be sustainable? Figure 12: QUESTION 2- SUMMARY OF EVALUATION DESIGN AND METHODS Evaluation Question Type of Analysis Conducted Data Sources and Methods Used Type and Size Sample Limitations 2. What processes, systems and management were put in place to ensure that the results and impact of ACT project will be sustainable? Description – based on content analysis of expert opinions Key informant interviews with key personnel in relevant government ministries; ACT staff. Interviewees identified by Evaluation Team and USAID Components 1, 2 (Legislative and regulatory reform for Business Enabling Environment; WTO; inter￾governmental cooperation; Public Private Dialogue; Public Consultation; M&E Framework for Policy Implementation etc. Component 3 Sustainability of assistance to Value Chains Opinion bias of interviewees as regards what is perceived ‘result’ versus ‘impact’. Impact is to be gauged in terms of outcomes and intermediate results. Robust counterfactual is not available and outside the scope of this evaluation. Findings The Contractor has rigorously pursued the approach of putting in place processes, systems and management arrangements in partner and beneficiary organizations. The Evaluation Team noted several examples of this approach, as reported by the Contractor in the ACT Project progress reports. During fact-finding meetings in Baku and regional trips, the Evaluation Team made every effort to verify ACT Project data directly with partner and beneficiary organizations and farmers/groups. The Evaluation Team’s interpretation of processes, systems and management did not include training activities themselves but rather evidence of change that occurred as a result of training and TA. The Evaluation Team therefore looked for such evidence occurring in the form of longer-term institutional and farm-based arrangements, any technology transfers that took place, installations (ICT, demo plots, etc.), and capacity measures that are very likely to have had impulse effect for each of the 3 Components. As per the limitations described in the evaluation design, the Evaluation Team was not in a position to verify all of the records of achievements reported by the Contractor. Key findings of the Evaluation Team include: Component 1: Domestic Business Environment Improved 1. In cooperation with the American Bankers' Association, ACT designed and installed a computerized bank simulation program for CBA. This system complemented a professional training program offered to a group of Azerbaijani banking supervisors. The Contractor installed and tested the operation for the activity-based management system. Training was provided for the operation and maintenance of the model. Assistance was also provided in the development of the Office of Risk Management through development of policies, procedures, and IT frameworks jointly with CBA. 2. ACT assisted the Center for Research and Development (CRD) to adopt new Keynesian macroeconomic models and estimation and simulation techniques for Dynamic Stochastic General Equilibrium (DSGE) models. It also assisted CRD in arriving at a comprehensive game plan for acquiring the necessary software (Matlab, Eviews, and RATS). 3. The Contractor also put processes in place with maintenance and improvements of the e-learning system for the Financial Monitoring Service. Component 2: Liberalized, More Transparent Trade Environment Developed 4. ACT worked to build the managerial and technical capacity of the following state bodies: Copyright Agency, Phytosanitary Service, and State Epidemiological Service. In addition, the project advanced FINAL PERFORMANCE EVALUATION OF ACT 32 | Page harmonization with Codex Alimentarius and with the International Plant Protection Convention (IPPC) International Standards for Phytosanitary Measures (ISPMs). 5. Per a request from the Ministry of Information Technologies and Telecommunications (MCIT), the project launched the preparation of an interconnection model consistent with the WTO Telecom Reference. 6. Systemic assistance was extended to MoED to prepare draft replies to over 220 questions from the US, EU, and Chinese Taipei, and to assist Azerbaijan with holding bilateral meetings in Geneva with Canada, South Korea, India and Taiwan on goods and services. The Contractor also launched the processes of:  Preparing ACC/8-TBT Conformity Checklist on the basis of Technical Regulation Law.  Updating ACC/9-TRPS Conformity Checklist.  Updating Annex 4 of ACC/1 – Customs Valuation Checklist.  Preparing the WTO GPA/35-Government Procurement Checklist.  Developing institutional mechanisms for the Ministry of Communications and Information Services.  Establishing the basic framework and tools for applying the law on anti-dumping, countervailing, and safeguard measures (trade remedies law). Component 3: Targeted Agricultural Value Chains Improved 7. The project worked to strengthen the technical and managerial capacity of BDS providers to deliver technical and administrative knowledge and skills. This was attempted by printing and distributing industry manuals, and working with the project and partner organizations’ local experts and agronomists. 8. Demonstration plots were established in several target areas to complement the ACT Project TA for apples, aquaculture, pomegranate and dairy. For example, a pomegranate GAP demo plot in the Bigir village in the Goychay region advanced pomegranate farming techniques and practices in the country. 9. The ACT Project also facilitated a change in management practices by assisting with the creation of business linkages and relationships, especially in the pomegranate and hazelnut sectors. Marketing plans for about 12 companies with high potential for export were prepared. 10. Databases of potential customers to contact many international buyers were developed with promising leads in Korea, Japan, China, Brazil and US. 11. The project worked with financial institutions (FIs) including four commercial banks (Demir Bank, Turan Bank, Bank Republika and AG Bank), a non-banking financial institution (AzerCredit), and two financial sector associations, AMFA and ARPA, to improve and expand lending to agriculture. The Evaluation Team concluded that efforts in this area have been a success as some of the banks have begun to officially roll￾out the new ag-lending products to all the branches as the pilot branches provide positive results. Other banks are increasing the number of pilot branches and hiring new staff to work exclusively according to the new technology communicated through project financial experts. 12. The Contractor developed strategic and marketing plans for the ESDER hazelnut processing company, including the establishment of a sustainable input supplement, promotion of products in international markets, and improvement of quality and safety of products in line with international standards. 13. The Vugar Hagverdiyev hazelnut processing company benefitted from strategic and marketing plans which focus on improving management capacity and human resources, and include a strategy for meeting international quality standards. The marketing plan also includes improvements in packaging and promotion of products in domestic and international markets. 14. ARAZ Business Center linked 62 farmers with pesticides distributors, EMA and ASIF Agro, to purchase chemicals recommended by the BDS agronomist, who also provided instruction on the proper use of pesticides and fertilizers. 15. Azsertifika implemented HACCP in six beneficiary companies: Orelay in Agdash, Interpak in Ganja, Azerstar in Zagatala, Canub-Agro in Lankaran and Ikar.S and Fresh Fruit in the Guba and Khacmaz regions. Several companies responded to advice by investing in the reconstruction or building of various facilities adhering to HACCP standards. AQA implemented HACCP in one beneficiary company, Sahliali (please see Question D4 appraising the role of cost-sharing approach in terms of creating ownership of the FINAL PERFORMANCE EVALUATION OF ACT 33 | Page beneficiaries and sustainability). Cost-sharing approach has been appreciated by consulting companies and processors. 16. Demir Bank successfully implemented its new application and overall risk-based lending strategy in all branches for loans below AZN3000. AG Bank hired three new loan officers to work exclusively in agro￾lending in the Shamakha Branch as part of the product rollout efforts to other branches. Cost-sharing approach has been appreciated by banks. 17. Systemic support was extended with the completion of the package of required documents for both the hazelnut and pomegranate associations, which were submitted to the Ministry of Justice (MOJ) for formal registration. In anticipation of formal registration, the management and members of the associations have been provided with necessary training on legal management of the associations, fundraising, financial management during the reporting period and sustainability. Conclusions 1. While it is premature to assess the outcomes or impacts of ACT operations, the findings indicate that the Contractors’ effort aimed at delivering a mostly robust approach to ensure sustainability has started yielding some intermediate results. Given the content of data gathered, the Evaluation Team concluded, however, that the results are more clearly evident in the private banking, hazelnut, pomegranate, and aquaculture sectors (see Question B1 for more detail). 2. Various examples of processes launched, systems designed and installed, and management practices adapted, tend to give confidence that such tools and mechanisms may be applied in the future without external USG support. It appears that the cost-sharing approach might have contributed to sustainability. 3. It appears that delays faced by hazelnut and pomegranate associations with official registration may be wedged by weaknesses of antimonopoly legislation which is a factor of political will and is considered to be outside of the Contractor control. Recommendations Component 1: Domestic Business Environment Improved Component 2: Liberalized, More Transparent Trade Environment Developed It is recommended that future USAID projects place an emphasis on combining TA with efforts to address systemic changes in partner and client institutions. It would also be advisable to thoroughly assess impacts of such measures following USG support rather than at intermediate level. Component 3: Targeted Agricultural Value Chains Improved TA assistance at the bank institution and farm levels should ideally be complemented by a cost-shared or matching grant approach (depending on what is most practical) aimed to enhance theory-based training and TA with practical adaptation of new products, technologies and practices. 4.4.3 Question 3: What were the obstacles to sustainability and what measures should be taken to increase sustainability in future projects? Figure 13: QUESTION 3- SUMMARY OF EVALUATION DESIGN AND METHODS Evaluation Question Type of Analysis Conducted Data Sources and Methods Used Type and Size Sample Limitations 3. What were the obstacles to sustainability and what measures should be taken to increase sustainability in future projects? Description – based on content analysis of expert opinions Key informant interviews with key personnel in relevant government ministries; ACT project staff Interviewees identified by Evaluation Team and USAID; Components 1,2,3 Legislative and regulatory reform for Business Enabling Environment; WTO; inter￾governmental cooperation; Public Private Dialogue; Public Consultation; M&E Framework for Policy Implementation etc.); Cost sharing arrangements clarified.. FINAL PERFORMANCE EVALUATION OF ACT 34 | Page Figure 13: QUESTION 3- SUMMARY OF EVALUATION DESIGN AND METHODS Evaluation Question Type of Analysis Conducted Data Sources and Methods Used Type and Size Sample Limitations Sustainability of assistance to Value Chains; Interviewees identified by Evaluation Team and USAID. Findings 1. ACT interventions have supported key elements of GOAJ reforms for improving the business and trade environment in Azerbaijan, including changes in legislation, support to local financial institutions, technical assistance to improve agribusiness knowledge and skills among others. However, there remain serious obstacles to sustainability, as evidenced by the significant gap existing between economic growth and institutional development. This issue presents significant difficulties for Azerbaijan’s transition from quantitative indicators to qualitative indicators, and from a supply-based economy to demand-driven economy. 2. According to ACT reports, agricultural productivity has increased by 20%-25% with project support. However, a lack of statistical data kept by farmers has prevented any meaningful commentary on the sustainability of project interventions. 3. Feedback from several government agencies appears to suggest that they were not sufficiently consulted or involved in discussions with the ACT Project prior to the preparation of its Work Plan, which appears to have unsettled them in terms of project expectations and the extent to which they are expected to acquire ‘ownership’ of interventions. Conclusions 1. The large gap between country economic growth indicators and quality of institutional changes remains a very serious obstacle for USAID on-going and future efforts for increasing the sustainability of projects. 2. The absence of reliable statistical data from farmers means that it is difficult to determine how sustainable the project’s interventions have been in the agricultural sector. 3. On-going consultation between government and international experts regarding project Work Plans in some instances has proved an obstacle to some GOAJ organizations embracing ‘buy-in’ of the project’s goals. Recommendations 1. Continued support for institutional changes in existing economic policy-making processes and procedures at national and local levels are core for increasing the sustainability of international support. At the local level, support could be given to help establish farmers’ and rural SMEs business associations or specialized agricultural producers cooperatives. At the regional level, research centers could be supported together with consulting organizations. At the national level, support might be envisaged to help restructure MoA and its regional service units although such an initiative would probably exceed the managerial capacity of the project. Continued assistance towards removing current barriers to competitiveness and trade (over￾centralization, state capturing of private sector) could be the subject of future USAID interventions since they continue to present real obstacles regarding project sustainability. 2. A requirement for similar projects should be that where the impact of an intervention needs to be measured, then intended beneficiaries should be provided with the tools or means to record the required data upon which those results can be measured. 3. To ensure government ‘buy-in’ and subsequent ‘ownership’ of project interventions it is advisable that relevant officials and departments are fully aware of project objectives and have the opportunity to contribute towards the development of Work Plans. FINAL PERFORMANCE EVALUATION OF ACT 35 | Page 4.4.4 Question 4: Appraise the role of the cost-sharing approach to Short-Term Technical Assistance (STTA) in terms of creating ownership of the beneficiaries and sustainability Figure 14: QUESTION 4- SUMMARY OF EVALUATION DESIGN AND METHODS Evaluation Question Type of Analysis Conducted Data Sources and Methods Used Type and Size Sample Limitations 4. Appraise the role of the cost sharing approach to STTA in terms of creating ownership of the beneficiary and sustainability Description – based on content analysis of expert opinions ACT project staff. ACT project partners and beneficiaries. Azsertifika; Quality Associates AgBank; AzerCredit; AZNAR JSC. Analysis of project documentation; Structured Interviews (questionnaires); FGDs Interviewees identified by Evaluation Team and USAID Sufficiently robust sample Unavailability of Demir Bank and Turan Bank representatives; Assistance to AzCredit has not been subject to cost￾sharing. Unavailability of some processors due to religious/summer holiday period. Findings The Contractor has introduced a cost-sharing approach to STTA in an attempt to create ownership of the beneficiaries and reinforce sustainability. In particular, this approach was designed to encourage buy-ins and to incentivize ACT clients to co-finance project interventions or invest additional funds and resources in business and enterprise. Cost-sharing represented a combination of the traditional approach to STTA with that based on both non-commercial and commercial principals of project-client cooperation. The hypothesis behind this approach was that a relationship based on 100% investment by ACT in STTA is unlikely to encourage commitment of businesses and banks to buy-in or sustain the outcomes of STTA without further external support of the USG. This approach is also directly relevant to Component 2 objective: Increased Private Sector Support for WTO Accession and Related Reforms. Willingness of food processing enterprises to cost share STTA and invest in the introduction of HACCP, indicated their increased awareness of and commitment to raising food safety requirements in the light of the Azerbaijan accession to the WTO. The cost-sharing approach was applied in two specific areas: 1) Banking sector reform; and 2) Introduction of HACCP. The Contractor entered into Memorandums of Understanding (MoU) with a number of organizations, including:  Azsertifika  Quality Associates  AgBank  Demir Bank  Turan Bank  AzerCredit  AZNAR JSC The Evaluation Team met Azsertifika and Quality Associates, which worked with ACT to advise processing enterprises on the introduction of HACCP. This type of STTA was designed to help food business operators look at how they handle food, and introduce procedures to make sure the food produced is safe to eat. The Contractor reported40 that AgBank, Demir Bank and Turan Bank have co-financed STTA from their own resources to pay towards the design and introduction of new financial products or adaptation of best practice lending, management, and HR practices. The Evaluation Team also met with the representatives of AgBank in Baku and Barda. Demir Bank and Turan bank declined a meeting. The Team also met AzerCredit but discovered that no cost-sharing approach was agreed and achieved within the framework of cooperation with ACT. Banking Sector Reform                                                              40 ACT FY2012 Q2 Report January 1 – March 31, 2012 Final; ACT Q3 FY11 Quarterly Progress Report, April 1 – June 30, 2011 Final.final FINAL PERFORMANCE EVALUATION OF ACT 36 | Page AgBank has made a contribution of 10,000 AZN towards the cost of STTA provided by the Contractor to the bank. ACT has extended support in the analysis of bank procedures and practices, development of strategic directions, and the design of a new micro-finance product. ACT provided wider analysis and strategic advice to the bank over the course of six months in 201241. The bank considered that a more focused STTA on the subject of micro-finance product development was better tailored to specific needs of the bank at that time. In 2012, the management of the bank identified growing demand for competitive micro-finance products in the regions. In this respect, ACT’s STTA to assist with the design and launch of the new product in the period May – November 2012 was considered directly relevant and timely. Key features of the new micro-finance product are:  Loans subject to successful non-financial (i.e., social and psychological assessment) of the applicant  Analysis of credit history that takes into account repayment and behavioural characteristics  Simplified 2 page-long loan application  Five days disbursement period (as opposed to 21+ days in the past)  Maximum loan value 2,700 AZN  On avarege14 or up to 24 months repayment period  Approximately 25% APR  Risk based loan analysis  Estimated 0% lending risk  Currently available in Yevlakh, Barda and Shamaki Since the launch of the new product in May 2012, AgBank has issued 140 loans in the target locations with the total current loan portfolio of 220,000 AZN. The bank believes that the size of the loan portfolio, due to ACT project support, has increased by at least 10%. At the time of the evaluation, the bank’s management was making plans to roll out the product to several other regions. AgBank has accepted the ACT Project advice regarding the need to build up and strengthen the bank’s ToT in￾house program. The ToT team has been further augmented by the bank with ACT support to provide training in such specialized subjects as:  Fraud prevention and audit for experiences staff  Direct client loan marketing, client relationship and sales for new staff  Basics of micro-finance for new staff and interns The Evaluation Team also met for an in-depth discussion with the staff of the representative office of the AgBank in Barda, which handles one of the busiest agricultural and rural business portfolios of the bank in Central Azerbaijan. While bank competition is high, with branches of 24 other banks present in Barda, AgBank office has extended loans to 40 new customers in 2012-2013, sealing the portfolio by an additional 90,000 AZN. Around 55%-60% of the loans financed the development of husbandry, dairy and wheat production, which are the primary agricultural sectors in the rayon. The bank’s analysis concludes that given the popularity of the new product, growing demand for micro-finance could lead to an extension of the AgBank portfolio by 100% before year end in Barda alone. Introduction of HACCP Quality Association has worked to assist Sahliayali, a leading meat processing factory, with the introduction of HACCP. The initial assessment at the factory identified a number of weaknesses with regard to infrastructure, equipment and operational practices. Advice and assistance were provided to remove weaknesses contributing to subsequent TUV42 and ISO 22,000 certification of Sahliayali. The company has reportedly invested over 1,000,000 AZN in better food safety by upgrading and reconstructing its facilities, and has contributed 35,000 AZN as part of a cost–sharing approach versus 9,000 AZN contribution of ACT project. ACT extended support of TCN Professionals to Quality Association and Sahliayali, which was highly regarded by both clients. Azsertifika has been active in delivering HACCP-related assistance to a number of clients among food processors, including Orely, IKAR-S, Freshfruit, Canub Agro, Azestar, Kristall, and Interpak, based on a 70% (ACT                                                              41The assistance extended by U.S. National Professionals was considered less relevant and effective than that of Non-U.S. Expatriates/Third Country National (TCN) Professionals. 42 TUV = Technischer Überwachungs-Verein, English: Technical Inspection Association). FINAL PERFORMANCE EVALUATION OF ACT 37 | Page project) versus 30% (client) cost-sharing agreement. ACT provided added value to this approach by extending comprehensive STTA to clients. The hypothesis behind the cost-sharing approach to STTA was that financial contributions made by processors would stimulate buy-ins and encourage private investment in attaining HACCP (or ISO) standards. The Evaluation Team noted growing appreciation among processors of the importance of HACCP, particularly for export operations, and as means of boosting competitiveness. For example, Russia’s recent accession to the WTO, demands that Azerbaijani exporters adhere to HACCP and ISO standards closer than ever before. The Evaluation Team noted several examples of the cost-sharing approach leading to various investments by processors: Name of firm Speciality Amount of investment (AZN) HACCP/ISO ready (%) 1. Aznar Pomegranate/other fruit juice/ pomegranate seed oil ≥3,000,000 ISO 22,00043 (100) 2. IKAR-S Cold storage/Apple ≥50,000 ≥70 3. Freshfruit Cold storage/Apple ≥50,000 ≥70 4. Canub Agro Vegetables ≥100,000 ≥100 5. Azestar Hazelnuts ≥440,000 ≥100 6. Orely Processed dry fruit ≥200,000 ≥70 7. Inter-Pak Pomegranate juice ≥400,000 ≥70 Other cost-sharing examples to STTA or resulting investments by clients include: Aznar. Respondents confirmed that shared costs of agronomic advice and training to pomegranate farmer associations in Goychay reinforced the importance of synergy in achieving stable supply of raw material. Girkhbulag Trout Farm. Farm management confirmed that it has provided 10,000 AZN matching contribution to a 6,500 AZN USG grant towards the creation of a fish feed processing unit in Sheki. Furthermore, since the cooperation with ACT started in 2010, the farm invested over 120,000 AZN in various improvements for the business, mostly in response to advice and STTA received. In particular, investments targeted improvements in such areas as incubation, feeding, growth management, medical treatments, and farm infrastructure. One of the main outcomes of cost-shared STTA and investments is the growth of farm output from 60 to 100 tonnes per year in period 2012-2013, compared with 2-3 tonnes output prior to 2010. The farm’s management currently forecasts further increase of capacity to 200 tonnes by the year 2015. Another noticeable outcome is the reduction of costs from 78% to 50%. Investment in upgrading of a 2 km long water supply line has also led to a 10 times increase in water supply to support the farm’s current growth levels. The farm has also reported that following several unsuccessful attempts in 2007- 2008 to secure a commercial loan, banks now appear to be able to offer a more favorable outlook on the business. In 2011, the farm secured and repaid 2 loans amounting to 30,000 AZN. AzerStar. Respondents confirmed that the cost-sharing approach has proved to be a more incentivized and commercially oriented partner relationship. The approach was found to reinforce the commitment of partners and yield better prospects of sustainability of STTA. The Evaluation Team found that STTA by TCN Professionals led to systemic changes in some client’s approach to their international exports strategy. In addition to the development of marketing plans, the company has put emphasis on the advancement of HACCP adoption. Largely due to STTA received, AzerStar’s management now considers HACCP adherence and implementation as means of achieving and maintaining the competitiveness of the enterprise. Over the period of 2012-2013, the company has invested 40,000 AZN in revamping its facilities as per HACCP standards; 350,000 AZN in new equipment; and 50,000 AZN into the construction of a new building, for a total of 440,000 AZN. Respondents confirmed that given the quality of STTA provided and the evident outcomes, the enterprise is prepared to increase its contribution to the cost-sharing approach to 50% or even higher depending on the scope of STTA offered. ESDER. While the enterprise did not take part in the cost-sharing agreement with ACT, it invested 100,000 AZN for revamping its infrastructure to make it HACCP compliant, mainly in response to STTA.                                                              43 2005. FINAL PERFORMANCE EVALUATION OF ACT 38 | Page Conclusions Evidence gathered by the Evaluation Team leads to conclusion that the cost-sharing approach to STTA applied by the Contractor has been very effective.  In particular, cost-sharing with partners and clients of ACT represented added value to USG’s assistance through creation of buy-ins among clients and promotion of greater ownership with regard of sustaining outputs and outcomes of STTA. There is clear evidence available that cost-sharing of STTA contributed to investments by clients in infrastructure upgrading and adoption of better food quality standards. The Evaluation Team concluded that clients’ willingness to share costs of STTA and the investments they have made towards HACCP compliance is an indication of local ownership and evident likelihood of sustainability.  The Evaluation Team noted that not all investment was necessarily driven by clients partaking in the cost￾sharing of STTA. In some cases, interviewees indicated that a process of adapting HACCP was on-going prior to the start of ACT support. However, respondents also indicated that ACT’s assistance has greatly enhanced and streamlined their efforts in advancing with HACCP. The Contractor has also been able to embrace private investment initiatives with multi-faceted TA, both short- and long-term, which was an additional incentive for clients to engage with the ACT Project on this approach.  Cost-sharing triggered an increased client demand and expectation with regard to focus, quality, practical orientation, and closer alignment of STTA to specific needs of the clients. This, for example, led clients to articulate their preference to short-term assistance received from non-U.S. Expatriates/TCN Professionals over that provided by U.S. Nationals, which was considered less relevant and effective.  The outcomes of the cost-sharing approach, combined with provisions of tailored STTA, are more likely to lead to systemic changes in strategic and managerial behavior of clients. The banking sector clients have augmented outdated financial products with new offers that are based on modern risk assessment methodologies relatively new to Azerbaijan. AgBank, for example, is keen to build on the success of the new financial product designed, launched and piloted with ACT support in Barda, Yevlakh and Shamaki by rolling out the product to several other regions without external assistance. Recommendations  The Evaluation Team recommends that the cost-sharing approach to STTA be applied as standard measure in projects where it is considered relevant and practical to achieve greater level of client commitment and local ownership.  Inception screening of potential clients will be of paramount importance in order to establish the level of their commitment to the cost-sharing approach. Prior assessment of clients’ capacity and experience, any visions or strategic plans in place, financial standing, and investment potential (i.e. HACCP) will be required in order to avoid clients’ withdrawals.  Application of the cost-sharing approach reinforces the importance of careful alignment of STTA with specific needs of clients. Effective management of expectations and clients’ ability to sustain STTA are equally important to the success of the cost-sharing approach. This recommendation is primarily based on the Evaluation Team’s conclusion that in some cases (banking and financial reform; hazelnuts; dairy, livestock) the clients of the ACT Project have given preference to the assistance extended by non￾U.S./TCN Professionals, which they found more relevant and effective than that offered by U.S. National Professionals. An example of close alignment of STTA with specific clients’ needs and evident positive outcomes to that approach, was the STTA by U.S. National Professionals extended to Aznar (i.e. international export linkages, market dynamics) and Girkhbulag trout farm (Troutlodge inputs of eyed salmon eggs, wider adaptation of modern trout farming technologies and practices). 5.0 LESSONS LEARNED 1. Projects such as ACT, with their wide scope of activities, have proven to be too ambitious to be implemented successfully within the time period and budget originally envisaged. Therefore, USAID should FINAL PERFORMANCE EVALUATION OF ACT 39 | Page seriously consider the efficacy and value of designing a project that can realistically be implemented relative to the anticipated results as defined by pre-determined indicators in RFTOPs and SOWs. 2. Changing the SOW in the middle of the project, as defined by Amendment 3 of the SOW in May 2013 agreed between USAID/Azerbaijan and the Contractor, does not necessarily lend itself to a smooth transition towards implementing the envisaged changes. In this case, there was clear confusion among members of the ACT Project team as to what really initiated such a significant realignment in project activities and objectives. Neither is there any official explanation in project documents or reports regarding such a change, which arguably adds to the confusion. Perhaps USAID should consider developing as set of criteria, which would need to be satisfied before SOWs can be amended; simply changing a project’s Contracting Officer Representative (COR), often more than once, should not be sufficient rationale or grounds for such amendments. 3. International donors such as USAID often draw the assumption that by simply implementing a long-term technical assistance program in a host country, it means that beneficiaries and key stakeholders have ‘bought into’ the rationale and purpose of the program. This in fact is often not the case as more than one donor can be advising or working with those same beneficiaries or implementing partners at the same time. Evidence in this case appears to suggest that to some extent this has indeed occurred (GIZ, EC and others). Perhaps USAID might consider a more robust stance in SOWs regarding donor co-ordination to avoid duplication and to ensure that direct beneficiaries understand and appreciate the differences in assistance between USAID projects and others. This would best be achieved at the start of a project. 4. One of the issues arising from this evaluation was the fact that there is no guarantee that by simply advising the GOAJ on the preparation of legislation and regulations, this support will necessarily lead to their implementation. In this respect, it may be more beneficial for USAID to employ a broader interpretation of business enabling environment rather than limit itself to working towards improving the legislative and/or regulatory framework of a host country. FINAL PERFORMANCE EVALUATION OF ACT 1 | Page ANNEXES FINAL PERFORMANCE EVALUATION OF ACT 2 | Page ANNEX A: STATEMENT OF WORK FINAL PERFORMANCE EVALUATION OF ACT 1 | Page Task Order Number: AID-112-TO-13-00002 Indefinite Quantity Contract Number: AID-RAN-I-00-09-00018 SECTION C - DESCRIPTION/SPECIFICATIONS/STATEMENT OF WORK FINAL PERFORMANCE EVALUATION OF THE AZERBAIJAN COMPETITIVENESS AND TRADE PROJECT (ACT) I. GENERAL This is a Task Order Statement of Work (SOW) to provide USAID with a performance evaluation of the following project: Project Title: Azerbaijan Competiveness and Trade (ACT) Project Contract Number: AID-112-A-11-00001 Period of Performance: October 1, 2010 - September 30, 2013 Total Funding: $21,997,184 (co-financed with GOAJ) Implementing Organization: Sibley International a. PURPOSE The purpose of the Azerbaijan Competiveness and Trade (ACT) Project performance evaluation is to: 1. Determine the accomplishments (results) and impacts of ACT Project interventions, and their sustainability; 2. Provide lessons learned and recommendations from ACT’s design and implementation to inform the implementation of the follow on Azerbaijan Trade and Agricultural Linkages (ATLAS) Project. Further, the Contractor must: i) determine the progress in regulatory reforms achieved in Government of Azerbaijan (GOAJ) institutions that have participated/benefited from ACT Project; ii) determine the project’s success in strengthening the value chains (production, processing storage and market linkages) in which it worked; iii) determine the level of development of farmer advisory and agribusiness support services; iv) determine the project’s effectiveness in coordinating and collaborating with stakeholders (GOAJ, other donors, beneficiaries); v) identify the various factors and conditions in the country that have enhanced or limited the effectiveness and result of ACT Project assistance in Azerbaijan; and vi) determine whether or not deliverables outlined in the Work plans have been achieved. The evaluation will cover the project implementation period of from October 1, 2010 – September 31, 2013. The primary stakeholder for the ACT Project performance evaluation is the USAID/Azerbaijan Mission. Other stakeholders include USAID/Washington (Europe and Eurasia Bureau (E3 Bureau) and Policy, Planning, and Learning Bureau (PPL Bureau), Sibley International, Inc. other USAID implementing partners, GOAJ (particularly the Ministry of Economic Development), and beneficiaries of USAID activities. The Mission may share the final evaluation report with other donors, other implementing partners, and other United States Government (USG) agencies operating in Azerbaijan. The Mission will use the evaluation findings to guide the Mission’s future project designs and implementation. b. OBJECTIVES The objectives of the evaluation are as follows: 1. Evaluate the progress made by the ACT Project under each component based on established targets; and 2. Provide recommendations for follow-on programming for USAID interventions in the value chain development, business enabling environment (agricultural sector and in competitiveness in particular) further assistance for World Trade Organization (WTO) accession. II. BACKGROUND The Azerbaijan Competiveness and Trade (ACT) Project is a three year, $21.97 million project that started in October 2010. It aims to improve the domestic business enabling environment, develop a liberalized, more transparent trade environment, and enhance private sector competitiveness through the improvement of FINAL PERFORMANCE EVALUATION OF ACT 2 | Page targeted value chains in the agriculture sector. These overall objectives were supported through the elimination or mitigation of technical and administrative barriers that hinder progress in all three areas. Formally, the project is divided into the three components: 1. Component 1: Domestic Business Environment Improved, especially in agriculture This component emphasizes improvements to the domestic business environment regulatory framework for competitiveness, financial sector stabilization, cost/benefit analysis, anti-money laundering, among other activities 2. Component 2: Liberalized, more Transparent Trade Environment Developed This component emphasizes support to the GOAJ for World Trade Organization (WTO) accession 3. Component 3: Targeted Agricultural Value Chains Improved This component emphasizes support for selected agricultural value chains, grades and standards for agricultural products to meet international standards for trade, improvements in cross-border agricultural trade, among other activities. In addition, there are several cross-cutting technical areas that support all three components. These include the development of a public-private dialog to support discussions between the private and public sector to improve understanding of changes; the analysis of costs of various initiatives (legislation implementation, administrative barriers, trade facilitation, etc.) to facilitate the understanding and possible change of various legal and administrative measures through the use of objective cost information; overall monitoring and evaluation to ensure the project objectives are met; and development and implementation of a grants program to support the development of partner organizations in building local institutional capacity. a. THE PROJECT’S ANTICIPATED RESULTS i. Steady progress toward an improved economic policy environment; more openness and security for private sector businesses; reduced officially-tolerated corruption, particularly within tax and customs ministries; with Azerbaijan ultimately integrated into world economies through WTO accession; ii. For private businesses and farms, increased revenues and enhanced livelihoods through more efficient and productive operations, leading to a vibrant private domestic market and export opportunities for locally￾produced food products; iii. Improved oversight and transparency of financial transactions so as to better combat money laundering and other international criminal behavior; iv. Improved investment climate; v. Improved export climate; and vi. Improved technical capacity on the part of ministries, universities, civil society in the areas of economic growth, trade and agriculture value chains. b. ACT’s PROJECT’S PRECURSOR AND COMPLEMENTARY PROJECTS In March 2000, USAID launched the three-year Participatory Agriculture Project in Azerbaijan (PAPA) that was designed to work in livestock and hazelnuts to increase market responsiveness of client firms and developing private membership associations which would provide services and become financially self￾supporting In September 2003, it launched the five-year Rural Enterprise Competitiveness Program (RECP) that was designed to improve domestic product quality and expand focus on competitive products. In August 2004, it launched the five-year Azerbaijan Business Assistance and Development Project (ABAD) that was designed to build the productive capacity of targeted communities and establish rural information services and outreach campaigns for targeted communities. In August 2007, USAID launched the three-year Trade and Investment Reform Support Project (TIRSP) that was designed to improve the transparency, predictability, and market-friendliness of the trade and investment policy and institutional environment in Azerbaijan, to support further progress towards WTO accession and build the capacity of institutions to comprehensively implement the WTO Agreement. It also supported the FINAL PERFORMANCE EVALUATION OF ACT 3 | Page Government’s efforts to develop a transparent and efficient trade and investment enabling environment and provided training on cost/benefit analysis in various public sectors, including water, energy, and education. In September 2007, USAID launched the two-year (which was extended December 2009 – February 2011) Financial Sector Support Program in Azerbaijan (FSSP) that was designed to strengthen the financial sector through work with the Central Bank of Azerbaijan and some commercial banks. In particular, it provided support on bank supervision, anti-money laundering, human resources management, information management, macro and monetary policy, and financial services for SMEs. In September 2008, USAID launched the two-year Private Sector Competitiveness Enhancement Project (PSCEP) that was designed to enhance the competitiveness of targeted non-oil sectors and sub-sectors in order to promote economic diversification and sustainable expansion of investment and employment in the Azerbaijani economy. In October 2009, it launched the three-year Community-Based Support to Producers of Dairy Products Project that was designed to expand dairy production of small/medium sized farmers and promote regional collection center linkages to commercial processors for the sustainable development of the existing private dairy farming households and small farmers. In September 2010, it launched the three-year Azerbaijan Competitiveness and Trade (ACT) Project that was designed to improve the domestic business enabling environment; develop a liberalized, more transparent trade environment; and improve targeted value chains (apples, hazelnuts, pomegranate, dairy, aqua culture) in the agricultural sector in Azerbaijan. In April 2012, it launched the Micro-Enterprise Support Project (MESP) whose goal is to contribute to a more conducive environment for the sustainable development of micro and small enterprises in Azerbaijan by improving the dialogue and interaction between the state and civil society on policy issues related to development of micro and small enterprises; improving the legislative and institutional framework for MSEs; and providing assistance to women-led home businesses to improve their viability. In September 2012, it launched the Youth Business Leadership Project (YBL) to provide internships and mentorships to third and fourth year university students. In addition to these projects, USAID also initiated two loan guarantees with four commercial banks in order to increase the amount of credit available to micro, small, and medium sized enterprises especially in the agricultural sector. The following questions are provided as a guide in formulating findings and recommendations of this evaluation. A. Meeting Targets/Reporting Progress: 1. How effective was Contractor in working towards meeting USAID and GOAJ goals and objectives in all three years of the ACT Project? a. Have there been any significant or critical gaps in ACT’s implementation? b. Were the assumptions established at the beginning of the project realistic throughout the period of implementation? Identify the assumptions changed and their impact on project progress and results 2. Was the data collected sufficient to measure program progress and results in all three program components? Were performance reports (weekly, monthly, quarterly, other) sufficient to monitor progress or identify problems in implementation and progress toward results? 3. Assess whether the data collected by the ACT Project provides evidence of progress towards accomplishment of intermediate results in the three areas. 4. Did the project design take into consideration gender issues such as gender division of labor, time use, control of resources, access to finance, etc.? FINAL PERFORMANCE EVALUATION OF ACT 4 | Page 5. Are various resulted reported accurate and verifiable? Have the results achieved by the project been cost effective and sustainable? B. The Contractor must describe progress in Key Interventions Areas and Results and Impacts as noted below: 1. Assess the progress and impacts of the value chain interventions under ACT Project in terms of i) technology transfer and the relevance of technologies transferred, ii) strengthening farm advisory and input delivery services, iii) capacity building of post-harvest and processing facilities and iii) linking producers with markets. 2. Assess the range of policy reform efforts supported, progress and the status (e.g., legislative status of each reform supported) and priority reforms to be supported in future. 3. Evaluate the progress made in Azerbaijan’s accession to WTO and further support to be extended. C. The Contractor must evaluate the Effectiveness of Technical Assistance: 1. Short-Term Technical Assistance (STTA): How effective was the STTA s in meeting the needs of ACT Project clientele? Assess the impact of STTA on the beneficiaries in terms of relevance of STTA recommendations and their adoption. 2. How effectively has the ACT Project created awareness about the issues it worked on among the private sector, donors, civil society, and Government agencies in Azerbaijan? D. The Contractor must evaluate the Sustainability of the ACT Project as noted below: 1. Assess the extent of stakeholder ownership of interventions undertaken by the ACT Project (provide examples). 2. What processes, systems and management were put in place to ensure that the results and impact of the ACT Project will be sustainable? 3. What were the obstacles to sustainability and what measures should be taken to increase sustainability in future projects? 4. Appraise the role of cost-sharing approach to Short-Term Technical Assistance (STTA) in terms of creating ownership of the beneficiaries and sustainability. IV. EVALUATION METHODOLOGY USAID recommends using both quantitative and qualitative methods (data, observations and anecdotes) in assessing progress, results and impacts. The Contractor must review the Contract, annual Work plans, quarterly performance and financial reports, baseline data on established indicators, results/outcomes and annual targets, quality and relevance of data collected to track and monitor progress and accomplishments. The Mission recommends that the evaluators use rapid appraisal methods, face-to-face interviews with key informants, focus groups, group discussions, and community discussions as appropriate to the question to the issues being evaluated and also to gauge satisfaction by the Government and key stakeholders about project performance. The Contractor is encouraged to employ other methods as long as they do not add to the duration or cost of the evaluation. The Contractor must ensure that all relevant issues are covered in the evaluation in instruments that may be used e.g., questionnaires and or list of topics. The Contractor must review the documents cited in the Reference Documents section of this Statement of Work. These documents provide the context of the economic development in Azerbaijan and information on the ACT’s work over the years: the management, implementation, monitoring, and reporting of activities and their financing. FINAL PERFORMANCE EVALUATION OF ACT 5 | Page The evaluation design and evaluation Work plan must be finalized upon arrival to Azerbaijan. V. TASKS A. Desktop review of key documents: Prior to any field work the Contractor must review key documents to develop a Work plan. All available documentation describing ACT Project activities carried out in Azerbaijan must be reviewed. Documents for review include but are not limited to those listed in the reference section. The Evaluation Team must meet ACT Project Contractor home office managers in Washington, D.C and USAID officials (suggested by USAID/Azerbaijan) prior to arriving in Baku. The Contractor must contact USAID designated Contracting Officer’s Representative (COR) for ACT Project evaluation on access to relevant documents. B. In-briefing: Upon arrival in Azerbaijan the Evaluation Team must provide an entrance briefing to the designated USAID officials. Introduce the Evaluation Team, discuss logistics, scheduling, discuss submission of the Work plan, and any other issues. USAID will assist with identification of the relevant stakeholders to meet with and provide additional suggestions for interviews. C. Work Plan: The Work Plan must be in accordance with the USAID prepared timeline for all work to be included, dates for submission of draft and final reports. The Work Plan must include the following elements: i. Evaluation design/methodology to be employed; ii. Schedule of contracts and site visits (regions, beneficiaries and collaborators); iii. Arrangements for local logistics iv. Schedule of briefings and submission of deliverables; and v. Delineate the roles and responsibilities of the other members of the Evaluation Team to ensure coverage of all elements of the Statement of Work. USAID will provide comments within two days. D. Evaluation Design: The Contractor must prepare and submit to USAID for approval a final evaluation plan and schedule with the following major elements: Schedule, methodology for conducting the evaluation (data and information collection, field interviews), beneficiary groups to be conducted and regions to be visited. The Contractor must ensure that its findings and conclusions about the effectiveness of the ACT Project activities are based on available data are both accurate and reliable, and that information gathered is representative of and reasonably reflects results actually achieved. The Contractor must submit the final detailed evaluation design, which must consist the following: i. List of topics and relevant questions, methods and data sources for data gathering; ii. A matrix of regions and beneficiaries to be contacted; iii. Data analyses for each question and presentation plan; iv. Data collection instruments; and v. Limitations of the evaluation design if any. The Evaluation Team must share the evaluation design with the Implementing Partner for comment, but, in the interest of objectivity and independence, the Implementing Partner will not participate in the design, implementation, analysis, or presentation of the evaluation. E. Field work: After the finalization of the work plan and evaluation design and its approval by USAID the Contractor can then begin field work. F. Interim briefing at the end of the Field Work: An interim oral briefing must be provided to the Program Office and the Technical team of the USAID Office at the completion of field work to share observations etc. After data collection, the Evaluation Team must share with USAID data to be discussed at the Stakeholder’s Meeting and used in the written reports. G. In-country Mission Debrief: Upon completion of the evaluation and prior to departing from Azerbaijan, the Contractor must provide an oral debriefing for USAID/Azerbaijan Mission. Evaluation findings must include FINAL PERFORMANCE EVALUATION OF ACT 6 | Page facts, evidence, and data. Recommendations must be specific, concise and supported by evidence. Recommendations must be action-oriented and implementable. H. Stakeholders meeting: The Contractor must develop a formal PowerPoint presentation for the stakeholder’s meeting and share it with the Contracting Officer’s Representative (COR) at least one work day before the meeting for review and approval. I. Evaluation report: i. Prior to departing Azerbaijan, the Contractor must provide USAID with a detailed Outline of the Evaluation Report, main findings and recommendations. ii. The Contractor must submit a draft report of its findings within five working days after departing Azerbaijan for review and comments. The first draft must address comments and recommendations made by USAID and stakeholders during the out-briefing. iii. The Contractor must submit the final evaluation report to USAID after receipt of the comments. The length of the final report is not predetermined, but the report must be concise, well written, and comprehensive. Recommendations must be action-oriented, practical, and specific; define responsibilities and timelines for the action; and identify milestones and deliverables. Unresolved issues that highlight what remains to be done must also be included in the final report. The final report must include an executive summary, introduction, the development l context and the background of the project being evaluated, evaluation questions, explanation of evaluation methodology, the limitations of the evaluation, findings, conclusions and lessons learned, and recommendations for the sustainability of ACT Project activities. The executive summary must summarize the purpose, background of the project being evaluated, evaluation questions, evaluation methodology, major findings, lessons learned, conclusions, and recommendations.). The evaluation methodology must be explained in the report in detail. Limitations to the evaluation must be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (e.g., selection bias, recall bias, unobservable differences between comparator groups, etc.). The annex to the report must include: 1. The Evaluation Statement of Work. 2. Schedule of Evaluation 3. Evaluation design/methodology (s) employed questionnaire and list of questions by topic etc. 4. Names and contact information of key respondents, sites visited and other sources of Sources of information, properly identified and listed. 5. Information statements as appropriate regarding significant unresolved issues, difference of opinions (among members of the Evaluation Team, the Implementing Partner, GOAJ and other relevant stakeholders) and availability of data and its quality. 6. The Evaluation Design Disclosure of conflicts of interest forms of all Evaluation Team members, either attesting to a lack of conflict of interest or describing any existing conflicts of interest. VI. REPORTING RELATIONSHIP USAID will provide overall direction to the Evaluation Team, identify key documents. Primary contacts for the Contractor will be the designated COR for the Evaluation Award technical related issues, and COR of the ACT Project and Economic Growth Office Director for technical issues. VII. PERFORMANCE MONITORING FINAL PERFORMANCE EVALUATION OF ACT 7 | Page The Evaluation Contractor’s performance shall be evaluated based on the completion of specific tasks as outlined in the Task Order, adherence to the Work Plan, and reports submitted to the COR. VIII. REPORTING REQUIREMENTS TO ENSURE THE QUALITY OF THE EVALUATION REPORT Per the USAID Evaluation Policy, draft and final evaluation reports will be evaluated against the following criteria to ensure the quality of the evaluation report. 1. The evaluation report should represent a thoughtful, well-researched, and well-organized effort to objectively evaluate what worked in the project, what did not, and why. 2. Evaluation reports must address all evaluation questions included in the scope of work. 3. The evaluation report must include the Statement of Work as an annex. USAID must agree in writing with all modifications to the Statement of Work, whether in technical requirements, evaluation questions, Evaluation Team composition, methodology, or timeline. 4. The evaluation methodology must be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists, and discussion guides must be included in an Annex in the final report. 5. The evaluation findings must ensure that, where relevant, data analysis takes gender into consideration. 6. Limitations to the evaluation must be disclosed in the report, with particular attention paid to the limitations associated with the evaluation methodology. 7. The evaluation findings must be presented as analyzed facts, evidence, and data. Findings must be specific, concise and supported by strong evidence. 8. Sources of information need to be properly identified and listed in an annex. 9. Recommendations need to be supported by a specific set of findings. 10. Recommendations must be action-oriented, practical, and specific, with defined responsibility and timelines for the action. - END OF SECTION C – FINAL PERFORMANCE EVALUATION OF ACT 8 | Page ANNEX B: WORK PLAN FINAL PERFORMANCE EVALUATION OF ACT 1 | Page WORK PLAN Final Performance Evaluation of the USAID/Azerbaijan Competitiveness and Trade (ACT) Project 1 August 2013 Submitted by: Mendez England & Associates 4300 Montgomery Ave., Suite 103 Bethesda, MD 20814-4413 Contact Person: Thomas C. England, President Tel: (301) 652-4334  Fax: (301) 652-3733  Email: tengland@engl.com FINAL PERFORMANCE EVALUATION OF ACT 2 | Page 1. INTRODUCTION In the preparation of this Work Plan, the Evaluation Team has followed the guidelines outlined in the USAID’s Request for Task Order Proposal (RFTOP) and accompanying Statement of Work (SOW) (see Annex 1), together with ME&A’s Technical Proposal for the Performance Evaluation of the Azerbaijan Competitiveness and Trade (ACT) project implemented in Azerbaijan between October 2010 and September 2013. In overall terms, the purpose of evaluating this project according to the SOW is to: 1. Determine the accomplishments (results) and impacts of ACT interventions and their sustainability. 2. Provide lessons learned and recommendations from ACT’s implementation to inform the implementation of the follow-on Azerbaijan Trade Linkages and Agribusiness Strengthening (ATLAS) Activity. The primary objectives of the evaluation are to: 1. Evaluate the progress made by the ACT project under each component based on established targets. 2. Provide recommendations for follow-on programming for USAID interventions in the value chain development, business enabling environment (agricultural sector and in competitiveness in particular) and further assistance for World Trade Organization (WTO) accession. In short, the main thrust of the Evaluation Team’s mission is to determine the extent to which the ACT project contributed towards improving the domestic business enabling environment, developing a liberalized, more transparent trade environment, and enhancing private sector competitiveness through the improvement of targeted value chains in the agriculture sector. In so doing, the Evaluation Team will make reference to the targets, results and indicators outlined in the original USAID/Sibley International Task Order Number: AID￾112-TIO-10-00002 (09/30/2010). In pursuit of its activities, the Evaluation Team (see Section 2 below) is further tasked with answering a specific set of evaluation questions posed by USAID/Azerbaijan (Annex 2). Additional attached Annexes (3 to 6) include a revised Mission Schedule, List of Stakeholder Interviewees, Evaluation Design Chart, and Draft Outline of the Final Evaluation Report (Table of Contents). Specific issues relating to the Evaluation Design will be addressed in a separate document. 2. EVALUATION TEAM The evaluation of ACT will be conducted by a team of four (4) experts: 1) international expert Mr. Colin Maclean (Team Leader); 2) Mr. Danil Samoilenko (Trade/Economic Policy Reform Specialist); 3) Dr. Fagan Aghayev (Local Agriculture Specialist); and 4) Dr. Ibrahim Mammadzadeh (Local Trade and Economic Growth Specialist). In addition, the team will be assisted by Mr. Toghrul Aliyev as interpreter/logistics expert. Mr. Maclean will assume overall responsibility for the management of the evaluation in collaboration with USAID/Azerbaijan. This encompasses all activities specified in the Mission Schedule (Annex 3) related to the evaluation mission, including pre-mobilization, on-site implementation, and end-of-assignment deliverables. Mr. Danil Samoilenko will assist the Evaluation Team in various activities including the selection of the appropriate methods and techniques for data collection, especially for the economic growth and trade component. Dr. Aghayev and Dr. Mammadzadeh will contribute to the evaluation mission in the scheduling of meetings; conducting interviews and participating in focus group sessions; and carrying out additional research as identified by the Team Leader. Their input will also be required with respect to data gathering and analysis as well as report writing. They will also provide a valuable insight into local customs and culture, which will add to the team’s overall understanding of the ‘lie of the land’ in Azerbaijan. Finally, oversight of the evaluation mission will fall under the remit of Ms. Mirela McDonald, Evaluation IQC Manager with ME&A and Ms. Audra Stark, Project Manager with ME&A. 3. EVALUATION TASKS AND SUB-TASKS 3.1 Pre-Mobilization Activities Conference Call and Inter-Evaluation Team Communication On Monday, 22 July 2013, a conference call took place between Ms. Audra Stark, (ME&A), Mr. Colin Maclean (Team Leader), and representatives from (USAID/Azerbaijan), Ms. Elnara Bayramova FINAL PERFORMANCE EVALUATION OF ACT 3 | Page (Evaluation COR), Ms. Aytan Gahramanova, Ms. Tamilla Mammadova and Mr. Samir Hamidov. In summary, a number of issues were raised by the Team Leader requiring clarification from USAID in particular with regard to ACT’s budget breakdown, amendments to the original ACT contract, and questions related to the project’s M&E procedures. It was agreed that USAID Mission would look into the issues raised and provide feedback either before or during the Evaluation Team’s in-brief on Monday, 29 July 2013, in Baku. In addition to the above conference call, the Evaluation Team members conducted ongoing email exchanges among themselves on project-related matters, including logistical arrangements and input to the work plan and evaluation design. Literature Review Members of the Evaluation Team were provided with a selection of project-related documentation by USAID, all of which were reviewed prior to the in-country start of the mission. These included ACT Work Plans, Annual Reports, Quarterly Reports, Survey Reports, PMP, M&E Plan, Contracts and Budget Breakdown Statements. The documents reviewed provided a useful insight into the planning and operating activities of ACT and will be the source of reference for the Evaluation Team for data and information related to project objectives, indicators and targets. Preparation of Draft Work Plan and Evaluation Design This activity was initiated prior to the mobilization of the Team Leader in collaboration with the Evaluation Team members and the ME&A Evaluation Project Manager. Some adjustments were made to the Work Plan following the Team’s in-briefing with USAID in Baku on Monday, 29 July 2013 – (see Section 3.2 below). 3.2 In-Country Activities Initial Meetings On Monday, 29 July 2013, a meeting was held between the Evaluation Team members and USAID/Azerbaijan in Baku represented by Ms. Donna Stauffer (Mission Director (Acting), Ms. Elnara Bayramova (Evaluation COR & Program Development Specialist - Budget), Ms. Tamilla Mammadova (Evaluation Activity Manager & Monitoring & Evaluation and Project Design Specialist), Ms. Aytan Gahramanova (ACT project COR and Project Management Specialist), and Mr. Samir Hamidov (Project Management Specialist). This meeting was essentially an in-briefing for all parties to review the evaluation’s objectives and to provide everyone present with the opportunity to raise questions and points of interest for discussion and/or clarification. Issues regarding the proposed methodology to be used for data gathering and analysis were discussed together with the proposed list of site visits (additional locations were suggested by USAID and these would be included in the field trip schedule) and potential list of interviewees. Following these highly productive deliberations and exchange of views, it was agreed that the Evaluation Team should start organizing meetings without further delay given that it was holiday season in Azerbaijan and that the availability of interviewees might be somewhat restricted. The USAID team advised that it would draft and forward a letter of introduction to official bodies such as ministries advising them to expect imminent contact from the Evaluation Team regarding requests for meetings. The Evaluation Team was required to submit its Draft Work Plan to USAID on Wednesday, 31 July 2013, and its Evaluation Design to USAID on Friday, 2 August 2013. Re-submission of both documents incorporating USAID comments would take place on Saturday, 3 August 2013, and Thursday, 8 August 2103, respectively. However, the Evaluation Team proposed that it would probably be more productive to forward both documents at the same time for review and comment and advised USAID that it would aim to do this by Thursday, 1 August 2013, one day before the original deadline for the Evaluation Design. Some additional refinements to the Mission Schedule were discussed, including the fact that the out-brief with USAID would be re-organized for Friday, 23 August 2013 instead of Monday, 26 August 2013, and that a stakeholders meeting where the Evaluation Team would present its main findings to ACT’s main beneficiaries would also take place on the same day (at a location to be decided). This event would be planned and organized by the Evaluation Team. FINAL PERFORMANCE EVALUATION OF ACT 4 | Page Data Gathering Meetings with stakeholders would begin on Tuesday, 30 July 2013, and scheduled to continue through to Tuesday, 20 August 2013. These will include key members of the ACT Project team, direct beneficiaries and project partners and representatives from a cross-section of other relevant individuals and organizations with an interest in the project’s activities and outcomes – (see Annex 5 for suggested List of Interviewees). In some instances, required information will be sourced through quantitative research methods (review and analysis of facts that can be statistically verified from recorded data, e.g. ACT records such as Work Plans, Quarterly Reports and Annual Reports). In other cases, qualitative research methods (opinions, observations and explanations gathered from participating interviewees and/or focus group sessions) will be employed to act as a support mechanism (or otherwise) for quantitative i.e. numerical data – (see Section 5. below for a more in￾depth explanation of the Evaluation Team’s planned methodology). In terms of the overall approach to data gathering it was agreed that Components 1 and 2 would essentially concentrate on qualitative aspects related to ACT’s interventions and that Component 3 would effectively be evaluated from both a quantitative and qualitative point of view given that a number of indicators formed part of the project’s required result areas. This clarification provided valuable guidance to the Evaluation Team in support of its mission and more specifically to ensure that its efforts would focus on evaluating ACT activities that could realistically be assessed and/or measured following information gathered from carefully targeted sources. Qualitative evaluation would focus on the extent to which ACT assisted beneficiaries reappraise their way of thinking in terms of processes and procedures together with the steps needed for them to ultimately assume ownership of these planned changes in working practices. Quantitative evaluation would effectively comment on the extent to which ACT managed to achieve (or otherwise) pre-determined targets and whether those targets were realistic and/or relevant in the first place. Data Analysis This activity involves digesting and analyzing data and information gathered during the in-country mission. Findings will be tabulated both within the body of the final report and in stand-alone matrix format as annexes. An Evaluation Design & Methodology Matrix44 will summarize the qualitative aspect of the evaluation relative to each evaluation question and will be depicted within the main body of the final report. In addition, an Overall Monitoring & Evaluation Matrix, which will address the quantitative elements of the evaluation, will appear as an annex in the final report. Section 5 below further elaborates on both approaches. Out-Briefing with USAID/Azerbaijan As mentioned above, on Friday, 23 August 2013, the Evaluation Team will have a final out-briefing with USAID/Azerbaijan, during which initial findings will be presented in summary format together with some of the key issues arising from the evaluation. At this stage, comments and/or suggestions offered by USAID/Azerbaijan will be acknowledged and addressed in the Draft Final Report. 3.3 End-of-Mission Activities Completion and Submission of Draft Report to USAID On completion of the in-country mission and following the Team Leader’s return to home base, a Draft Final Report will be prepared and submitted to USAID/Azerbaijan on Wednesday, 4 September 2013. Final Report submitted to USAID with integrated comments By Friday, 13 September 2013, the Team Leader will receive their comments for integration into the Final Report which will subsequently be re-submitted to USAID/Azerbaijan on Thursday, 19 September, 2013. 4. EVALUATION DESIGN PLAN                                                              44 See the Evaluation Design document submitted separately to USAID/Azerbaijan further detail FINAL PERFORMANCE EVALUATION OF ACT 5 | Page In their Technical Proposal in response to the RFTOP No. SOL-112-13-000005 for the Final Performance Evaluation of the Azerbaijan Competitiveness and Trade (ACT) project, ME&A presented an Evaluation Process Chart (Annex 2), which depicted how the evaluation would be conducted – for reference purposes the chart is attached to this report as Annex 4. In short, this outlines the Evaluation Team’s modus operandi for the evaluation clearly outlining the steps taken for the implementation of the mission from the development of the methodology to data collection to analysis of results and reporting. A separate document – Evaluation Design - elaborates on the design process in greater detail and will be submitted separately to USAID/Azerbaijan by the above-mentioned deadline (Thursday, 1 August as suggested earlier). The main thrust of this document relates to data gathering and collection methods, data analysis for evaluation questions and how overall findings will be presented in the final report. 5. METHODOLOGY The Evaluation Team recognizes that the credibility of an evaluation’s findings, conclusions and recommendations rests for the most part on the quality of the research design as well as data collection methods and analysis used. As discussed and agreed with USAID/Azerbaijan, both quantitative and qualitative methods would be employed although these would be specific to individual components and not simply employed in a ‘broad brush’ approach. At the Pre-Mobilization Activity stage of the evaluation process, during which the Evaluation Team conducted an in-depth review of the documents made available to its experts, it became apparent that significant changes had been made to ACT’s original SOW as described in the four Amendments of Solicitation/Modification of Contract between USAID/Azerbaijan and Sibley International (24/06/11; 28/09/11; 03/05/12; 20/09/12). As the third amendment completely replaced Section A5 of the original SOW, introducing new (or revising) goals under each of the three components together with the introduction of new targets and results, it was clear that the Evaluation Team would need clarification on how to proceed with the evaluation given the above￾mentioned changes. This was in fact discussed at length at the in-brief meeting where agreement was reached that the evaluation should effectively only start its review post-modification. This therefore means that the methodology adopted would now focus on qualitative matters for Components 1 and 2 and both quantitative and qualitative issues for Component 3 (as mentioned above). With this in mind, the following approach is now how the Evaluation Team envisages tackling this evaluation process. 5.1 Quantitative Research and Analysis In summary, quantitative data (e.g. number of actions / activities to be accomplished against plan) will be sourced from ACT Annual Work Plans and other project-related periodic reports. In addition, particular reference will be made to the project’s M&E Plan including PMP indicators and monitoring data in Quarterly Reports. Review and analysis of these documents will enable the Evaluation Team to comment on actual results against plan and also note any actions and/or strategies that were taken to address anticipated deviation from the plan. The collective outcome of this evaluation will be a thorough assessment of the performance of ACT at the end of the project relative to the planned objectives as envisaged in the original SOW and elaborated and/or revised in subsequent ACT contract amendments. It is also important to note that verification of performance from a statistical point of view can essentially only focus on a review of ACT records, which the Evaluation Team can only assume accurately reflects whether a numeric indicator was achieved or not. The Team will therefore comment on this within the body of the final report in the relevant sections supported by an annex for tabulated reference purposes. The findings here will be cross-referenced with findings accumulated from our qualitative research approach to determine the extent to which evidence gathered contributes towards the Evaluation Team’s understanding of ACT’s impact on its intended beneficiaries. This in turn will enhance the team’s prospect of being able to provide satisfactory and meaningful answers to USAID’s pre-determined evaluation questions as highlighted in the Evaluation Team’s SOW. 5.2 Qualitative Research and Analysis The required approach under this activity assumes particular importance given the geographic spread of ACT’s direct and indirect beneficiaries and the time available to the Evaluation Team to conduct its enquiries. Here, the team’s approach will be to identify, locate and meet with as many representatives FINAL PERFORMANCE EVALUATION OF ACT 6 | Page as possible with direct or indirect knowledge and/or experience of the project throughout its lifetime (see Annex 5 for a proposed list of potential interviewees which has taken account of suggestions from USAID). In effect, the Evaluation Team will largely focus its review on the extent to which beneficiaries have been positively influenced by ACT’s interventions in terms of embracing and integrating changes in working practices to enhance policy-making decisions particularly in Components 1 and 2. In so doing, the Evaluation Team will pay particular attention to avoiding subjective opinion and hearsay as these effectively add little or no value to understanding the facts. On the other hand, perceptions are valid and will be included in the team’s findings as they may enhance the quality of recommendations for any future planned interventions by USAID/Azerbaijan. In order to effectively carry out the qualitative research, the Evaluation Team has developed a specific design approach, which is elaborated in the separately prepared Evaluation Design report. Here, in line with USAID’s published TIPS practical advice and suggestions for constructing an evaluation report, we address each evaluation question one-by-one depicting how the relevant research will be conducted together with any identified limitations. 6. LIST OF INTERVIEWEES Prior to arrival in Azerbaijan the Evaluation Team prepared a list of prospective interviewees, which was subsequently discussed with USAID at the in-brief meeting on Monday, 29 July 2013. Following comments and suggestions from USAID, a final list of individuals and organizations to be contacted was agreed with particular reference to those deemed to be essential interviewees. This revised list is attached as Annex 5 of this report. It should be noted of course that while the Evaluation Team will endeavor to arrange meetings with identified individuals and organizations, it cannot guarantee their availability not least due to it being holiday season in Azerbaijan including Thursday 8 and Friday 9 August as religious holidays when no-one will be available. 7. ARRANGEMENTS FOR LOCAL LOGISTICS At the time of writing, the Evaluation Team was in the process of finalizing contracts with a logistics/interpreter person, a second interpreter and 2 drivers. All will support the Evaluation Team throughout the in-country evaluation mission. FINAL PERFORMANCE EVALUATION OF ACT 7 | Page ANNEX C: EVALUATION DESIGN & METHODOLOGY FINAL PERFORMANCE EVALUATION OF ACT 8 | Page 1. Introduction The objective of the evaluation of the ACT project will be to: 1) determine the progress made by ACT under each of its three components, based on established targets; and 2) provide recommendations for follow-on programming for USAID interventions in the value chain development, business enabling environment, and accession to WTO, in particular. More specifically, the evaluation will determine: 1) the progress that GOAJ institutions that have participated in the ACT project have achieved in regulatory reform; 2) ACT’s success in strengthening the targeted value chains; 3) the level of development of farmer advisory and agribusiness support services; 4) ACT’s effectiveness in coordinating and collaborating with stakeholders; 5) the various factors and conditions in the country that have enhanced or limited the effectiveness of ACT’s assistance in Azerbaijan; and 6) achievement of deliverables/results outlined in the ACT’s Work Plans. This document elaborates on the design process in detail and should be read in conjunction with the Evaluation Work Plan submitted separately to USAID/Azerbaijan. The main thrust of this document relates to data gathering and collection methods, data analysis for evaluation questions and how overall findings will be presented in the final report. 2. Data Gathering and Collection Methods The Evaluation Team will use a number of techniques to gather data. We will combine techniques that balance each other: quantitative vs. qualitative data, individual vs. group responses, focus groups, community and group discussions, questionnaires, rapid appraisal methods, etc. Such techniques will help capture the diversity of opinions and perceptions of beneficiaries and stakeholders about the achievements of the ACT project and its effectiveness and sustainability. They will also uncover unexpected positive or negative impacts of ACT interventions on strengthening value chains, improving policy reform, and advancing efforts for Azerbaijan’s accession to WTO. Data gathering and collection methods are designed to support two sub-activities to be carried out simultaneously: a qualitative evaluation and a quantitative evaluation. The work for the qualitative evaluation will be primarily to conduct semi-structured, open-ended interviews with those organizations and individuals, as well as other stakeholders and partners, that were involved in the ACT activities. This work will focus on Component 1. Data will be collected by using a number of methods including: 1. A critical desk-top review of materials provided by USAID such as project reports and annual work plans, PMP, M&E Plan, contract amendments and performance indicators, etc. 2. Interviews with USAID/Azerbaijan staff. 3. Interviews with ACT project implementer staff. 4. In depth, semi-structured interviews with selected government organisations, CSOs, University partners and financial institutions. We believe that because ACT activities addressed complex issues, interviewees may be intimidated by structured interviews and formal interviewing techniques. Accordingly, we will use semi-structured interviews, a more appropriate and valuable technique, because they will allow beneficiaries to present and explain points freely. 5. Interviews with other donors, including the World Bank, GIZ, IFC, as well as other USAID projects. FINAL PERFORMANCE EVALUATION OF ACT 9 | Page A similar mix of methods will be used for qualitative evaluation of the results achieved in the Component 2 particularly regarding progress with the WTO accession process. Quantitative indicators, particularly to gauge the successes achieved in the Component 3, will also be looked at when available (laws amended; number of WTO Working Group meetings). This approach will ensure that both qualitative and quantitative parameters are examined when gauging the ACT Project assistance in these areas and its contribution. Data will be collected by using a number of methods including: 1. Meetings with Executive Power representatives in Guba, Goychay, Sheki, Khachmaz, Zagatala, Barda, Agjabadi and Lenkaran to gauge overall image and contribution of the ACT Project and to gather supplementary information concerning the socio-economic environment in which the project operates. 2. Individual meetings with the ACT project partners and beneficiaries in Baku, Guba, Goychay, Sheki, Khachmaz, Zagatala, Barda, Agjabadi and Lenkaran. 3. Focus groups: The use of focus groups will provide both quantitative and qualitative data to provide context and background on the data obtained through the individual interviews. Two Focus Groups will be organized by the evaluation team. Focus groups are useful for assembling qualitative information: the opinions and judgments of beneficiaries and intermediary stakeholders. They are particularly useful for generating project design to be investigated by other means. The exchange of opinions in such discussions results in better understanding of differing interpretations and justifications. A focus group may provide more objective responses than individual or group interviews. One potential disadvantage of focus groups is that some stakeholders may take the lead, attempting to impose their opinions on others. The evaluation team will ensure moderation. The objective of the Focus Groups will be to gauge the opinions and judgments of BDS providers, non-government organizations (NGOs), business and farmer associations, think-tanks and business support organizations related to key business environment, trade and finance constraints experienced by enterprises in Azerbaijan. The exchange of opinions in these discussions will result in better understanding of differing interpretations of issues raised during the individual interviews. The Focus Groups will promote free expression of views on those aspects of business environment improvement that are perceived as most important and relevant for enterprises. 4. Interviews with women that participated in the project in order to gauge the impact of ACT activities on them. 5. One Expert Panel will be organized by the evaluation team in the form of round-table discussions on the subjects of improvements of domestic business environment; liberalization and transparency of trade environment and improvements in the agricultural value chains as perceived by men- and women-run businesses in the regions of Azerbaijan. 6. Appraisals to obtain information from farmers that have received assistance from ACT to adopt best management practices, effective farming techniques, increase sales, productivity and employment. 7. Direct observation to cross-check information (e.g. comparing statements to observed practice) and identification of factors not previously recognized. 3. Data Analysis for Evaluation Questions FINAL PERFORMANCE EVALUATION OF ACT 10 | Page This activity involves digesting and analyzing data and information gathered during the in-country mission. To conduct the data analysis, the evaluation will use mixed methodology that will include qualitative and quantitative techniques. Qualitative techniques designed to elicit opinions and normative points of view often do not result in quantifiable results. However, they can provide meaningful indications if change has taken place and whether this change is viewed in a positive or negative manner by the beneficiary or stakeholder. In addition, qualitative techniques can often be used to support development of quantitative approaches, suggest new directions for further assessment, and provide additional in-depth perspectives on changes induced by ACT’ interventions. For qualitative data resulting from stakeholder interviews, where much of the evidence may be anecdotal or inferred, the team will use triangulation to identify any inconsistencies and ensure reliability. Triangulation will assist the Team to reduce the “response bias” in which respondents tend to tell the evaluators what they want to hear. Throughout the analysis, the Team Members will share and compare notes taken during the interviews, identify any variations in the information provided to Team Members by different stakeholders, and reveal their different expectations and opinions about the project. Primarily quantitative evaluation will consist of analyzing numeric data sourced from the PMP, M&E plan, project and progress reports, performance indicators. These data will be verified through semi-structured focus and discussion groups with BDS providers and farmers that participated in or benefitted from the ACT project. During data analysis, we will use the triangulation method in order to increase reliability and validity of our findings, especially for information collected through rapid appraisals. However, we realize that even though the triangulation method may yield convergent findings, this does not mean that these findings are unquestionable. Evaluation findings and recommendations for future USAID interventions will be presented to the Mission for comment before the Team’s departure. We will consult all our findings with the Mission before drafting the Final Report. The final report will be submitted after comments of the Mission are fully integrated. Evaluation Design and Methodology Matrix will be presented for each of the Evaluation Questions included in the SOW in the body of the Final Report along with findings, conclusions and recommendations related to each of the questions. The Matrix will summarize the qualitative aspect of the evaluation relative to each evaluation question and will be included in the consolidated form in the Annex 1. In addition, an Overall ACT Project Monitoring & Evaluation Matrix (Annex 2), which will address the quantitative elements of the evaluation will appear as an annex in the Final Report. The SOW provides guidance on the use of evaluation questions in formulating findings and recommendations of this evaluation. As a guide, four groups of key evaluation questions are included in the SOW within the 4 broad areas:  Group A – Meeting Targets/Reporting Progress  Group B – Progress in Key Interventions Areas (Components 1-3), Results and Intermediary Impacts  Group C – Effectiveness of Technical Assistance  Group D – Sustainability of the ACT Project FINAL PERFORMANCE EVALUATION OF ACT 11 | Page Group A - Meeting Targets/Reporting Progress Question 1 Figure 1: QUESTION 1 - SUMMARY OF EVALUATION DESIGN AND METHODS Evaluation Question 1 Type of Analysis Data Sources and Methods Type and Size Sample Limitations How effective was Contractor in working towards meeting USAID and GOAJ goals and objectives in all three years of the ACT Project? Description – based on content analysis of expert opinions Key informant interviews with ACT project personnel; ACT Project partners; ACT Project reports and M&E Plan. Interviewees identified by Evaluation Team and USAID. Components 1,2,3 Opinion bias; open ended & semi-unstructured interviews a) Have there been any significant or critical gaps in the ACT Project’s implementation? Description – based on content analysis of expert opinions Key informant interviews with ACT project personnel; Key ACT Project partners. Interviewees identified by Evaluation Team and USAID. ACT Project reports and contract amendments. Components 1,2,3 None perceived b) Where the assumptions established at the beginning of the project realistic throughout the period of implementation? Identify the assumptions changed and their impact on project progress and results. Description – based on content analysis of expert opinions Key informant interviews with ACT project personnel and partners; USAID/Azerbaijan staff; ACT Project reports and contract amendments. Interviewees identified by Evaluation Team and USAID. Components 1,2,3 None perceived FINAL PERFORMANCE EVALUATION OF ACT 12 | Page Question 2 Figure 2: QUESTION 2 - SUMMARY OF EVALUATION DESIGN AND METHODS Evaluation Question 2 Type of Analysis Data Sources and Methods Type and Size Sample Limitations Was the data collected sufficient to measure project progress and results in all three programme components? Were performance reports (weekly, monthly, quarterly, other) sufficient to monitor progress or identify problems in implementation and progress towards results? Description – based on content analysis of expert opinions PMP; M&E Plan; Project reports; ACT project staff. ACT project documentation; Interviewees identified by Evaluation Team and USAID. Components 1,2,3 Process based project design; Limited perceived robustness of the initial baseline survey; unavailability of the end line survey; Lack of pre-determined quantitative targets (C3 – sales; productivity etc. benchmarks). Question 3 Figure 3: QUESTION 3 - SUMMARY OF EVALUATION DESIGN AND METHODS Evaluation Question 3 Type of Analysis Data Sources and Methods Type and Size Sample Limitations Did the data collected by ACT provide evidence of progress towards accomplishment of intermediate results in the three intervention areas? Description – based on content analysis of expert opinions Key informant interviews with ACT project personnel; ACT Project partners; ACT Project reports and contract amendments. ACT project documentation; Interviewees identified by Evaluation Team and USAID. Components 1,2,3 Process based project design; Limited perceived robustness of the initial baseline survey; unavailability of the end line survey; Lack of pre-determined quantitative targets (C3 – sales; productivity etc. benchmarks). Question 4 FINAL PERFORMANCE EVALUATION OF ACT 13 | Page Figure 4: QUESTION 4 - SUMMARY OF EVALUATION DESIGN AND METHODS Evaluation Question 4 Type of Analysis Data Sources and Methods Type and Size Sample Limitations Did the project design take into consideration gender issues such as gender division of labour, time use, control of resources, access to finance, etc.? Description – based on content analysis of expert opinions Focus group sessions with gender organisations/female workers in selected regions (Value chains: (1) pomegranate, (2) dairy, (3) aquaculture; (4) dairy, (5) hazelnut, (6) cold storage). Gender Focus Group Discussion i. Public Union for Gender Equality and Women’s Initiatives ii. “Women of the XXI Century” Public Union iii. Women’s Association for Rational Development (WARD). To be organized by Evaluation Team in sample regions – target = 4-8 per focus group session including representatives from all identified participants Components 2 (Gender mainstreaming in legislation and regulatory framework). Component 3 (Value chains as regards gender division of labour, time use, control of resources, access to finance). Limited availability of sex disaggregated data; Unavailability participants due to holiday period. FINAL PERFORMANCE EVALUATION OF ACT 14 | Page Question 5 Figure 5: QUESTION 5 - SUMMARY OF EVALUATION DESIGN AND METHODS Evaluation Question 5 Type of Analysis Data Sources and Methods Type and Size Sample Limitations Are various results reported accurate and verifiable? Have the results achieved by the project been cost effective and sustainable? Description – based on content analysis of expert opinions PMP; M&E Plan; ACT progress reports. Institute for Scientific Research on Economic Development (under MoED). Interviewees identified by Evaluation Team and USAID. (WTO: cost benefit analysis and WTO related activities). Components 1,2,3 A robust cross-component cost effectiveness analysis may not be readily available in the project FINAL PERFORMANCE EVALUATION OF ACT 15 | Page Group B – Progress in Key Interventions Areas (Components 1-3), Results and Impacts Question 6 Figure 6: QUESTION 6 - SUMMARY OF EVALUATION DESIGN AND METHODS Evaluation Question 6 Type of Analysis Data Sources and Methods Type and Size Sample Limitations Assess the progress and impacts of the value chain operations under the ACT Project in terms of : v. Technology transfer and the relevance of technologies transferred; vi. Strengthening farm advisory and input delivery services; vii. Capacity building of postharvest and processing facilities; viii. Linking producers with markets. Description – based on content analysis of expert opinions Focus group sessions with farmers in selected regions; ACT PMP; M&E Plan (Records and evidence of new technology/ies introduced; Financial and sales records; Loans received; new supply orders and contracts signed). To be organized by Evaluation Team in sample regions – target = 8-10 per focus group session including representatives from all identified participants. Component 3 (1. Percent of farmers, processors, and others who have adopted new technologies or management practices 2. Percentage change in the value of overall portfolios of agricultural loans or leasing agreements made by financial institutions receiving project assistance; 3. Number of new agricultural loans or leasing agreements made by financial institutions receiving project assistance; 4. Percent change in the value of Unavailability participants due to holiday period. Impact is to be gauged in terms of outcomes and intermediate results. Robust and clearly defined counterfactual is not available to conduct an impact assessment and is outside of the scope of this evaluation. FINAL PERFORMANCE EVALUATION OF ACT 16 | Page Figure 6: QUESTION 6 - SUMMARY OF EVALUATION DESIGN AND METHODS Evaluation Question 6 Type of Analysis Data Sources and Methods Type and Size Sample Limitations international exports of the following targeted agricultural commodities as a result of USG assistance etc.) (Value chains: (1) pomegranate, (2) dairy, (3) aquaculture; (4) hazelnut, (5) cold storage). Question 7 Figure 7: QUESTION 7 - SUMMARY OF EVALUATION DESIGN AND METHODS Evaluation Question 7 Type of Analysis Data Sources and Methods Type and Size Sample Limitations Assess the range of policy reform efforts supported, progress and the status (e.g. legislative status for each reform supported) and priority reforms to be supported in the future. Description – based on content analysis of expert opinions. Key informant interviews with key personnel in relevant government ministries; ACT progress reports; PMP and M&E plan. (President Office – Legal Department; Office for Cooperation in South Caucasus. SDC, SECO; American Chamber of Commerce etc. and WTO – as Interviewees identified by Evaluation Team and USAID. Components 1,2 Overall legislative and regulatory reform (Business Enabling Envoronment; Trade and Agriculture etc.) and inter-governmental policy reform cooperation. None perceived FINAL PERFORMANCE EVALUATION OF ACT 17 | Page Figure 7: QUESTION 7 - SUMMARY OF EVALUATION DESIGN AND METHODS Evaluation Question 7 Type of Analysis Data Sources and Methods Type and Size Sample Limitations per question 8) Question 8 Figure 8: QUESTION 8 - SUMMARY OF EVALUATION DESIGN AND METHODS Evaluation Question 8 Type of Analysis Data Sources and Methods Type and Size Sample Limitations Evaluate the progress made in Azerbaijan’s accession to WTO accession and further support to be extended. Description – based on content analysis of expert opinions. Key informant interviews with key personnel in relevant government ministries. (WTO Department, MOED; WTO Accession Commission, MFA; Copyright Agency; Ministry of Agriculture; Ministry of Finance State; Ministry of Health; Tariff Council; Committee on Standardization, Metrology and Patent; American Chamber of Commerce; Independent Consumers Union). WTO Working Group notes and records of progress. Interviewees identified by Evaluation Team and USAID. Component 2 (Consumer seal program, HACCP; policy reform; Sanitary and phytosanitary control; TBT/SPS; Tariffs Council; Policy reform) Seljan Verdiyeva – Attaché (currently in Geneva) Group C – Effectiveness of Technical Assistance FINAL PERFORMANCE EVALUATION OF ACT 18 | Page Question 9 Figure 9: QUESTION 9 - SUMMARY OF EVALUATION DESIGN AND METHODS Evaluation Question 9 Type of Analysis Data Sources and Methods Type and Size Sample Limitations How effective was STTA provided by Contractor in meeting the needs of the project clientele? Assess the impact of the STTA on the beneficiary in terms of relevance of STTA recommendations and their adoption. Description – based on content analysis of expert opinions Key informant interviews with key personnel in relevant government ministries (MOED; MFA; MinFin; Ministry of Agriculture), other partners, BDS, CSOs and farmers (Value Chains). Interviewees identified by Evaluation Team and USAID Components 1,2,3 Opinion bias of interviewees as regards semi-structured and open ended interviews. Impact is to be gauged in terms of outcomes and intermediate results. Robust counterfactual is not available and outside the scope of this evaluation. Relevance of STTA to the needs is to be assessed in this case. Output and outcomes is to be assessed in terms of (i.e. Legislation amended; Value Chains adoption of technologies, methods, practices etc.) and to gauge outcome of the STTA in real teams). Question 10 Figure 10: QUESTION 10 - SUMMARY OF EVALUATION DESIGN AND METHODS Evaluation Question 10 Type of Analysis Data Sources and Methods Type and Size Sample Limitations How effectively has the Description – based on Key informant interviews Interviewees identified by Evaluation Opinion bias of interviewees FINAL PERFORMANCE EVALUATION OF ACT 19 | Page Figure 10: QUESTION 10 - SUMMARY OF EVALUATION DESIGN AND METHODS Evaluation Question 10 Type of Analysis Data Sources and Methods Type and Size Sample Limitations ACT Project created the awareness about the issues it worked on among the private sector, donors, civil society and government agencies in Azerbaijan? content analysis of expert opinions with key personnel in relevant government ministries, CSOs, private sector partners and farmers. M&E reports (PR; visibility; media) & records. Team and USAID. Components 1,2,3 as regards semi-structured and open ended interviews. Group D – Sustainability of the ACT Project Question 11 Figure 11: QUESTION 11 - SUMMARY OF EVALUATION DESIGN AND METHODS Evaluation Question 11 Type of Analysis Data Sources and Methods Type and Size Sample Limitations Assess the extent of stakeholder ownership of interventions undertaken by ACT Project (provide examples) Description – based on content analysis of expert opinions Key informant interviews with key personnel in relevant government ministries, partners and farmers Interviewees identified by Evaluation Team and USAID Components 1, 2, 3 On-gong legislative and regulatory reform for Business Enabling Environment; on-going WTO accession; inter-governmental cooperation; Public Private Dialogue; Public Consultation; M&E Framework for Policy Limited effective means of verifying responses. FINAL PERFORMANCE EVALUATION OF ACT 20 | Page Figure 11: QUESTION 11 - SUMMARY OF EVALUATION DESIGN AND METHODS Evaluation Question 11 Type of Analysis Data Sources and Methods Type and Size Sample Limitations Implementation in place and functioning etc.) Government support to Value Chain development. Question 12 Figure 12: QUESTION 12 - SUMMARY OF EVALUATION DESIGN AND METHODS Evaluation Question 12 Type of Analysis Data Sources and Methods Type and Size Sample Limitations What processes, systems and management were put in place to ensure that the results and impact of the ACT Project will be sustainable? Description – based on content analysis of expert opinions Key informant interviews with key personnel in relevant government ministries; ACT staff. Interviewees identified by Evaluation Team and USAID Components 1, 2 (Legislative and regulatory reform for Business Enabling Environment; WTO; inter￾governmental cooperation; Public Private Dialogue; Public Consultation; M&E Framework for Policy Implementation etc.) Component 3 Sustainability of assistance to Value Chains. Impact is to be gauged in terms of outcomes and intermediate results. Robust counterfactual is not available and outside the scope of this evaluation. Opinion bias of interviewees as regards what is perceived ‘result’ versus ‘impact’. Question 13 Figure 13: QUESTION 13 - SUMMARY OF EVALUATION DESIGN AND METHODS FINAL PERFORMANCE EVALUATION OF ACT 21 | Page Evaluation Question 13 Type of Analysis Data Sources and Methods Type and Size Sample Limitations What were the obstacles to sustainability and what measures should be taken to increase sustainability in future projects? Description – based on content analysis of expert opinions Key informant interviews with key personnel in relevant government ministries; ACT project staff. Interviewees identified by Evaluation Team and USAID; Components 1,2,3 Legislative and regulatory reform for Business Enabling Environment; WTO; inter￾governmental cooperation; Public Private Dialogue; Public Consultation; M&E Framework for Policy Implementation etc.) Sustainability of assistance to Value Chains. Some limitations with effective means of verifying responses. FINAL PERFORMANCE EVALUATION OF ACT 22 | Page Question 14 Figure 14: QUESTION 14 - SUMMARY OF EVALUATION DESIGN AND METHODS Evaluation Question 14 Type of Analysis Data Sources and Methods Type and Size Sample Limitations Appraise the role of cost sharing approach to STTA in terms of creating ownership of the beneficiary and sustainability. Description – based on content analysis of expert opinions Key informant interviews with key personnel in relevant government ministries; ACT project staff. Interviewees identified by Evaluation Team and USAID Cost sharing arrangements clarified FINAL PERFORMANCE EVALUATION OF ACT 23 | Page 4. Matrix of Regions and Beneficiaries The Evaluation Team will ensure that chosen matrix of the regions is balanced and is fully representative of the key project activities that have taken place in Baku and in-country. The Team will break into two teams each consisting of 2 specialists. Given time constraints and perceived impact of summer holiday period on the planning of evaluation, the priority will be given to clusters and panels of meetings when travelling to the regions. On completion of the Desk Study of the ACT key project documentation, the following breakdown of destinations for visits is proposed: Team 1: Mr Colin Maclean and Dr. Ibrahim Mammadzedeh Destination: Baku, Quba, Khachmas, Agjabadi & Barda. Destination Component 1 & 2 Partners and beneficiaries Baku Component 1 – BEE / overall legislative, regulatory, tax and financial reform o Key line ministries; o AmCham; o Central Bank of Azerbaijan Gender Focus Group Discussion o Ms. Ulviyya Mammadova, Leader of the Public Union for Gender Equality and Women’s Initiatives o Ms. Mahira Amirhuseynova, Chairperson of the “Women of the XXI Century” Public Union o Ms. Shahla Ismayilova, Women’s Association for Rational Development (WARD) Quba Component 3 o Executive Committee Deputy o Kheyreddin Jabbarov, Marketing Center o Guba Experiment Station o Guba Regional Agro Science Centre – (Apple and Aquaculture sectors) o Agricultural Input Dealers Association (AKTIVTA) o Business Innovation Center (BIC) Khachmas Component 3 o Executive Committee Deputy o The Northwest Aquaculture Association o Mahir Abbasov, Trout Hatchery o Mr. Mahir Abbasov, northern aquaculture hatchery o Business Innovation Center (BIC) Agjabadi Component 3 o Executive Committee Deputy o AIM o Dairy farmers Barda Component 3 o Executive Committee Deputy o AgroTech LLC of Barda (dairy implementing partner) o Dairy farmers Team 2: Mr Danil Samoilenko & Dr. Fagan Aghayev Destination: Baku, Goychay, Zheki, Zagatala & Lenkaran FINAL PERFORMANCE EVALUATION OF ACT 24 | Page Destination Component 2 & 3 Partners and beneficiaries Baku Component 2 – WTO accession related o WTO Department, MOED; WTO Accession Commission, MFA; Copyright Agency; Ministry of Agriculture; Ministry of Finance State; Ministry of Health; Tariff Council; Committee on Standardization, Metrology and Patent; American Chamber of Commerce; Independent Consumers Union. Goychay Component 3 o Executive Committee Deputy o Mr. Mustafa Badalov, the director of the Goychay M-PRO dairy and juice plant located in the Goychay region. o Agricultural Input Dealers Association (AKTIVTA) Sheki Component 3 o Executive Committee Deputy o Sheki Business and Innovation Centre o Mr. Hamid Mikaelov / Anar Mikayilov Girkhbulag trout fish farm o Azerbaijan Agribusiness Center (AAC) Zagatala Component 3 o Executive Committee Deputy o Parviz LLC (hazelnuts implementing partner) o Mr. Eyyub Jankayev, local carp farmer o Azerbaijan Agribusiness Center Lenkaran Component 3 o Executive Committee Deputy o Janub Agribusiness Centre o JAC of Lankeran (dairy implementing partner) o Two dairy farms in Lenkaran 5. Limitations of Evaluation Design There are several limitations inherent to the design of this evaluation. 1. Evaluation is being implemented across the summer holiday period. This may cause unavailability if some key respondents among ACT project partners and beneficiaries. Emphasis will be given to focus group and expert panel discussion as opposed to individual interviews. 2. Several evaluation questions represent broad areas that may be subject to separate evaluations are likely to be beyond the scope of this evaluation. 3. As some key informants may decline to be interviewed, there is a possibility of selection bias, i.e. those respondents who choose to be interviewed might differ from those who do not in terms of their attitudes and perceptions, affiliation with government/non-government structures, and socio-demographic characteristics and experience. 4. Since a number of questions during the interviews will deal with issues that took place in the past, recall bias cannot be excluded. As ACT activities were launched in 2010, some respondents may find it difficult to accurately compare organizational arrangements/access to services three years from prior to now. 5. ACT project employed large numbers of teams of international and local staff not all of whom are still available to answer evaluator’s questions. Given high turnover of project staff over the course of 2 years, continuity and intellectual memory are additional challenges faced by the evaluation. 6. Geographical areas under evaluation are potentially vast and remote which may limit time for data gatherings given travel involved reaching some of the remoter based beneficiaries and partners. FINAL PERFORMANCE EVALUATION OF ACT 25 | Page 7. There is a known tendency among respondents to under-report socially undesirable answers and alter their responses to approximate what they perceive as the social norm (halo bias). The extent to which respondents will be prepared to reveal their true opinions may also vary for some questions that call upon the respondents to assess the performance of their colleagues or people on whom they depend upon for the provision of services. To mitigate this limitation, ME&A will: provide the respondents with confidentiality and anonymity guarantees, where possible; conduct the interviews in the settings where respondents feel comfortable; and establish rapport between the interviewer and the respondent. 8. Other possible limitation may include low response rates and disproportionate regional participation in one-on-one interviews. If this is the case, the results will represent opinions of the most active and open respondents whose experience with ACT is likely to differ from that of those who decline to be interviewed. To avoid this outcome, full cooperation of ACT staff and national counterparts in arranging interview process is required. FINAL PERFORMANCE EVALUATION OF ACT 26 | Page ANNEX 1 Interview Profile and Illustrative Questionnaires FINAL PERFORMANCE EVALUATION OF ACT 27 | Page INTERVIEWEE QUESTIONNAIRE FOR FARMER FOCUS GROUPS (ACT Project) Interviewee Background and Questionnaire 1 Farmer focus group 2 Main activities of focus group 3 Region and district of the operation 4 When businesses started 5 Interviewees have direct experience working with ACT Project ACT Project Component 3 Interviewee Questionnaire Q.1 Have the ACT Project and BDS providers been able to meet your expectations in development of your activities? How? A.1 Q.2 Are you satisfied with the services provided to you by the BDS selected? A.2 Q.3 How was relation between Project/BDS providers and you as partner/beneficiary? Were there any obstacles? A.3 Q.4 Were there any requirements by ACT Project and BDS providers for reporting purposes? What was the format and frequency? A.4 Q.5 Was short and long term technical assistance provided by ACT Project and BDS providers relevant and effective for your activities? How do you rate the quality of assistance provided? A.5 Q.6 What in your opinion were the best aspects of short and long term assistances provided? FINAL PERFORMANCE EVALUATION OF ACT 28 | Page A.6 Q.7 Did the ACT Project and BDS providers in Key Intervention Areas have an impact on your development? What do you think could have been done differently? A.7 Q.8 Were results achieved by BDS and ACT Project able to impact on sustainability of created interventions? If “Yes” than how? If “No” please give an example of obstacles. A.8 Q.9 Have your businesses reached the level of being sufficiently profitable to assure sustainability? What financial records do you keep to track and monitor performance? A.9 Q.10 Was any analytical study assistance provided by the ACT Project and/or BDS to you? How often? A.10 Q.11 How often did the Project team and/or BDS representatives visit your business and what were the reporting routines required by them? Were any monitoring visits conducted by them? How often? A.11 Q.12 What have the ACT Project and BDS done regarding gender issues? A.12 Q.13 Were the Project and BDS able to raise awareness about issues they worked on among partners and beneficiaries? A.13 Q.14 Have the ACT Project and BDS had any missing elements that you would advise as useful to have in the future? A.14 FINAL PERFORMANCE EVALUATION OF ACT 29 | Page INTERVIEWEE QUESTIONNAIRE FOR LOCAL FINANCIAL INSTITUTION PARTNERS (ACT Project) Interviewee Background and Questionnaire 1 Name of organization 2 Main activities of organization 3 Name of person interviewed 4 Position of interviewee 5 Interviewee had direct experience working with ACT Project? ACT Project Component 1 Interviewee Questionnaire Q.1 How were the local partner F.I. s selected A.1 Q.2 Is the ACT approach to local FIs relevant? Is it consistent with the policies of the GOAJ? A.2 Q.3 Are the objectives and achievements of the country specific ACT project consistent with the policies of the GOAJ? A.3 Q.4 Has the situation changed since the approval and/or implementation of the ACT project? A.4 Q.5 To what extent has the ACT project complemented the USAID strategy for economic growing though the agribusiness and WTO accession in Azerbaijan? A.5 Q.6 Have the contributions by the Local partner FIs been provided as planned? A.6 Q.7 How efficient is ACT project approach to Local FIs? What is comparative parameters for credit and lending, risk management, MIS improvement, etc.? A.7 FINAL PERFORMANCE EVALUATION OF ACT 30 | Page Q.8 Do the Local IFs interventions support the thematic objectives of the ACT project in principle and in practice? A.8 Q.9 Which indicators show that the intended changes are starting to take place? A.9 Q.10 Who will take over the responsibility of financing the activities, or have they become self￾sustaining? A.10 INTERVIEWEE QUESTIONNAIRE FOR LOCAL WTO ACCESSION INSTITUTION PARTNERS (ACT Project) Interviewee Background and Questionnaire 1 Name of organization 2 Main activities of organization 3 Name of person interviewed 4 Position of interviewee 5 Interviewee had direct experience working with ACT Project? ACT Project Component 2 Interviewee Questionnaire Q.1 Are the roles and responsibilities for WTO accession between the involved institutions clear and appropriate? A.1 Q.2 What is the quality of ACT project management, incl. technical assistance, monitoring and reporting, resource and personnel management, financial management, cooperation and communication between stakeholders, for example PPD? A.2 FINAL PERFORMANCE EVALUATION OF ACT 31 | Page Q.3 What are the distinctive features of USAID’s support to GOAJ for WTO accession? A.3 Q.4 From the perspectives of the USAID and the central government institutions involving the designing WTO accession legislation, how has worked ACT project approaches? A.4 Q.5 Has the capacity of the national institutions to carry out ACT project related works increased organizational potential for Azerbaijan WTO accession? A.5 Q.6 Has the ACT project stakeholder’s networking led to more permanent professional relationships between the parties? A.6 Q.7 Conferences/Seminars have been organized to raise the awareness – has this been a useful tool for understanding consequences of WTO accession? Trainings have been implemented by ACT project to improve the knowledge about WTO accession legislative, regulatory and institutional aspects – has this been a useful tool? A.7 Q.8 Have the cross-cutting themes of the USAID development policy been included thoroughly into the ACT project focused on WTO accession and which themes were considered? A.8 Q.9 How sustainable USAID/ACT project's outcomes related to country WTO accession can be expected to be (also in terms of local ownership)? A.9 Q.10 Which kinds of capacity enhancements the ACT project’s WTO accession activities are expected to lead to, and for whom (legislation and institutions)? A.10 INTERVIEWEE QUESTIONNAIRE FOR VALUE CHAIN PARTICIPANTS (Component 3) (ACT Project) Background Information of Interviewee 1 Name of interviewee 2 Name of organization 3 Main activities of business 4 Position of person interviewed 5 Region and district of the organization 6 Legal status of organization 7 When business started 8 Interviewee have direct experience working with ACT 9 Nature of interviewee’s relationship with ACT 10 Nature of organizations relationship with ACT Measure of effectiveness, progress and outcomes – SOW Evaluation Questions: 1, 5, 6, 9, 11, 12 Direct Beneficiaries – Farmers (Dairy, pomegranates, apples, hazelnut, aquaculture) Q.1 What were the benefits to you of BDS services? For example:  farming advice;  inputs (seeds, fingerlings);  support with marketing linkages and partner search;  cold storage;  technology advice and transfer;  adaptation and adoption of new technologies;  support with higher productivity and income;  reduced costs; increased sales, income and productivity). Do you attribute these benefits directly to ACT activities? A.1 Q.2 Did ACT improve the quantity and quality of business services available to you? If yes, in which way? A.2 Q.3 Have the costs of using professional advice reduced and availability of required increased as a result of the ACT? A.3 Q.4 Did you or your employees participate in any training events sponsored by ACT directly or an BDS assisted by ACT? A.4 Q.5 What training was received? A.5 Q.6 How effective was the training? What in your opinion were the best aspects of the training? A.6 Q.7 In which areas would you like to see more training or professional/technological support? A.7 Q.8 What other Did you receive any other technical assistance from ACT or an ACT￾assisted BDS? What was the nature of the technical assistance? A.8 Q.9 How do you rate the quality of technical assistance provided to you? A.9 Q.10 In terms of the quality of training/technical assistance, how do you rate the quality of: - ACT and BDS consultants? - Partners and specialists identified with their support? A.10 Q.11 Did training and professional support provided by ACT and BDS increase your yield? Can you provide examples in percentage prior to the ACT and after ACT assistance? A. 11 Q.12 Did training and professional support provided by ACT and BDS increase your income and reduced costs? Can you provide examples in percentage prior to the ACT and after ACT assistance? A. 12 Q.13 Did training and professional support provided by ACT and BDS increase your sales? Can you provide examples in percentage prior to the ACT and after ACT assistance? A. 13 Q.14 Did training and professional support provided by ACT and BDS increase your productivity? A.14 Q.15 Has the level of investment increased? From external sources or you? A.15 Q.16 Has the lending process improved? Did you find it easier to apply and obtain loans? A.16 Q.17 Do you feel that overall as a result of the ACT support your business has become more competitive? A.17 Q.18 How relevant was the assistance provided to you and your needs and concerns on a scale 1 to 10? A.18 Q.19 Has your business practice, management and arrangements changed as a result of the ACT support? If yes, what arrangements have you put in place to continue same practices without similar support from an external provider? A.19 Q.20 Do you feel that as a result of ACT support your access to markets improved? A.20 Q.21 Do you feel that as a result of ACT support your design of products, services, or processes improved? A.21 Q.22 Do you feel that as a result of ACT support your customer service improved? A.22 ANNEX D: OVERALL MONITORING & EVALUATION MATRIX ANNEX D                                                              45 Total aggregate figure targeted till end of the project i.e. 30 September 2013 46 Data available up to 30 June 2013 47 Includes Year 1 figure which in this case is actual and not target due to change in PMP 48 Data taken from Year 3, 3rd Quarterly Report ACT Overall Monitoring and Evaluation Matrix Period of Performance: October 2010 –September 2013 Indicators Target 45 Level of Achievement (Actual)46 Source(s) of Verification Comments/ Reasons for Deviation (if any) COMPONENT 1 AND COMPONENT 2 1-2.1 No. of direct exposures of public education / advocacy 173247 150748 Component 1&2 Teams; CSOs; ACT M & E Records Expected to be achieved in last quarter 1-2.2 No. of indirect exposures of public education / advocacy 42 40 Component 1&2 Teams; CSOs; ACT M & E Records Result achieved 1-2.3 No. of institutions / organizations undertaking capacity/competency strengthening as a result of USG assistance 19 16 Component 1&2 Teams; CSOs; ACT M & E Records Expected that at least 2 more will be added in the last quarter COMPONENT 1 1.1 Implementation of modern risk assessment system in line with international best practices by Central Bank n/a No Component 1 Team; Central Bank; ACT M & E Records The CBA chose a consultant that required additional resources directly from CBA which substantially delayed the process. ACT presented several highly qualified alternative candidates but they were rejected by the CBA 1.2 No. of people trained in using the outputs of the risk assessment system for banks at the Central Bank 40 None Component 1 Team: Central Bank; ACT M & E Records This cannot be completed without the system being implemented which is not expected 1.3 No. of people trained with the goal of improving or working more effectivley with the business enabling environment 394 703 Component 1 Team; ACT M & E Records Result exceeded 1.4 No. of agencies that introduce improvements into their inspection regimes as the result of project assistance 4 1 Component 1 Team; ACT M & E Records ACT is expecting to complete one more by the end of the project. So this may become 2 1.5 No. of business environment laws, regulations or administrative procedures drafted, submitted for public/stakeholder consultation, submitted for legislative consultation or passed / approved 25 17 Component 1 Team; ACT M & E Records; GOAJ ministries / agencies ACT anticipates at least 4 more in the last quarter to be drafted. 1.6 No. of people completing a project supported e-Learning program on AFL/CFT 1200 11 Financial Monitoring Service (FMS) Implementation of the testing was delayed by the client post launching of the system. Target included wider audience which was scaled back by the client to focus just on banks and other financial institutions which is a much smaller number. 1.7 No. of currency transaction reports sent from reporting entities to the FMS 568,030 571,858 Financial Monitoring Service (FMS) Result achieved 1.8 No. of suspicious transaction reports sent from reporting entities to the FMS 63,623 57,993 Financial Monitoring Service (FMS) Largely completed 1.9 No. of cases referred from the FMS to the appropriate authority 16 35 Financial Monitoring Service (FMS) Targets were reduced in Years 2 and 3 with the revision of the PMP as agreed with USAID. This is reflected in the year 3 Q3 quarterly report. Target exceeded COMPONENT 2 2.1 No. of USG supported training events that relate to improving the trade and investment environment 24 34 Component 2 Team; ACT M & E Records Result achieved 2.2 No. of participants in trade and investment environmental trainings 300 172 Component 2 Team; ACT M & E Records Apparently Year 1 numbers were under￾reported. Many events were provided but not included in the totals for some reason. An M&E person was only hired in July 2011. Project anticipated having more in Q4 bringing it closer to the target. 2.3 No. of new requests, offers, revised offers, or other formal text that are submitted by a host country as part of international trade talks attributable to USG assistance 15 44 Component 2 team; ACT M & E Records Result exceeded 2.4 No. of bilateral, plurilateral and multilateral meetings toward negotiating accession to agreements, conventions and international organizations 16 38 Component 2 Team; ACT M & E Records; Implementing Partners Result exceeded 2.5 No. of legal, regulatory, or institutional actions taken to improve implementation or compliance with international trade and investment agreements due to support from USG-assisted organizations 18 41 Component 2 Team; ACT M & E Records; Implementing Partners Result exceeded COMPONENT 3 3.1 No. of individuals who have received USG supported short-term agricultural sector productivity training 22,000 23,501 Component 3 Team; ACT M & E Records; Implementing Partners Result exceeded 3.2 No. of firms receiving USG supported assistance to improve their management practices 220 164 Component 3 Team; ACT M & E Records; Implementing Partners 3.3 No. of firms receiving USG assistance compliant with international quality control, environmental and other process voluntary standards or regulations 10 7 Component 3 Team; ACT M & E Records; Implementing Partners This may wind up only being 6. Companies are closer but may not be finished due to financial resource limitations by the participating companies 3.4 Percentage of farmers, processors and others who have adopted new technologies or management practices as a result of USG assistance 40% See Comments Component 3 Team; ACT M & E Records, Farmer Survey, Producer Survey The rate of adoption is the one reported at 62% (farmers) and 23% (processors). If comparing baseline to Year 3 there is a change of 15% for dairy and another 10% for aquaculture. However, overall levels of adoption are at 74% and 76% respectively 3.5 Percentage change in yield of targeted commodities as a result of USG assistance 1) 12% - Pomegranate (2) 25% Component 3 Team; ACT M & E Records, Further improvements are expected in productivity in each of                                                              49 Achieved in Year 2 – nothing to date in Year 3 50 Only global figure given for target and not allocated to a specific product as in the results 51 Achieved in Year 2 – nothing to date in Year 3 52 Achieved in Year 2 – nothing to date in Year 3 Dairy (3) 33% Aquaculture (1) 33.6%49 (2) 8.7% (3) 21.8% Farmer Survey the sectors especially for pomegranate. Dairy productivity has increased by 14% over the two years and aquaculture 17% over two years 3.6 Percentage change in value of purchases from smallholders of targeted commodities as a result of USG assistance 25%50 (1) 9.2% (2) 44.6% (3) 25.8% Component 3 Team; ACT M & E Records, Farmer Survey Results in process for Year 3 3.7 Percentage change in sales in the following targeted commodities by processors as a result of USG assistance (1) 12% Dairy (2) 13% Hazelnuts (3) 13% Pomegranate (4) 13% Cold storages (1) 5%51 (2) 6% (3) 17% (4) 0% Component 3 Team; ACT M & E Records, Processor Survey Results in process for Year 3 3.8 Percentage change in the value of international exports of the following targeted agricultural commodities as a result of USG assistance 1) 12% Hazelnuts (2) 7% Pomegranate (1) 5% (2) 16% Component 3 Team; ACT M & E Records, Processor Survey Results in process for Year 3 3.9 No. of jobs created as a result of USG assistance 1300 160552 Component 3 Team; Farmer Survey; Processor Survey; ACT M & E Records Result exceeded 3.10 No. of new agriculture loans or leasing agreements made by financial institutions receiving project assistance 24,821 28,566 Component 3 Team; ACT M & E Records; Implementing Partners Result exceeded 3.11 Percentage change in the value of overall portfolios of agricultural loans or leasing agreements made by financial institutions receiving project assistance. 111% 79.14% Component 3 Team; ACT M & E Records; Implementing Partners Results in process ANNEX E: LIST OF INTERVIEWEES Evaluation of the USAID - ACT Project List of Meetings CONTACT PERSON POSITION ORGANISATION LOCATION DATE 1 Donna Stauffer Acting Mission Director, A.I. USAID Azerbaijan Baku July 29, 2013 2 Elnara Bayramova Program Development Specialist (Budget) USAID Azerbaijan Baku July 29, 2013 3 Aytan Gahramanova Project Management Specialist Economic Growth Office USAID Azerbaijan Baku July 29, 2013 4 Samir Hamidov Project Management Specialist Economic Growth Office USAID Azerbaijan Baku July 29, 2013 5 Tamilla Mammadova Monitoring & Evaluation and Project Design Specialist USAID Azerbaijan Baku July 29, 2013 6 Fikret Shirinov Deputy Head of Tax Policy Department, tax modeling Ministry of Finance Baku July 31, 2013 7 Emil Guliyev Tax policy and revenue department, activities related to property tax modeling Ministry of Finance Baku July 31, 2013 8 Namig Tagiyev Head of Technical Regulation and Standardization Department State Committee for Standardization, Metrology and Patent of the Republic of Azerbaijan Baku July 31, 2013 9 Rahima Guliyeva Assistant to Chairman State Committee for Standardization, Metrology and Patent of the Republic of Azerbaijan Baku July 31, 2013 10 Gulchin Alasgarova Head of Division, Division for WTO Accession Issues Department of Foreign Trade Policy and WTO Ministry of Economic Development Baku August 01, 2013 11 Sanan Shabanov Chief Advisor, Department on Cooperation with International Organizations Ministry of Economic Development Baku August 01, 2013 12 Fuad Aliyev Head of Department, International Cooperation Department Financial Monitoring Service under the Central Bank of the Republic of Azerbaijan Central Bank of the Republic of Azerbaijan Baku August 01, 2013 13 Anar Salmanov Deputy Director, Financial Monitoring Service Central Bank of the Republic of Azerbaijan Baku August 01, 2013 14 Parviz Bakhshaliyev Chief Specialist Central Bank of the Republic of Azerbaijan Baku August 01, 2013 15 Melani Schultz Chief of Party USAID Contractor, ACT Project Baku August 01, 2013 16 Elkhan Mikayilov Component 2 Team Leader USAID Contractor, ACT Project Baku August 01, 2013 17 Jhale Hajiyeva Executive Director Azerbaijan Microfinance Association (AMFA) Baku August 02, 2013 18 Matthias Feldmann Deputy Regional Director, First Secretary Swiss Confederation (Swiss Embassy) Baku August 02, 2013 19 Eyyub Huseynov Director ICU Mediator Independent Consumers Baku August 02, 2013 Union 20 Musayev Nizami Expert Independent Consumers Union Baku August 02, 2013 21 Ramiz Salmanov Head of Legal Department State Veterinary Service Baku August 02, 2013 22 Alekper Guliyev Head of Administration State Procurement Agency Baku August 05, 2013 23 Samir Jafarli Deputy Chairman of Board of Directors AzerCredit, Non-Banking Organization Baku August 05, 2013 24 Shahla Ismayil Chairwoman Women`s Association for Rational Development (WARD) Baku August 05, 2013 25 Seymur Jafarov Director Azsertifika Baku August 06, 2013 26 Gubad Ibadoglu Member of Management Board Economic Research Center Baku August 06, 2013 27 Samir Novruzlu Chairman of the Board Quality association Baku August 06, 2013 28 Viktor Qasimov Head of State Sanitary Services Ministry of Health Baku August 06, 2013 29 Nargiz Nasrullayeva￾Muduroglu Executive Director AMCHAM Baku August 06, 2013 30 Yashar Farajov Component 3 leader Sibley, ACT Project Baku August 06, 2013 31 Olaf Heidelbach Attache, Programme Manager EU Delegation Baku August 07, 2013 32 Mahmud Mamedguliyev Deputy Minister of Foreign Affairs Ministry of Foreign Affairs Baku August 07, 2013 33 Chingiz Farajov Head of Department Ministry of Agriculture Baku August 07, 2013 34 Henning Twesten Country Director GIZ Baku office Baku August 07, 2013 35 Shalala Kamilova Project Officer, Private sector development in South Caucasus Program GIZ Baku office Baku August 07, 2013 36 Saida Bagirli Senior Operations Officer The World Bank Group Baku August 07, 2013 37 Bahram Ismayilov Project Coordinator Isgandarov and Partners, LLC Baku August 07, 2013 38 Tokay Alizade First Deputy Chairman of the Board AGBank Baku August 07, 2013 39 Zaur Haciyev Local expert ACT Project Goychay August 12, 2013 40 Ibrahim Bakhishov Farmer, Head of Pomegranate Association Head of Pomegranate Association Goychay August 12, 2013 41 Zakir Ahmadov Farmer Member of Pomegranate Association Goychay August 12, 2013 42 Nizameddin Ismayilov Farmer Member of Pomegranate Association Goychay August 12, 2013 43 Habil Ahmadov Farmer Member of Pomegranate Association Goychay August 12, 2013 44 Nazim Asgarov Head of Production department AZNAR Goychay August 12, 2013 45 Asif Ahmadov International relations department AZNAR Goychay August 12, 2013 46 Shirzahid Khamiyev Dairy Specialist ACT Project Lankaran August 12, 2013 47 Mobil Penjaliyev Director JANUB Agribusiness Center Lankaran August 12, 2013 48 Lutveli Zeynalli Director Araz LLC Goychay August 13, 2013 49 Bayram Huseynov Manager-Coordinator Agro Tech LLC Barda August 13, 2013 50 Yasin Hasanov Trainer Agro Tech LLC Barda August 13, 2013 51 Ramil Safarli Branch director AGBank Barda August 13, 2013 52 Hasanov Asif Farmer Nazirli village Barda August 13, 2013 53 Muradov Ceyhun Farmer Nazirli village Barda August 13, 2013 54 Zeynalov Xosrov Farmer Nazirli village Barda August 13, 2013 55 Asgarov Farrux Farmer Nazirli village Barda August 13, 2013 56 Babayev Isa Farmer Nazirli village Barda August 13, 2013 57 Abbasov Maharram Farmer Nazirli village Barda August 13, 2013 58 Sattarli Tayyar Farmer Nazirli village Barda August 13, 2013 59 Islam Bayahmadov Apple specialist ACT Project Guba August 13, 2013 60 Minad Pashayev Apple local specialist Cold storage Guba August 13, 2013 61 Nazir Jafarov Dairy specialist ACT Project Agcabedi August 14, 2013 62 Aliyev Telman Farmer Shahsevan village Agcabedi August 14, 2013 63 Aliyev Yaqub Farmer Shahsevan village Agcabedi August 14, 2013 64 Haciyev Feruz Farmer Shahsevan village Agcabedi August 14, 2013 65 Alizade Dilqam Farmer Shahsevan village Agcabedi August 14, 2013 66 Aliyev Adil Farmer Shahsevan village Agcabedi August 14, 2013 67 Guliyev Xaleddin Farmer Shahsevan village Agcabedi August 14, 2013 68 Aliyev Faiq Farmer Shahsevan village Agcabedi August 14, 2013 69 Aliyev Ramil Farmer Shahsevan village Agcabedi August 14, 2013 70 Aliyev Ilkin Farmer Shahsevan village Agcabedi August 14, 2013 71 Sabir Ganizade Aquaculture specialist ACT project Sheki August 15, 2013 72 Hamid Mikayilov Manager Girxbulag trout farm Sheki August 15, 2013 73 Anar Mikayilov Manager Girxbulag trout farm Sheki August 15, 2013 74 Ziyaveddin Aliyev Hazelnut local expert ACT Project Zagatala August 15, 2013 75 Aliyev Muslim Agronomy expert Parviz LLC Zagatala August 15, 2013 76 Mahmudov Eldar Finance manager Parviz LLC Zagatala August 15, 2013 77 Mollayev Suleyman Processing manager Parviz LLC Zagatala August 15, 2013 78 Dan Fitzpatrick Component 1 Team Leader ACT project Baku August 15, 2013 79 Himalay Mamishov Senior Advisor, International Relations Department Cabinet of Ministers Baku August 16, 2013 80 Vugar Hagverdiyev Co-founder, Board member Azerbaijan Hazelnut Association Zagatala August 16, 2013 81 Kamaladdin Maharramov Co-founder, Chairman North West Aquaculture Association Zagatala August 16, 2013 82 Isayev Mahammad Farmer, Member of Association North West Aquaculture Association Zagatala August 16, 2013 83 Ibrahimov Ramil Farmer, Member of Association North West Aquaculture Association Zagatala August 16, 2013 84 Ahmadov Mustafa Farmer, Member of Association North West Aquaculture Association Zagatala August 16, 2013 85 Mammadov Ramil Farmer, Member of Association North West Aquaculture Association Zagatala August 16, 2013 86 Dadash Maharramov Manager ESDER LLC Zagatala August 16, 2013 87 Zulfiqar Qurbanov Manager AzerStar LLC Zagatala August 16, 2013 88 Matanat Rasulova Head of macroeconomic research’s department Institute for Scientific Research on Economic Develop. (under MoED) Baku August 17, 2013 89 Huseynov Fuad Head of Department for International Relations Central Bank Baku August 19, 2013 90 Aliyev Toghrul Head of Risk Management Department Central Bank Baku August 19, 2013 91 Sevda Amirova Head of the Financial Management Department Central Bank Baku August 19, 2013 92 Araz Aliyev Deputy Chief of State Service on Anti-Monopoly and Consumer Rights Protection State Service on Anti￾Monopoly and Consumer Rights Protection Baku August 20, 2013 93 Islam Musayev Head of Legal Department State Service on Anti￾Monopoly and Consumer Rights Protection Baku August 20, 2013 94 Kanan Mustafayev Development Assistance Specialist, Program Office USAID Azerbaijan Baku August 23, 2013 95 Robert Lopes Acting Mission Director USAID Azerbaijan Baku August 23, 2013 ANNEX F: LIST OF DOCUMENTS REVIEWED ACT Documentation Work Plans Name of Document Date ACT Year 1 Annual Work Plan October 1, 2010 – September 30, 2011; ACT Year 2 Annual Work Plan October 1, 2011 – September 30, 2012; ACT Year 3 Annual Work Plan October 1, 2012 – September 30, 2013 ACT Year 3 Annual Work Plan: Supplemental Activities October 1, 2012 – September 30, 2013; Annual Reports Name of Document Date ACT FY2011 Year1 Progress Report October 1, 2010 – September 30, 2011 ACT FY2012 Year 2 Progress Report October 2011 – September 2012 Quarterly Reports Name of Document Date ACT FY2011 Q1 Progress Report October 1 - December 31, 2010 ACT FY2011 Q2 Progress Report January 1 -March 31, 2011 ACT FY2011 Q3 Progress Report April 1 - June 30, 2011 ACT FY2012 Q2 Progress Report January1 - March 31, 2012 ACT FY2012 Q3 Progress Report April 1 - June 30, 2012 ACT FY2012 Q4 Progress Report July 1 - September 30, 2012 ACT FY2013 Q1 Progress Report October 1 - December 31, 2012 ACT FY2013 Q2 Progress Report January 1 - March 31, 2013 Local Expert’s Monthly Report to ACT project December 2012, Guba Project Papers Name of Document Date ACT Monitoring and Evaluation Plan – Years 1,2,3 ACT Baseline Farmer Survey Report, Executive Summary 2011 ACT Midterm Farmer Survey Report October – November, 2012 Business Training & Extension Training Materials Name of Document Date Statement of Work, Final Performance Evaluation of ACT Project Other Name of Document Date Assessment of the Private Sector Competitiveness Enhancement Project December 29, 2010 Trade and Investment Reform Support Program in Azerbaijan, Final Report December 29, 2010 State Program for Socio Economic Development of Regions of Azerbaijan for 2009-2013 years Assessment of the Agriculture and Rural Development Sectors in the Eastern Partnership Countries December 2012 Agriculture Credit Insurance Mechanism – Executive Summary – Prepared by Oxfam on behalf of Azerbaijan Micro-Finance Association 2010 Azerbaijan Micro-Finance Association 2012 Annual Report AMCHAM 2012 Annual Report AMCHAM – Tax Code Amendments – Impact - Issue 17 2013 ANNEX G: ILLUSTRATIVE QUESTIONNAIRE INTERVIEWEE QUESTIONNAIRE FOR BSPs PARTICIPANTS (Component 3) (ACT Project) Background Information of Interviewee 1 Name of interviewee 2 Name of organization 3 Main activities of organization 4 Position of person interviewed 5 Region and district of the organization 6 Legal status of organization 7 When organization was established 9 Nature of interviewee’s relationship with ACT 10 Nature of organizations relationship with ACT Measure of effectiveness, progress and outcomes – SOW Evaluation Questions: 1, 5, 6, 9, 11, 12 Direct Beneficiaries – BDS providers and consulting companies Q.1 Have you received TA from international staff of ACT project? If yes, how do you rate its quality and relevance to your organization on a scale 1 to 10. A.1 Q.2 Have you received TA from local staff of ACT project? If yes, how do you rate its quality and relevance on TA to you organization on a scale 1 to 10. A.2 Q.3 How efficient (timely) was ACT in fielding such expertise? A.3 Q.4 How effective was the training – has the team of your ToT strengthened? What in your opinion were the best aspects of the training? A.4 Q.5 How efficient was ACT in fulfilling its contractual obligations (payments etc.)? A.5 Q.6 Has you organization launched new services (trainings, advisory packages etc.) or products (training packages; know-how) as a result of ACT project support? If yes, specify each. If not, why? A.6 Q.7 Have the number of your clients among farmers or farmer associations increased as a result of the ACT support? A.7 Q.8 Have you acquired any additional private sector clients? A.8 Q.9 Have you acquired any additional public sector (donors, GOAZ) clients? A.9 Q.10 What in your business practice do you do differently as a result of the ACT project support? A.10 Q.11 In which areas would you like to see more training or professional/technological support? A. 11 Q.12 Do you feel that overall as a result of the ACT support your organisation has become more competitive and therefore more sustainable? A.12 ANNEX H: SUMMARY OF BDS PROVIDER’S RESPONSES Table 1: Summary of BDS providers’ responses BDS provider/ Consulting company Type of TA received Overall rating of TA (1-10 scale) Internal BDS ToT strengthened Increased competitive ness Areas of key achievements Areas of weaknesses/further support needed ARAZ LLC Internatio nal STTA; Local STTA 9-10 Significant Significant a. Training materials; b. Soil analysis and fertilizer use; c. Planting schemes and patens; d. Marketing; e. Attained additional clients among farmers/groups and processors (AZNAR; ARAZ processing; Sabirabard conserve etc.). a. Packaging; b. Cold-storage; c. Business training (i.e. cash flows etc.) PARVIZ LLC Local STTA 7-8 Overage Overage a. Training materials; b. Soil analysis and fertilizer use; c. Planting schemes and patens; d. Number of clients among farmers increased from 150 to 1.500 as a result of ACT support; e. Private sector contracts signed with processing enterprises. a. International export requirements; b. Visits to Italian and Turkish BDS providers and processors; AgroTech LLC Internatio nal STTA; Local STTA 3-4 To a limited extent (BDS is currently not active as a corporate entity) To a limited extent (BDS is currently not active as a corporate entity) a. Updated training materials on artificial insemination. a. Theory based class room training on artificial insemination has produced no or limited outcomes; b. Group of clients has not been increased; c. Pilot artificial insemination bank/insemination programme; d. High yield livestock. Quality Associates Internatio nal STTA; TCN professio nals 0 9-10 Overage To a limited extent a. Updated knowledge of best practice HACCP. a. None reported. ANNEX I: ACT – SPENDING PER COMPONENT ACT Spending per Component through June 2013 Year 1 Year 2 Year 3 Total Component 1 CBA Support $  667,705.32 $  605,340.98 $  416,610.88 $  1,689,657.18 Business Environment $  837,961.10 $  678,222.32 $  444,034.67 $  1,960,218.09 Subtotal $  1,505,666.42 $  1,283,563.30 $  860,645.55 $ 3,649,875.27 Shared Costs ‐ Cross Cutting Labor and ODCs $  983,155.91 $  898,302.72 $  623,910.12 $ 2,505,368.75 Total $  2,488,822.32 $  2,181,866.02 $  1,484,555.67 $ 6,155,244.01 Component 2 Trade Reform $  1,137,235.57 $  940,264.79 $  778,278.85 $ 2,855,779.21 Subtotal $  1,137,235.57 $  940,264.79 $  778,278.85 $ 2,855,779.21 Shared Costs ‐ Cross Cutting Labor and ODCs $  742,581.40 $  658,045.01 $  564,199.80 $ 1,964,826.20 Total $  1,879,816.97 $  1,598,309.80 $  1,342,478.65 $ 4,820,605.42 Component 3 Agriculture Activities $  1,042,933.05 $  1,073,172.69 $  1,007,895.64 $ 3,124,001.37 Access to Finance $  362,798.64 $  533,719.14 $  281,645.45 $ 1,178,163.24 HACCP Compliance $  220,897.26 $  176,373.95 $  96,024.82 $ 493,296.04 Subtotal $  1,626,628.96 $  1,783,265.79 $  1,385,565.91 $ 4,795,460.65 Shared Costs ‐ Cross Cutting Labor and ODCs $  1,062,140.89 $  1,248,019.87 $  1,004,442.06 $ 3,314,602.82 Total $  2,688,769.85 $  3,031,285.65 $  2,390,007.97 $ 8,110,063.47 Project Total $  7,057,409.14 $  6,811,461.47 $ 5,217,042.29 $ 19,085,912.90 ANNEX J: ACT – STTA DAYS BY COMPONENT ACT – STTA DAYS BY COMPONENT Year 1 Year 2 Year 3 TOTAL STTA Ex‐Pat DAYS STTA Local DAYS STTA Ex‐Pat DAYS STTA Local DAYS STTA Ex‐Pat DAYS STTA Local DAYS STTA Ex‐Pat DAYS STTA Local DAYS Component 1 CBA n/a n/a 340 80 451 130 791 210 Business Environment n/a n/a 241 35 191 20 432 55 SUB TOTAL n/a n/a 581 115 642 150 1,223 265 Component 2 Improving Trade n/a n/a 510 ‐‐ 475 130   985 130 WTO Accession n/a n/a 100 ‐‐ 225 ‐‐   325 ‐‐ SUB TOTAL n/a n/a 610 ‐‐  700 130 1,310 130 Component 3 Agriculture Activities n/a n/a 523 120 520 378 1,043 498 Access to Finance n/a n/a   80 ‐‐  178 ‐‐    258 ‐‐  HACCP Compliance (Resource Firms $) ‐‐  ‐‐  ‐‐  ‐‐  ‐‐  ‐‐  ‐‐  ‐‐  SUB TOTAL n/a n/a 603 120 698 378 1,301 498 Cross‐cutting M & E n/a n/a 80 ‐‐ 35 ‐‐    115 ‐‐    TOTAL n/a n/a 1,874 235 2,075 658 3,949 893 Source: ACT ‐ Annual Work Plans ANNEX K ACT – COST BREAKDOWN Oct 2010 – Sept 2011 Oct 2011 – Sept 2012 Oct 2012 – Sept 2013 Technical Assistance Year 1 Q1 Oct 1 – Dec 31 Year 1 Q2 Jan 1- Mar 31 Year 1 Q3 Apr 1 – Jun 30 Year 1 Q4 July 1 – Sept 30 Year 2 Q1 Oct 1 – Dec 31 Year 2 Q2 Jan 1 – Mar 31 Year 2 Q3 Apr 1 – Jun 30 Year 2 Q4 July 1 – Sep 30 Year 3 Q1 Oct 1 – Dec 31 Year 3 Q2 Jan 1 – Mar 31 Year 3 Q3 Apr 1 – Jun 30 Year 3 Q4 July 1 – Sep 30 TOTAL $ Expat LTTA 109,923 235,707 293,003 305,690 400,612 271,687 322,850 282,877 134,079 148,716 212,717 -- 2,717,861 Expat STTA 100,950 564,125 868,385 948,327 528,843 370,179 419,261 360,011 270,719 256,966 553,355 -- 5,241,121 Local Staff 173,841 304,429 353,783 387,848 531,804 394,481 463,562 457,099 382,171 475,461 549,820 -- 4,474,299 LTTA Allowances 30,326 41,047 74,085 80,800 286,800 52,626 83,923 60,860 60,535 55,071 68,228 -- 894,301 TOTAL 415,040 1,145,308 1,589,256 1,722,665 1,748,059 1,088,973 1,289,595 1,160,847 847,504 936,214 1,384,120 -- 13,327,582 GRAND TOTAL 4,872,269 36.55% 5,287,474 39.68% 3,167,838 23.77% 100% ANNEX L CONFLICT OF INTEREST FORMS Attachment J.4 Disclosure of Conflict of Interest for USAID Evaluation Team Members Name Colin maclean Title Mr Organization ME & A Evaluation Position?  Team Leader Evaluation Award Number (contract or other instrument) USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) I have real or potential conflicts of interest to disclose. No  If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1.Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2.Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3.Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4.Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5.Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6.Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 06/05/13 40 Attachment J.4 Disclosure of Conflict of Interest for USAID Evaluation Team Members Name Danil Alexander Samoilenko Title Mr Organization Mendez England & Associates (ME&A) Evaluation Position? Team Leader X Team member Evaluation Award Number (contract or other instrument) RFTOP SOL-112-13-00005 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Final Performance Evaluation of the Azerbaijan Competitiveness and Trade (ACT) Project I have real or potential conflicts of interest to disclose. Yes X No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1.Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2.Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3.Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4.Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5.Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6.Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 08/05/2013 40 Attachment J.4 Disclosure of Conflict of Interest for USAID Evaluation Team Members Name Ibrahim Mammadzadeh Title Mid-level Trade/Economic Growth Evaluation Specialist Organization ME & A Evaluation Position Team Leader X Team member Evaluation Award Number (contract or other instrument) USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) ACT I have real or potential conflicts of interest to disclose. Yes x No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1.Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2.Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3.Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4.Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5.Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6.Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date May 01, 2013 40 Attachment JA Title Organization Evaluation .Position? If yes;'a:Wswel:ed above,Tdisdosetbe{ollowing facts: Real or potential coi1}1icts ofinterest maYinc(ude; but are notlimited to~ · · ... · · .. · • ..··· I. Close family i?Jetnber who is brrenip/oft¢ o/t~e USAID operating unit mimalfiHgtHe pl'o}eCI(s) being evaluaiedbrihe iltq:J{eme!lti'ng orgm1iZ~tion(s) whose profect(s)dre1/ei11g evalua(ecf. · · · · . 2: Financial inteh:st; t}falis direct, or iS sigflificant. thoilgh ibditec;t; Jf(the impl¢menting .. · · organi;atlon(s) whose pfo]e.cts ,are be.ing. evaluat~d or in·the·9.utc¢i1Je ofthe e\'aluati()n, 3. Current or previous dif'ecf orsig11ij]p.ai1tthough. ivdirecc~perieiJ.cftwiJhth.g_l2i:oft_t1(iL!ieiiJ~ evaluated, inc111ftlngjnvo/v~i1Jeht{ntlie.project design or previous Iterations ojthe•:j:/rbj~Ci. · 4. Current or-previous workexperieitce o.t•seeking . eitzplo;iment .wilhjhe (!SAID operating unit · · lJI(lrw,ging tljeevalt~ation (Jr the Impleme11ting. otgaitfiation(s)whose pr:oject(s) are be:tng •. · evalutlied: · · .. 5. Curreri or prel!i()us work experience wttJ{ ciri · •.. • organizatim1.that'mCiylJe·seel1·as. q(l tndusiry cpinpeiitqr with tfte i117pleme.riting otgilhization(s) whose project(s) are beil'lg di:Huated .. .· .· .. ·.·•·· .. • 6; Preconceived ideas towirrd i'tldiv~~tkllS, gttJZ!ps; · organizatfohs, or objective~ ofth'e:piu;ticular projects and (Jrgailizatiol'ls beiiJg e.vaii!ated that could bias the eva{udtion: I Certi!Y {l) .that lhave compiHe'd