For Public Use Assessment of the Private Sector Competitiveness Enhancement Project (PSCEP) December 29, 2010 For Public Use [1] TABLE OF CONTENTS EXECUTIVE SUMMARY ............................................................................................................ 2 A Review of the Principal Project Activities under PSCEP .......................................................................... 4 Sector Selection ....................................................................................................................................... 5 Impact Measures – Targets and Benchmarks .......................................................................................... 6 Attribution ................................................................................................................................................ 7 A Deeper Look at Project Activities and Outcomes – Key Questions from the Assessment SOW ...........9 Lessons-Learned and Recommendations for Future Projects .............................................................15 Annexes: .............................................................................................................................. 19 Statement of work for evaluation of Private Sector Competitiveness Enhancement Project (PSCEP) ................................................................................................................................ 19 Background ............................................................................................................................................19 Objectives ...............................................................................................................................................20 Scope of work .........................................................................................................................................22 List of Interviewees....................................................................................................................................25 Notes from Beneficiary Interviews ......................................................................................... 27 Aquaculture ............................................................................................................................................27 Fruits and Vegetables .............................................................................................................................29 Different Businesses ...............................................................................................................................32 Business Development Organizations ....................................................................................................33 For Public Use [2] EXECUTIVE SUMMARY The Private Sector Competitiveness Enhancement Project (PSCEP) is a $6,588,557, three year project of USAID/Azerbaijan’s Economic Growth office and implemented by Chemonics International. The project began in September 2008, and will end in December 2010, ten months earlier than originally anticipated. USAID/Azerbaijan has commissioned this assessment both as way of judging the effectiveness and impacts of the project, and also to provide guidance to USAID’s successor project – Azerbaijan Competitiveness and Trade (ACT) project – which is starting in late 2010. Although there were some design and implementation problems that will be described below, and a difficult business environment overall, PSCEP did achieve at least two major successes: improvements in the dairy sector that have brought about substantial income gains for thousands of farmers, and re￾establishment of the aquaculture industry in Azerbaijan after it had declined by 90% over the past twenty years. In most respects, PSCEP follows a fairly conventional competitiveness/value chain methodology. However, three aspects of the design and/or implementation stand out as being different from similar projects in other countries, and it does appear that these differences have contributed to short-falls in overall project effectiveness: An unnecessarily complex analytical process used in sector selection Too much emphasis on investment as a primary project objective An unrealistic timeline for achieving indicators As a result, after fifteen months of the three-year project, 62% of the budget had been consumed. Once the sector-specific technical assistance began, it does appear that much of it was effective. There were particularly beneficial results in the dairy sector, and aquaculture. Other firm-specific successes were noted with green-house farming and high-value agriculture (strawberries), but because often a small number of individuals benefitted, the cost-benefit of this assistance is questionable. It should be noted that private sector economic growth is difficult in Azerbaijan, mainly because of monopolization of key sectors in the economy and a general lack of transparency. Even a well-designed and well-run project will have difficulty achieving significant results in Azerbaijan. In addition, the project design contained certain mandated activities and targets that affected the implementation in negative ways. That said, it is also clear that the project fell short of achieving some of its benchmarks and targets and that the work of the implementer was not always cost-effective. Among the key lessons that can be taken from this project are: 1) the necessity of having targets and benchmarks that are appropriate for the project goals and design; 2) to realize that proper timing and sequencing of actions is essential to sound project management; and 3) to manage the use of project resources so that they are applied in the most cost-effective manner relative to project goals. For Public Use [3] Introduction Summary Background The Private Sector Competitiveness Enhancement Project (PSCEP) is a $6,588,557, three year project of USAID/Azerbaijan’s Economic Growth office and implemented by Chemonics International. The project began in September 2008, and will end in December 2010, ten months earlier than originally anticipated. USAID/Azerbaijan has commissioned this assessment both as way of judging the effectiveness and impacts of the project, and also to provide guidance to USAID’s successor project – Azerbaijan Competitiveness and Trade (ACT) – which is starting in late 2010. The assessment was conducted during October 2010. The principal members of the assessment team were Neal Nathanson, Senior Competitiveness Advisor from the E&E Bureau, and David Tsiklauri, Economic Growth Specialist with USAID/Georgia. The team was assisted by various members of the USAID/Azerbaijan staff including Shelia Young, Amanda Fong, Aytan Gahramanova, Kanan Mustafayev, and Ulyana Butenko. The assessment consisted of a review of project documents (scope of work, work￾plans, quarterly and annual reports, and other technical papers prepared by the project), followed by field visits and interviews with Chemonics staff in Washington and Baku, and approximately 40 beneficiaries and partners throughout Azerbaijan. The Annex contains the SOW for the assessment, a list of individuals interviewed, and summaries of key interviews. Summary Findings An assessment is a backward look at actions taken years earlier, and it must rely on a review of documents and interviews with individuals who were engaged in those activities at the time. In the specific case of PSCEP, none of the key personnel either with the implementer or USAID/Azerbaijan are present any longer, which makes it difficult for the assessors to gain a full picture of conditions at the time actions were taken. It is possible that circumstances at the time warranted certain actions that in retrospect may now seem questionable. For example, indicator targets were established at a time when Azerbaijan was the fastest-growing economy in the world, due to rapid expansion of the oil sector. This may have led to establishing some targets and benchmarks that now may seem unrealistic.1 1 The Implementer (Chemonics International) was given a draft of this report and allowed to submit responses. Some of those responses have been incorporated into the report: “ As the implementer, Chemonics was consistently seeking to be in compliance with contract requirements and highly responsive to the COTR. . . . It was at the strong insistence of the original COTR that the investment indicator objective be pursued and all other objectives and targets were secondary or tertiary.. . After the original COTR left and the sectors and sub-sectors changed and refocused, PSCEP was still being held accountable for the original indicator targets which continued to divert attention from areas and activities of possible greater impact and success.” For Public Use [4] It should be noted as well that private sector economic growth is difficult in Azerbaijan, mainly because of monopolization of key sectors and a general lack of transparency. Even a well-designed and well-run project will have difficulty achieving significant results in Azerbaijan. In addition, the project design contained certain mandated activities and targets that affected the implementation in negative ways. That said, it is also clear that the project fell short of achieving some of its benchmarks and targets, and that the work of the implementer was not always cost- effective. A Review of the Principal Project Activities under PSCEP In most respects, PSCEP follows a fairly conventional competitiveness/value chain methodology2 : sectors and sub-sectors with growth potential are identified through an analytical process; targeted sectors receive technical assistance to undertake improvements in productivity, quality, and market outreach in order to expand revenues and jobs; domestic and foreign investment is encouraged and promoted as a way to bring about productivity improvements; Business Development Service (BDS) providers are integrated into the service delivery efforts to provide a means of sustaining the technical assistance at the end of the project; and a grants component provides financial incentives that can encourage strategic actions and investments. However, three aspects of the design and/or implementation stand out as being different from similar projects in other countries, and it does appear that these differences have contributed to short-falls in overall project effectiveness: 1) The analysis to identify targeted sectors using a Domestic Resource Cost Analysis (DRC) along with sector-specific analyses and strategies was far more time and resource consuming than necessary. 2) The impact targets (particularly the investment targets) were quite high and possibly not attainable, and this biased the selection of sectors and activities in ways that were intended to help the implementer achieve the targets, but in fact were diversions from the real intent of the project. 3) There is a logical sequence to business and sector upgrading and expansion: a) targeted sectors/enterprises are identified; b) upgrade strategies are formulated; c) businesses receive knowledge transfer, new skills, new market opportunities, etc.; d) investments (both physical and human capital) are undertaken; e) productivity and/or revenues increase; f) employment (hopefully) increases. Typically this sequence can take two to three years to show tangible results. However, the PSCEP benchmark targets assume that all of these steps should happen simultaneously. Even before sectors and targets were identified and strategies formulated, PSCEP was already trying to influence investments to achieve the investment target, and PSCEP was expected to produce growth in revenues and employment from literally the first day. 2 Over the past five years, more than ten competitiveness/value-chain projects have been launched by missions in the E&E region. Of those, the majority have employed this type of methodology, although with some local variation. The most significant differences between these projects are the degree to which they concentrate on interventions to change the business environment vs. interventions at the firm and industry level. For Public Use [5] Sector Selection Domestic Resource Cost Analysis (DRC)3 The DRC is a complex and analytically intensive process for comparing production costs in domestic industries with similar industries outside of the country. Simply stated, those sectors whose domestic resource costs are lower than those from outside are considered competitively viable opportunities, while those with higher costs are considered non-competitive. During the period of November 2008 – April 2009, the principal consultant responsible for conducting the DRC billed a total of $155,041 for his preparation of this analysis. In addition, an unknown amount of local costs were incurred on data collection, and all of these costs were subject to other indirect costs from the prime contractor. Altogether possibly as much as $500,000 was spent on this analysis, which (in the opinion of the assessors) contributed little to the actual functioning of the project: The data assembled and analyzed by the DRC is of uncertain reliability4 , and subject to wide variance, so that it is difficult to use in decision-making. For example, the analysis of milk production concludes that domestic costs range from 16% to 96% above import costs, and cheese costs range from 6% below to 306% above the import costs. Based on the DRC analysis, dairy would not be a priority sector, but in fact the dairy industry appears to be one of the project’s most successful endeavors. The DRC methodology does not take into account changes in production costs that would result from the application of new technologies and processes – exactly what competitiveness and value chain projects are all about. As a result it ignores the very actions that are intended to be undertaken within the Competitiveness/EG project. In fact, of the six sectors that ultimately became the targets for PSCEP activities, only three were analyzed by the DRC, and of those three, the DRC data suggests that two would be poor choices. It appears that once completed, the DRC did not play a significant role in the actual sector selection5 Sector Action Plans Simultaneous with the DRC, individual sector action plans were also undertaken. According to reports submitted, 10 separate action plans were completed during the period of February – July, 2009. From 3 It should be noted that the use of the DRC within PSCEP was mandated by the project design and the first COTR was a strong advocate for its use and for the size and scope of the analysis. 4 The analysis relies on production cost information collected from local farmers and producers. In Azerbaijan, people are reluctant to provide financial information, so it is not clear that the data used is completely accurate. 5 During the debrief, the question was raised: “If not DRC, what is the best way to select sectors?” While this question cannot be answered adequately in a footnote, when sectors are targeted for project interventions, implementers typically employ a multi-screen methodology with the following features: 1) Market potential – analysis of the prospect for growth; 2) Leadership/willingness to change – gauging interest and willingness of sector leaders: 3) Constraints – Are there substantial constraints that might limit effectiveness of the effort; 4) Strategic direction – How to achieve significant improvement. Production cost analysis can be part of screens #3 and #4, but typically production costs do not drive sector selection. For Public Use [6] the information currently available, it is not possible to calculate the LOE and cost associated with preparing these sector strategies, but in total more than 700 person-days of international consulting was undertaken during the period of January – November, 2009 on sector-related activities.6 Since most of the Action Plans were submitted in July 2009, it does suggest that a substantial amount of this STTA went into the preparation of action plans. In any event, by December 2009, $4,074,857 (62% of the total budget) had been expended. In other words, almost 2/3 of the total budget was consumed during 1/3 of the project time, leaving only limited resources for technical assistance during the second and third years, and ultimately resulting in the project ending nine months early. Impact Measures – Targets and Benchmarks There were three primary quantifiable indicators to be achieved by targeted enterprises as a result of the activities of PSCEP: A total of $225 million in debt and equity investments An increase in revenues of 50% above the historical trend for firms in the targeted sectors An increase in employment of 30% above the historical trend for firms in the targeted sectors Clearly, these would appear to be ambitious targets, and perhaps unrealistic. However, it should be noted that in 2006-2008, Azerbaijan had the world’s fastest GDP growth rate (in excess of 20% per year), driven primarily by the completion of the oil pipeline to the Black Sea coupled with huge increases in world oil prices during that period. There was euphoria in the country that significant growth in non-oil sectors could follow as well. But there was a curious outcome from the sector selection that rendered most of the growth targets far less strenuous than the percentages would suggest. According to the April 7, 2010 Monitoring and Evaluation Report, most of the targeted sectors had historic growth rates in the 2% to 5% range. For a company or sector growing (for example) at 3% per year, a change in the growth rate of 50% over the historic trend would result in a growth rate of 4.5%. In numerical terms, a hypothetical firm or sector with $1,000,000 in annual revenues, at the historical 3% rate of growth, the company would be expected to generate $1,030,000 the following year. Increasing the growth rate to 4.5% would result in revenues of $1,045,000. In other words, the application of project assistance to this $1,000,000 enterprise would result in a $15,000 added benefit – a result hardly worthy of spending many thousands of dollars of USAID funds. This curious outcome is even more striking in regard to employment. Every one of the targeted sectors is shown to have a 1% annual growth rate in employment. The project target is to increase the growth rate in employment by 30% over the trend, or 1.3% annual growth. A hypothetical company with 100 employees would be expected to add one employee based on the historic growth trend. Increasing this trend by 30% would mean an additional 0.3 employees. A large company or sector with 1,000 6 Without analyzing or disclosing any proprietary cost data from the contractor, one can assume that the full cost of an international consultant – daily rate, indirect costs, travel, and per-diems – can exceed $1500 per day, meaning that more than $1,000,000 was spent in 2009 for international short-term technical assistance. For Public Use [7] employees would need to add only 3 employees as a result of the project assistance in order to achieve the target growth rate of 30% above trend. Despite the fact that on closer analysis these targets do not seem particularly difficult to achieve, it is impossible to tell from the reports what exactly has happened in regard to revenue and employment growth. Nowhere in any of the reports is there a clear table or narrative that identifies by name any company or sector with hard baseline revenues and employment, and how those changed as a result of project activities. The Monitoring and Evaluation report simply asserts that 99 of 135 firms receiving assistance crossed the revenue growth threshold and 47 of 141 crossed the employment threshold. It does appear that significant improvement did take place in two sectors: dairy (milk) and aquaculture. In the case of milk production, the improvement came from teaching small farmers proper sanitation, feeding and collection techniques. As a result, the rejection rate on milk sold to the major dairy plant has dropped from 30% to 3%. In addition, the total quantity of milk supplied has increased by 150% to 200%. Much of this increase has been accomplished through productivity gains and without a significant increase in employment. The revenue growth indicator would be relevant for this activity, but not the employment growth indicator. In the case of aquaculture, the sector had shrunk by 90% since Soviet times, and PSCEP has been instrumental in helping to reverse that trend. Most of the active fish farms have excess capacity because they have been operating at a low level. As they improve, revenues are likely to grow, but not employment – at least not until they reach existing capacity levels. As with dairy, revenue growth may be an appropriate target but not employment. The M&E report (and other reports as well) are more precise on investment targets. In fact, most of the quarterly and annual reports tout the finance and investment as among the most positive features of the project. The September 2010 monthly report claims that $142,466,977 in investment has been generated or leveraged, and that another $178,656,844 is in the pipeline. Attribution Beyond the problem of determining an exact outcome regarding indicators and benchmarks, there is a problem of identifying the extent to which PSCEP’s assistance did cause or contribute to the reported outcome. Whenever assistance was rendered by PSCEP to any partner or beneficiary, PSCEP is claiming credit for all positive activity generated by that partner or beneficiary from that time forward. In the financial sector, PSCEP did provide sector studies, training and technical assistance to local banks on SME finance, leasing, and trade finance. PSCEP now claims credit for all SME lending, leasing, and trade finance from six banks – a claimed impact of $109,364,895 through September 2010. Altogether PSCEP claims to have generated or leveraged $142,466,977 in financing and investment activity. The assessors interviewed senior management from two partner banks – Demir Bank and Rabita Bank. Both acknowledged the beneficial impact of technical assistance and training from PSCEP. Both indicated that the most significant benefit came from the improved quality of the loan portfolio (reduced rate of non-performing loans). Demir Bank believes that the training did help increase loan For Public Use [8] volume by about 10% based primarily on greater confidence of loan officers in preparing recommendations. Rabita Bank does not believe that the assistance added materially to the loan volume. Both indicated that for the future they have sufficient technical assistance from EBRD and IFC and are not seeking additional financial sector training from USAID. While it does appear that PSCEP contributed to the effectiveness of these partner institutions, it is difficult to accept that PSCEP should claim that its efforts were the sole or principal cause of this amount of financing activity. These banks were already engaged in SME lending, but there is no data about the extent of this lending before PSCEP’s assistance. At the same time PSCEP was training banks, the banks were also receiving assistance from other donor organizations including EBRD, IFC, and the German￾Azerbaijani Fund. It is impossible to determine with any degree of accuracy how much additional lending could be attributed to the assistance provided by PSCEP, but certainly it is an exaggeration to claim credit for 100% of the current lending volume of the six partner institutions. For Public Use [9] A Deeper Look at Project Activities and Outcomes – Key Questions from the Assessment SOW The Scope of Work for the PSCEP assessment identified questions in seven subject areas. This section will address these questions. Meeting targets: Have there been any significant or critical gaps in implementation? Are the assumptions established at the beginning of the program still realistic? (Note: This may include but should not focus on an analysis of whether or not the assumptions and/or targets as written at the beginning of the project were realistic then, and, if considered unrealistic, whether the implementer knew, or should have known, that such targets were unrealistic.) Is Chemonics accurately and adequately collecting data to measure program results? For practically every one of the key quantified indicators (investment, revenue changes, employment changes, and export sales), PSCEP has fallen far short of the target. However, because of the way the information is collected and reported, it is difficult to ascertain the true outcome in regard to the indicators and targets. Only the investment target is stated as an actual quantity ($225,000,000). The others are stated as a percentage increase over the long-term trend for the targeted sectors. In regard to the investment targets, as of September 30, 2010, PSCEP claims to have achieved $142,466,977 in investment, and an anticipated pipeline of an additional $178,656,844. However, it is difficult for the assessors to accept that PSCEP should claim full credit for this amount of investment. Their practice is to count all of the SME loans, leases, and trade financing from six banks with which they have MOUs. While it may be legitimate to claim that the banks did realize some benefit from trainings and technical assistance provided by the project, it seems a bit of an exaggeration to claim that all lending that occurred after those trainings is fully attributable to PSCEP’s assistance. At a minimum PSCEP should have applied a methodology similar to that which was to be used with revenue and employment data – to determine the historic rate of growth of SME lending, and then to show how the technical assistance has changed that rate of growth. Although PSCEP was not the sole provider of technical assistance to banks, at least claiming only the amount of new lending that exceeded the historic trend and not the entire amount of lending would be more credible. With regard to the revenue and employment growth targets, there is no data on which to make a reliable judgment. There are no reports that clearly and directly identify the revenues and employment of companies receiving project assistance before and after project assistance. The only report that attempts to get at the matter of revenue and employment targets is the Monitoring and Evaluation report of April 7, 2010. That report simply asserts that 99 of 135 firms assisted achieved the target threshold for revenue growth, and 47 out of 141 assisted firms achieved the employment growth threshold. For Public Use [10] Were the targets suitable and achievable? As was pointed out previously, when the historical growth trend is low, even a seemingly large change in percentage growth rate does not amount to much real change. So increasing a company or sector employment by 30% above an historic trend of 1% still does not amount to very much impact. These percentage increases over a very low historic trend probably could have been achieved although not from Day 1. But while achievable, some probably were not suitable, for at least two reasons: 1) High levels of under-employment particularly on farms would allow for significant growth in revenue with no growth in employment. In this case, revenue growth would be suitable but not employment growth. 2) Unless the project would have been able to directly affect a very large number of individual enterprises and farms, the total impact of this type of assistance would be inconsequential. Impact: Have there been any systemic changes as a result of PSCEP’s work? If so, identify and explain (to the extent possible) what accounts for that change. What are the factors limiting impact? How has the program benefitted the intended beneficiaries? PSCEP can legitimately claim to have contributed to systemic change in the dairy (milk) sector, and aquaculture. Working in the area around Lankaran (southern Azerbaijan), and in association with the Lankaran BDS, there has been a significant improvement in milk production. Azerbaijanis do not drink a great deal of milk, but they consume milk products such as cheese, yoghurt, and sour cream. Pal-Sud, an Azerbaijani￾Turkish company opened a modern dairy processing plant in Lankaran to make these products using a combination of imported milk powder and locally-purchased milk. Initially much of the locally￾purchased milk was considered substandard and unacceptable due to contamination and poor quality. Working with Pal-Sud’s network of milk collection stations, farmers and independent milk collectors have been taught better methods of collecting and handling milk, with the result that the rejection rate has dropped from 30% to about 3%. In addition, the total quantity of milk supplied to the collection stations has increased 150% to 200%. Together, these two improvements have brought about significant income increases for participating farmers. The estimates of true impact vary, but discussions with the manager of one milk collection station and the BDS director suggest the following: Several thousand farmers have made the necessary improvements and have realized this income gain A typical farmer with 10 cows can earn more than 500 Manat ($400) per month A 30% gain in income on $400 per month is about $120, or almost $1500 per year. In a country where many individuals live on $50 per month and $200 per month is considered a median income for workers, this gain and impact is significant. For Public Use [11] This gain amounts to more than $2,000,000 per year in income to farmers in the region and has the potential to grow many times that amount. Further gains can be realized as other farmers adopt these practices. In addition, farmers now have incentive to purchase more cows and to make use of more productive (although more expensive) commercial feed. Local banks do appear to be cooperating by providing loans for farmers to buy cows (although at quite high interest rates). Other regions as well can take up this model, and over the next few years it is not difficult to imagine $5 - $10 million per year in additional benefit coming to farmers. Similar systemic change is taking place in aquaculture. During the Soviet period, there was a significant fish-farming industry in Azerbaijan, but over the past 20 years it has deteriorated by 90%. PSCEP has played a major role in helping to revive this industry. International experts supplied through PSCEP have helped several of these companies with design, technology, input supplies, and management techniques. One of the most significantly beneficial actions by PSCEP has been to facilitate the purchase and import of 500,000 trout eggs from the US. Trout eggs purchased for as little as five-cents each will grow into live fish that can be sold for as much as five dollars one year later. All of the farmers who participated in this program say that had it not been for the help of PSCEP and the U.S. Embassy, this transaction never could have happened. A second purchase of 1,000,000 eggs is now being facilitated. An association of fish farmers is actively in development. Facilities are being expanded and new ones developed. PSCEP can rightfully claim that its assistance was a significant contributor to the restoration of this industry. There are a few scattered examples of success with green house farming, and with the introduction of new high-valued agricultural products (strawberries). While these are interesting and beneficial individual cases, they cannot be considered systemic in nature.7 Lack of cold storage was identified as a major deficiency by PSCEP, and cold storage was targeted as one of the PSCEP sectors. However, cold storage is a more complex situation. Modern cold storage facilities are highly capital-intensive, generally requiring several million dollars in capital. The GOAJ has created special financing sources to finance facilities including cold-storage development through the State Fund for Entrepreneurship and the investment funds such as Azerbaijan Investment Company (AIC). However, like most strategic sectors and activities in Azerbaijan, the tendency toward monopolization limits the broad availability of this funding. PSCEP did provide some technical assistance in support of this sector, and the sector itself is showing significant positive change. However, it is not realistic to claim that PSCEP has played a central role in transforming this sector. There are many factors that have limited the impact of PSCEP, but the most significant ones are outside of PSCEP’s ability to influence. The monopolization of key productive sectors and lack of transparency make it difficult for independent private businesses to operate effectively and efficiently in Azerbaijan. 7 The assistance given for green-house farming and strawberry planting, while interesting, was limited to a small number of beneficiaries. Unless there is an opportunity for major expansion of these programs, it is not cost effective to bring U.S. experts to benefit only a few individuals. For Public Use [12] While not every one of PSCEP’s activities were fully effective, the difficult business environment in Azerbaijan is by far the most significant factor limiting impact. Effectiveness and efficiency Was recruitment of advisors effective? Did foreign consultants leave behind any/adequate knowledge with local counterparts? Did Chemonics make wise use of STTA, LTTA and local hire time and resources? Was the project staffing plan properly designed and organized? Did the contractors build relationships with host country counterparts in a way that enhanced program objectives? Did Chemonics identify and engage the necessary and critical host country counterparts? Have the results achieved by Chemonics been at an acceptable cost? It appears that Chemonics did not always make good use of the STTA/international consulting resources. It appears that the least effective consultants came during the early stage of the project, and as the project became more grounded in sector and firm-level assistance, the consultants began to provide greater value. The Domestic Resource Cost (DRC) analysis seems to have contributed very little toward its stated purpose – to identify sectors and targets for value chain upgrading. It does seem that some government officials and others admire it for its analytical characteristics, but it has little practical value in achieving PSCEP’s objectives. Because of the early emphasis on finance, efforts were made to work with the two state investment funds: Azerbaijan Investment Company (AIC) and the Caspian International Investment Company (CIIC). Considerable technical resources from both LT and ST consultants was directed toward helping these funds with investment analysis and due diligence. However, it became clear after a while that investments by these entities were not made entirely on economic factors, and that the transactions were very slow coming. Consequently, after the first year, PSCEP decided to pull back from support to these entities because of limited results. Absence of focus in the beginning resulted in a scatter-gun approach to sector support. Ten separate sector action plans were produced between March and July, 2009. These action plans were of mixed usefulness. Some were quite useful and are being followed, while others are not. In the case of strawberry planting guide, only one farmer is making use of this work, raising questions about its cost-effectiveness. Host country counterparts, particularly governmental counterparts, are not so easy to engage in Azerbaijan. PSCEP attempted in the beginning to engage with Ministry of Economic Development and with the state investment funds, but in time they found that these relationships were not particularly helpful. PSCEP was not able to establish any kind of productive relationship with the Ministry of Agriculture. PSCEP does seem to have good relations with commercial banks – even to the point that the banks are providing monthly loan data that PSCEP reports as its own accomplishments. For Public Use [13] All things considered, it does not appear that the results which were achieved were at an acceptable cost – or perhaps better stated, much of the money that was spent did not produce particularly beneficial results. Sustainability What evidence is there that Azerbaijani partners have taken ownership of PSCEP activities and outcomes? What are the obstacles to sustainability? What measures should be taken in the last months of the project or just after its closure to increase sustainability? Are processes, systems and management in place to ensure that the results and impact will be sustainable? Did the program create institutional capacity to ensure that the partners properly carry on operations after exit? There are a few very clear examples of partners and beneficiaries taking ownership of project activities, and being in a position to sustain and expand those activities in the future. However, there are probably more examples where this has not happened. At the level of enterprises and farms, in the previously-cited examples of milk production and fish farming, partners and beneficiaries have taken ownership of the project activities. The BDS in Lankaran is in a position to continue working effectively with farmers to increase the numbers of cows and to bring more farmers into the production network. The fish farmers around Sheki, Guba and Ganja, and the BDS’s that serve those regions, are organizing an association that will continue to grow this industry. Assistance and support from USAID should continue under ACT – strengthening further the aquaculture industry and expanding milk production support to other regions. The role of and support to BDS organizations should have been enhanced and expanded by PSCEP. At the beginning PSCEP had a concept for paying incentive bonuses to BDS organizations as a way to encourage them to generate results and help PSCEP achieve its targets and benchmarks. As a result, for the first year, the support to BDS organizations was erratic and unpredictable for them. By the second year the support was reduced as part of overall budget cut-backs. Because of this many of the BDS organizations did not invest much in building their own capacities and did not fully absorb the skill￾building that might have been possible from associations with the international consultants.8 8 It should be noted as well that some of the BDS organizations do not seem very entrepreneurial and may not have extended themselves to take maximum advantage of the opportunity to learn from PSCEP’s consultants. For example, during interviews with the BDS organizations, it appeared that the only one or two of these organizations were very motivated toward seeking out new clients or establishing new fee-based client services. For Public Use [14] The BDS organizations existed before the project started, and most will survive after it ends. Two or three of the five do have some client income that will keep them alive. Some will join other international donor projects and gain additional support. All of the BDS organizations are small – the largest having five or six employees. That, combined with relatively low salaries for workers outside of Baku, means that it does not cost very much to keep an organization going, or to expand its technical capabilities. While it is not desirable to create dependencies and shift the focus of such organizations away from client services and toward donor￾project funding, it does seem that the relative expenditure levels applied to the BDS component in comparison to other project budget elements was out of balance. The project would have had absolutely no way to find and assist the local businesses without BDS-type organizations serving as outreach and delivery agents. The project relied on the BDS organizations to supply impact data, if not to actually deliver services. Unless the project was prepared to open several branch offices at considerably higher cost, it could not have functioned without the BDS organizations. The life-of-project support to each of the five BDS organizations was about $340,000 in total. That is about 5% of the total project budget. For a component that is so critically important to achieving results, this seems a rather small resource allocation. Management: Are the reported results accurate and verifiable? What specific recommendations could serve for future management of such a program? As has been described previously, some of the reported results are not fully verifiable9 . At best the measure of investment and finance seems exaggerated. It is impossible to tell from the reports the true benefit in regard to income and employment gains, but they do not seem substantial. It appears that USAID management was pushing hard in the first year for a robust application of international technical assistance. The implementer was willing to comply and/or unable to resist the demands of the USAID COTR, and as a result project resources were consumed much too fast. By the time the project started delivering beneficial services, 2/3 of project funding was gone. The project will end ten months early, having delivered barely one year’s actual firm and industry-level assistance. Certainly it is clear that resources need to be better and more cost-effectively applied. As noted previously, too much was spent on early-stage analysis, much of which was academic, theoretical, and unnecessary for the selection of sectors and development of sector strategies. Some analysis is necessary, but what PSCEP did seems excessive. Some of the firm-specific technical assistance was limited to a very small number of beneficiaries, which raises a question of cost-effectiveness. Questions for USAID Management: 9 See discussion on pages 5 – 7. For Public Use [15] Has USAID been able to provide appropriate oversight for each activity? Has USAID provided clear and consistence guidance to the implementing partners? How responsive has the implementing partner been to feedback and guidance? Communications: How effective has PSCEP been with highlighting success stories (for internal USAID and external audiences) or weekly reports (for internal USAID audiences)? It does appear that during the first year there were communication problems between USAID management and the implementer – and that at least some of the misapplication of funding was driven by USAID. As noted several times already, the Domestic Resource Cost analysis and other sector analyses were mandated by the project design and contract. The indicators and benchmarks were set by USAID and mandated in the contract. There was a critical period of time during the middle of 2009 when the COTR was away from post due to medical conditions, and management oversight was sporadic at best. FSN staff turn-over at USAID compounded the problem of management oversight. It is not possible (or even necessary) to pinpoint exactly where to place blame for project short-comings. There were weaknesses in the design, in the application of the design by the implementer, and in oversight of the implementer by USAID. Those short-comings combined with a difficult environment for independent small businesses led to a project which did have some notable successes but overall fell far short of its desired impact. The project did produce many reports – perhaps too many. The project submits monthly, quarterly, and annual reports and separately has submitted two Monitoring and Evaluation Reports and two Annual Work-Plans. The Domestic Resource Cost analysis and ten separate Sector Action Plans were prepared and made available to partners and beneficiaries.10 Copies of five “Success Stories” were supplied by the implementer’s home office – two on the aquaculture initiatives, one on financing a cold storage facility, one on ICT/Anti-Piracy, and one on leasing. In the opinion of the current USAID technical office, PSCEP was not very responsive in developing and disseminating success stories. Lessons-Learned and Recommendations for Future Projects This section will present some lessons specifically from PSCEP as well as some other suggestions and recommendation that may be relevant to the new ACT Project. The choice of indicators and benchmarks will drive the implementation strategy of the implementer. Implementers have a contractual obligation to produce results specified in the contract, so they pay a great deal of attention to the indicators and benchmarks. Furthermore, without some indicators, there is no way for USAID to judge the success of a project. 10 Actually nine sector action plans and one technical manual for strawberry production. For Public Use [16] However, the PSCEP experience shows how important it is to have indicators that are suitable to the desired goals and objectives. PSCEP’s implementer focused too much on the “investment” indicator, which in fact should have been considered secondary and not primary, and that misdirection resulted in a misallocation of project resources and a distortion of the true objective. In a relatively undeveloped and undifferentiated economy, identification of target sectors does not require a complex analytical process PSCEP consumed a considerable amount of its budget on the Domestic Resource Cost analysis as a way of selecting sectors. The project design called for a detailed analysis of 32 sectors and sub-sectors from which to pick six sectors and eight sub-sectors. In the end, the sectors that were selected were fairly obvious from the start (dairy, fruits and vegetables) or identified from sources other than the DRC (aquaculture). The terms “sector” and “sub-sector” are not clearly defined (this is not unique to PSCEP – it is a fairly common shortcoming throughout the development field). This lack of precision makes it particularly difficult in agriculture where there can be literally thousands of individual products and product groups, many of which are sub-sectors to other larger sectors. For future value-chain projects, it should be made clearer in the design how the terms “sector” and “sub-sector” will be applied. In a value chain project, “finance”, “cold storage”, and “transportation/logistics” are typically not independent sectors but rather support and facilitation to other productive sectors. The purpose of investment in a value chain project is to introduce new technologies and processes, not as an end in itself. Similarly, cold storage is important to reduce spoilage and make it possible to sell products out-of-season and realize higher prices. Transportation/logistics gets product to market. None of these should be considered as an independent sector in a value chain project. It does not mean that the project should not work with these kinds of businesses, but that they should be seen as supportive and facilitating primary productive sectors. Information technology can be seen as a similarly supportive sector, with the application of technology as a means to enhance the productivity of the primary productive sectors. Investment is important, but often the greatest returns come from relatively small investments The fact that PSCEP had an investment target is not in itself a bad thing – just the target was too large and the emphasis misdirected. Investment can be both physical capital (equipment, facilities, vehicles, etc) and human capital (training and capacity-building). Often, process improvements coupled with relatively inexpensive equipment can produce large returns. The dairy products example in Lankaran demonstrates that. Simple improvements in the way milk is handled produced an immediate 30% income gain. Investment (broadly defined) is usually the first indicator that businesses and farms are responding to a change in local conditions. The willingness of a business or farm to do something differently – to change procedures, to use new seeds or planting techniques, and then ultimately to spend money or incur debt For Public Use [17] for new equipment – indicates that the technical assistance or business environment changes are taking hold. Farmers and SMEs are generally not willing to do something differently unless they believe that by making these changes, they can see real beneficial results fairly quickly thereafter. It is not possible to implement an agricultural value-chain project without an effective system of delivering assistance to enterprises and farms in distant, rural communities. There are two ways to deliver the firm/farm-level assistance: through intermediaries (BDS organizations) or with project branch offices. Using independent intermediaries is usually less expensive than creating project branch-offices, and offers the prospect of creating and strengthening organizations that can carry on activities after the project ends. However, a BDS initiative is not easy to implement effectively. The local entities are typically not very strong. There is often a design contradiction within BDS components of EG/Competitiveness projects. Clients and beneficiaries are not accustomed to paying for services, and in any event the smaller farmers often do not have the means to pay even if they were so inclined.11 But designs often call for the BDS organizations to become sustainable from client fees, while at the same time asking BDS organizations to work with clients that are the least able to pay. With a predictable, longer-term, possibly declining payment formula for BDS organizations, the organizations do have some possibility of building internal skills and positioning themselves for eventually commercialization and sustainability. Value-chain projects often ignore the small-scale fabricators of metal, wood, and plastic products for domestic consumption. In every town in Azerbaijan there are countless small-scale fabrication shops that produce (or possibly repair) all kinds of products. Little is known about these SMEs. These businesses might be a client base for BDS organizations, and they should be encouraged to do more outreach to this group of SMEs. Projects must make an explicit and continuous effort to make sure that whenever international experts are brought into the country that the BDS organizations (or some other relevant entity in the country) can learn from the expertise so that the knowledge and capability is transferred into the country for future benefit. Transformation takes time. True transformation can be a generational matter – requiring 20 to 30 years to be fully realized. In Azerbaijan there is no organized system of education to introduce modern farming practices to local farmers. Farmers learn from their parents, so antiquated practices from Soviet (or maybe even pre￾Soviet) times are being passed down to the newer generation. Azerbaijan could benefit from having agricultural topics and curricula at the high-school level. Clubs similar to 4-H or Future Farmers of America might help introduce more modern farming practices to the next generation. It is often difficult for Azerbaijani farmers and SMEs to visualize 21st century practices without seeing them first-hand. While recognizing some of the negative aspects of study-tours (that they can be 11 The Agro-Information Centre (AIM) has demonstrated that even very small farmers will pay some nominal fees if they see that they can benefit from the services that they receive. For Public Use [18] expensive and difficult to organize and that they can deteriorate into shopping junkets), on a selective basis it could be quite beneficial for some of the more forward-looking farmers and SMEs to observe modern practices outside of Azerbaijan. The more advanced businesses are typically willing to cost￾share these learning opportunities. In general, trips to the US would not be recommended – better options exist in Turkey, Israel, and Eastern Europe. Similarly, STTA should not come only from the US – Turkish, Israeli, and Eastern European experts may be more suitable for Azerbaijan. Typically, USAID projects run three to five years, which is generally not enough time to show true transformation. However, within that time frame it is possible to set in motion some activities that can continue to build on initial successes. ACT is only three years. However, by building on some of the successes from PSCEP – mainly dairy and aquaculture – can go forward quickly, giving another three years on top of the two years of activity undertaken during PSCEP.12 12 The goals of ACT are different from PSCEP. ACT is not primarily a “value chain” project, but instead intends to use real-life examples from firm and sector-specific assistance to identify and attack more systemic problems which, when resolved, can open up greater opportunities for a larger number of farms and SMEs. For Public Use [19] Annexes: Statement of work for evaluation of Private Sector Competitiveness Enhancement Project (PSCEP) 1. Background PSCEP Background: USAID/Azerbaijan’s Private Sector Competitiveness Enhancement Project (PSCEP) is a 3 year, $6.5m project with the goal to enhance the competitiveness of Azerbaijan’s non-oil and gas sectors. It began in October 2008, and was awarded to Chemonics as a full and open competition contract. PSCEP began by conducting a Domestic Resource Cost Analysis of 38 sub-sectors of the Azerbaijani economy in order to answer the question “Which sectors have the greatest chance of becoming competitive in a global marketplace?” This DRC analysis has proved popular with GOAJ counterparts and a useful discussion tool in the agricultural subsidies discussions. Before the analysis was completed, the project began researching value chains in selected sectors. Partly using the outcomes of the DRC analysis and partly using the information collected in this research, PSCEP prepared Sector Action Plans for the following sectors: Dairy Poultry Aquaculture Access to Finance ICT Greenhouses Packaging Food Safety Cold Chain The project engaged, through a competitive local RFP, five local business development service (BDS) providers located in five different regions around the country to provide business development services to clients in their regions. The BDS providers selected are: Lankaran – Lankaran Regional Business Training Center and Azerbaijan Agribusiness Center Jalilabad – Jalilabad Agribusiness Company Guba – Guba Marketing Center Sheki – Sheki Business Innovation Center Ganja – Young Agrarians For Public Use [20] In addition to the sectors listed above, the project spent the first year offering assistance to a number of BDS clients around the country in various unrelated sectors, including furniture production. In September 2009, the project officially narrowed its selected sectors to the following: Dairy Poultry o Broilers o Eggs Aquaculture o Trout o Carp Fruits and Vegetables o Greenhouses o Cold Chain Finance o Debt o Equity ICT o Anti-Piracy o Human capacity As of the end of June 2010, PSCEP has spent 81% ($5,100,794) of its total contract value of $6,588.557. Chemonics plans to close out the project in the fall/winter 2010, with technical staff leaving at the end of October and administrative staff leaving by the end of December. As of the end of June, PSCEP is not meeting several key contract deliverables. Discussions during the spring 2010 semi-annual USAID/Azerbaijan Mission portfolio reviews pointed to the need for an in-house assessment of PSCEP before it closes. The Mission has agreed that this assessment shall be a brief summary of lessons learned rather than a detailed evaluation report conforming to ADS criteria and standards. Purpose of Evaluation This evaluation seeks to achieve the following: Determine the effectiveness, progress, impact and sustainability of PSCEP assistance; Identify the various factors and conditions in the country that have enhanced or limited the effectiveness and impact of PSCEP assistance; Analyze the factors leading to the non-achievement of PSCEP’s contract targets; Summarize major lessons learned for similar USAID projects. 2. Objectives The objectives of the evaluation are as follows: Evaluate the progress made by Chemonics toward meeting the targets. For Public Use [21] Evaluate the successes or failures that are evident for specific program activities, in terms of intermediate result and long term impact, and the reasons behind them. Provide recommendations for follow-on programming, specifically within the 6 sectors selected by PSCEP or in a non-selected but critical sector that may have been left out. (Note: The SOW for the follow-on project has already been written and includes some flexibility for continued work in private sector competitiveness in selected sectors) Key Questions: Meeting targets: Have there been any significant or critical gaps in implementation? Are the assumptions established at the beginning of the program still realistic? (Note: This may include but should not focus on an analysis of whether or not the assumptions and/or targets as written at the beginning of the project were realistic then, and, if considered unrealistic, whether the implementer knew, or should have known, that such targets were unrealistic.) Is Chemonics accurately and adequately collecting data to measure program results? Impact: Have there been any systemic changes as a result of PSCEP’s work? If so, identify and explain (to the extent possible) what accounts for that change. What are the factors limiting impact? How has the program benefitted the intended beneficiaries? Effectiveness and efficiency Was recruitment of advisors effective? Did foreign consultants leave behind any/adequate knowledge with local counterparts? Did Chemonics make wise use of STTA, LTTA and local hire time and resources? Was the project staffing plan properly designed and organized? Did the contractors build relationships with host country counterparts in a way that enhanced program objectives? Did Chemonics identify and engage the necessary and critical host country counterparts? Have the results achieved by Chemonics been at an acceptable cost? Sustainability What evidence is there that Azerbaijani partners have taken ownership of PSCEP activities and outcomes? What are the obstacles to sustainability? What measures should be taken in the last months of the project or just after its closure to increase sustainability? For Public Use [22] Are processes, systems and management in place to ensure that the results and impact will be sustainable? Did the program create institutional capacity to ensure that the partners properly carry on operations after exit? Management: Are the reported results accurate and verifiable? What specific recommendations could serve for future management of such a program? Questions for USAID Management: Has USAID been able to provide appropriate oversight for each activity? Has USAID provided clear and consistence guidance to the implementing partners? How responsive has the implementing partner been to feedback and guidance? Communications: How effective has PSCEP been with highlighting success stories (for internal USAID and external audiences) or weekly reports (for internal USAID audiences)? 3. Scope of work Proposed Methodology: This in-house USAID assessment will be led by Neal Nathanson, who is a private sector specialist working in the Market Transitions team of the USAID’s Europe and Eurasia bureau. A second person will assist Neal with this assessment and will be identified in the coming months. In addition, the Mission may propose additional staff from the Mission’s EG office to participate on the assessment team, for part or all of the assessment activities. The team will work with the Mission’s Economic Growth Office to prepare a planned methodology for the conduct of all assessment work, including an explanation of specific methods to be used to collect information necessary to evaluate PSCEP’s activities. Sheila Young, USAID/Azerbaijan EG Office Director and PSCEP COTR, and Amanda Fong, USAID/Azerbaijan Private Sector Officer and PSCEP Activity Manager, will serve as points of contact for the Mission. Assessment methodology may include: Key Informant Interviews – to be held with a broad sampling of PSCEP beneficiaries; national officials; other donors; implementing partner; academics; USAID and US Embassy economic staff; For Public Use [23] Focus groups (to the extent possible) – to be held with beneficiaries; GOAJ and commercial institutions participating in PSCEP, local organization engaged with PSCEP; other donors; implementing partners; among others; Document reviews – assessments, reports, action plans, evaluations, and financial documents. Key documents on the programs will be provided to the team by the Mission, but the team is expected to research documentation from other sources/organizations as well as needed. Duration of Tasks and Level of Effort: The following descriptions of tasks for the evaluation are as an illustrative guide in developing the work plan for this assessment. 1. Desktop Review of Key Documents (3 workdays): Prior to any field work, the team will review key documents to develop a work plan. The team will review all available documentation describing PSCEP activities, and identify to the Mission any additional documentation needs. The team will also conduct any required meetings with USAID officials in Washington DC and Chemonics home office staff. An initial list of documents for review is included in the annex to this SOW. 2. Development of Work Plan (2 workdays): The work plan will be developed by the assessment team and will be approved by COTR before the team travels to Azerbaijan. The plan will include a proposed schedule for all work to be conducted, proposed dates for submission of draft and final reports and a proposed methodology for the evaluation of specific activities, consistent with the methodology agreed to between the Mission and the team. 3. Entrance Briefing: Upon arrival in Azerbaijan the assessment team will meet with Mission staff to present the assessment objectives and discuss logistics, scheduling and any other issues. If the assessment team arrives in Tbilisi in place of Baku, this meeting can take place via DVC. The Mission will assist coordination with stakeholders and provide additional suggestions for interviews. 4. Field Work (10 workdays): After the entrance briefing the evaluation team will then begin field work according to the evaluation methodology. 5. Draft Write-Up (3 workdays): Upon completion of field work, the team will spend the rest of their time in country writing up a draft assessment report in the Mission. The team will complete a draft assessment report and submit it to the COTR prior to departure from Azerbaijan. 6. In-Country De-Brief: Upon completion of the draft assessment report and prior to departing from Azerbaijan, the team will meet with Mission staff to discuss findings. 7. Final Report and Public Version Final Report (2 workdays): Within two weeks of receipt of Mission comments on the draft report, the team will provide a final written report on the results of the assessment. The written final report shall not exceed 30 pages of text and will include the following: An Executive Summary Background discussion and rationale for the assessment For Public Use [24] An explanation of the methodology used and field work done Findings –empirical facts collected by the assessment team; findings supported by relevant quantitative and qualitative data Conclusions –Evaluators interpretations and judgments based on findings Recommendations –proposed relevant and practical actions for management based on clearly supported conclusions Unresolved issues –review of what remains to be done Annexes, as needed: Recommendations for possible future assistance Description of evaluation method used Data collection instruments Schedules List of persons interviewed Bibliography of documents reviewed Glossary of acronyms used For Public Use [25] List of Interviewees Rayon Name of Organization Interviewed Balacans Balakan Fruits and Vegetable Processed Fruits Abdul Akhsunsky Zagatala Zagatala Carp Hatchery and Association Kemalleddin Maharramov Sheki Sheki BDS Bahram Osmanov Sheki Executive Power of Sheki, Head Murad Jabbarli Sheki Sheki Carp Farm Khatira Akbarova Sheki Sheki Ghirkbulag farm and feed mill Hamid Mikaelov Ganja Ganja N.A.A Company Nadir Abbasov Shamkir Shamkir-Individual Farmer Goja Gasimov Goy-Gol Goy-Gol Gizil Yem (Golden Feed) Company Alovsat Huseynov Goy-Gol Goy-Gol Apex-1 Mirza Mammadov Agjebedi AIM – lab Fazil Jafarov Guba Guba Marketing center (BDS) Kheyraddin Jabarov Guba Guba Un-AGRO LLC BDS (Cold Storage) Zulfugar Mamamdov Gusar Gusar Agro-Line LLC Mahir Abbasoglu Jalilabad Jalilabad Agro Business Center BDS – Adalat Nehmatov Jalilabad Goytepe Village - Elvin 2 Agagulu Huseynov Lankaran Janub Agro-Business Center (Lankaran BDS) Mobil Penjaliyev Lankaran Cenub Agro LLC (fruits and vegetable greenhouse) Mirnazim Bashirov Lankaran Access Bank Rahman Aliyev Lankaran Mahmudavar village Milk Collection Point Elchin Aliyev Baku PSCEP Staff, COP Harvey Schartup For Public Use [26] Baku PSCEP Staff, DCOP Barbara Yale Baku PSCEP Staff, Value Chain Specialist Yashar Farajov Baku PSCEP Staff, Value Chain Specialist Indira Asgerova Baku PSCEP Staff, Value Chain Specialist Arif Agayev Baku USAID/Azerbaijan Economic Growth office Sheila Young Baku USAID/Azerbaijan Economic Growth office Amanda Fong Baku USAID/Azerbaijan Economic Growth office Aytan Gahramanova Baku Demir Bank (commercial bank lending) Rauf Akhundov Baku Rabita Bank (commercial bank lending) Elchin Gadimov Baku Barama Innovation Center Gahraman Kazimov Baku Softline (Microsoft Anti-Piracy GDA) Aliv Salakhov Baku Davachi Broiler LLC (poultry) Zakir Azimov Baku ASENA Ko (Broiler and Egg feed) Vahid Hummatov Washington DC Chemonics headquarters Steve Farkas Washington DC Chemonics headquarters Anna Slother Washington DC Chemonics headquarters Tania Kapoor For Public Use [27] Notes from Beneficiary Interviews Aquaculture 1. Zaqatala carp farm and hatchery. During interview owner of the farm, Mr. Kemalleddin Maharramov mentioned: Project and BDS instructed how to improve carp feeding process and production of fingerlings. With his own savings and a grant from PSCEP, established modern mid size hatchery and plans to improve carp fingerling production and sales. Attended aquaculture sector development meetings and conference organized by Sheki BDS. During those events developed important linkages with other sector players. With neighbor fish farmers plans to form business association to jointly improve their industry supply, marketing and TA needs. Despite the fact that carp business is very profitable, he is unable to receive credit to improve farm facility and purchase additional equipment. Overall he is very satisfied by PSCEP project and Sheki BDS activities. Observation conclusions: Construction of the carp hatchery was recently finished and as PSCEP shortly ends without additional TA, farm could have difficulties to properly operate it. Despite the knowledge received, is unclear how farmer plans to improve fish feeding practices. 2. Sheki carp farm During interview owner of the farm, Ms. Khatira Akbarova mentioned: Started carp business a year ago, and due to several operational mistakes made throughout the initial period, does not expect large income from this harvest. In spite of expected low income, Ms. Akbarova plans to stay in this business and improve farm operations. Ms. Akbarova has looked for business advice prior to the start of her business and during its development period. However, people she consulted with provided her with contradicting recommendations. The only proper information and advise how to run the carp business she received from Sheki BDS. Ms. Akbarova attended aquaculture sector conference, where she discussed her business with other carp farmers and realized mistakes made. At the conference, Ms. Akbarova had the chance to develop contacts with other farmers who promised to help in sales of harvested carp. Plans to continue partnership with BDS and even pay commercial fee for proper consultations. Observation conclusions: For Public Use [28] Ms. Akbarova needs substantial TA before she is able to properly run a carp farm. Demand for carp seems to be higher than supply, but she has no experience with marketing and could have some difficulties, if not assisted in this area. 3. Sheki Chirkbulag fish farm (primarily trout) During interview owner of the farm, Mr. Hamid Mikaelov mentioned: He has run a trout and carp farm for the last 5 years and thanks to a grant and TA from PSCEP and Sheki BDS substantially improved his farm operation. Due to project assistance from the US supplier, Mr. Chirkbulag purchased 400,000 eyed-trout-eggs and hatched 98%. As a result, he already knows where to buy trout eggs and how to hatch them with minimal loss. This year’s harvest proves that improved technology gives higher return. Some of the fingerlings from domestic and imported hatched eggs are sold to neighbor farmers. With his own savings and a grant from project, upgraded feed mill, and thanks to improved feed will satisfy own and other farmers needs. With Sheki fish farmers founded association and are developing services association should provide to its members. He attended several meetings and conference organized by project and Sheki BDS. Recently company was assisted to participate in fish industry exhibition organized in Turkey. Has a concern that after project ends they will have problems in importing trout eyed-eggs, specifically due to the situation on custom. Overall is very satisfied by project and Sheki BDS activities. Initially BDS contacted him to PSCEP and thinks this relationship was very sustainable. Observation conclusions: This farm has well developed business structure: has own feed mill and knows how to produce proper fish feed; is able to produce fingerlings from self produced and imported eggs; is located next to major road and easily can attract new buyers travelling between Sheki and Baku. 4. Apex-1 fish farm (primarily trout) During interview owner of the farm, Mr. Mirza Mammadov mentioned: With assistance from Ganja BDS developed business plan and was connected with PSCEP. Project assisted in purchase of trout eyed-eggs from the US supplier and received TA on how to hatch those eggs. Part of produced fingerlings is sold to other farmers; He attended conference organized by project and Sheki BDS and plans to join association under development. Needs investment to grow business but is not willing to apply for bank loan as they charge high interest rates. Overall he is satisfied by project and Ganja BDS activities. For Public Use [29] Observation conclusions: This farm is well located and has big potential, but needs substantial investment and TA to utilize its capacity. Overall the farm received reasonable assistance from PSCEP and Ganja BDS. 5. Gusar Agro-Line LLC (trout) During interview manager of the farm, Mr. Mahir Abbasoglu mentioned: Project assisted in purchase of trout eyed-eggs from the US supplier and provided TA on how to hatch those eggs. Received recommendations on how to clean water used for farm. Will continue partnering with Guba BDS to import fish eyed-eggs from the US and find appropriate fish feed supplier. Overall project and Guba BDS assistance was helpful. Observation conclusions: Of all trout farms visited, this was the best designed and implemented. Manager has good trout farming experience from working in a similar business in Turkey. Farm is new and up to now was not able to find proper supply of fish feed and eyed-eggs and is ready to pay for BDS services if they will assist is solving those and other similar problems. Fruits and Vegetables 6. Fruit processing company – Balacans During interview owner of the farm, Mr. Akhsunsky mentioned: Project conducted trainings in food safety, but without reaching a successful conclusion as no follow-up was provided from training, and no testing of personal was conducted. Company needs TA on quality management issues, on food safety, in developing retail shelf design for their products, and for developing fruit orchards. TA from previous USAID projects was more useful (implemented by Pragma). Overall project assistance was somewhat helpful. Observation conclusions: The company is well equipped, has diversified production and market base. Partner with Coca-Cola, supplies Ministry of Defense of Azerbaijan and several international markets. Recently company is in process of developing its own apple orchards and looks for diversifying international markets. Despite their level of development company needs specific TA primarily in food safety and is ready to pay for proper service. 7. Ganja NAA company – Cold Storage For Public Use [30] During interview owner of the company, Mr. Nadir Abbasov mentioned: International expert provided some TA on how to store products in cold storage facility. This was the only useful advice from project. From projects like PSCEP they need only knowledge and TA, how to process leftover products, how to operate cold storage facility. International study tours are needed to see how such business works in developed countries. Observation conclusions: The company is well equipped, total capacity seems higher then demand/supply exists at this stage. They have good linkages to Russian market and could play important role in linking local SME farmers to Russian and other international market. Such companies expect professional attitude and well thought and delivered TA. 8. Guba Un-AGRO LLC - Cold Storage During interview owner of the company, Mr. Zulfugar Mamamdov mentioned: They partner with Guba BDS for three years and most useful assistance from them was help during identification and purchase of fruit drying line. Thanks to this investment they can operate their fruit business more profitably. Major portion of fruit supply for their cold storage and drying facility comes from own orchards and remainder 20 % of capacity is rented out to benefit neighboring SME farmers. Instead of theoretical consultations would be better to see how similar businesses operate in other countries. Free consultation is good, but if needed is ready to pay for it. Observation conclusions: The company has diversified income sources. Understand local and Russian market needs and knows how to run business. Has own supply sources but also benefits about ten neighbor farmers. Company needs some TA and if properly delivered is ready to pay for it. 9. Shamkir individual farmer (greenhouse) During interview owner of the farm, Mr. Goja Gasimov mentioned: He partners with Ganja BDS for almost ten years. BDS systematically communicates with him, invites on different meetings and trainings; provides seeds for testing; links with different development programs. He exports his products (tomatoes and cucumbers) to different cities of Russia. Also produces and sells vegetable seedlings to neighbor farmers, helps them with advises and in reaching Russian market. Last year started livestock business for which Ganja BDS provided some needed TA and helped in receiving bank loan. Initially the business looked very attractive, but during implementation had big For Public Use [31] losses and sold the business. Unfortunately income from it was not big and has to pay debt from other sources. He never paid for TA and if will be asked to pay prefers not to receive it. He himself knows lot of things about greenhouse business and even teaches others. Observation conclusions: This farmer seems to know greenhouse business well. He could benefit from appropriate TA (new vegetable varieties, greenhouse technologies, etc), but is not willing to pay for it. It is also unclear why Ganja BDS doesn’t assist other greenhouse farmers in this area. 10. Jalilabad individual farmer (potato and strawberry) During interview owner of the farm mentioned: He partners with Jalilabad BDS and systematically communicates with them. From PSECP project was advised on how to seed and run strawberry farm. He followed those recommendations and expects doubled yield next year. Project offered to create underground cold storage facility for storing potatoes, despite interesting proposal is not ready to invest in it as has to take loan and is not willing to pay high interests. He runs his business without loan and is very happy by it. Despite the fact that Jalilabad BDS is very helpful, they lack capacity to provide practical advice. He would like to see how strawberry business works in developed countries and is ready to co￾finance such trip. He is willing to purchase jam production line and if BDS can help in finding is ready to pay for that service. Overall project assistance was very useful. Observation conclusions: This farmer seems to know his business. Understands how to find proper information and use in practice. He values good TA and is ready to pay for it. 11. Cenub Agro LLC (greenhouse) During interview owner of the farm, Mr. Mirnazim Bashirov mentioned: He decided to start this business in 2007 after visiting similar farm in Bulgaria; the trip was organized by preview USAID project. Vegetable traders buy his products (tomatoes and cucumbers) from farm and sell in Baku and Russia. Greenhouse business was more profitable before gas price increased. He also produces and sells vegetable seedlings to neighbor farmers. Wants to improve technology used in his greenhouse and is willing to receive appropriate TA. Best would be to see how those technologies are used in practice, so a trip to such farm would be very beneficial for him. They are ready and able to pay for such trip. For Public Use [32] Want to start citrus production in greenhouse and in open field, plans to import rootstocks from Italy and Holland. Observation conclusions: This farmer seems to know greenhouse business very well. He plans to introduce new technologies in farm operation and could benefit from appropriate TA (new vegetable varieties, greenhouse operation technologies, etc). He is ready to pay for proper help. Financial Institutions 12. Demir Bank (Commercial Bank). During interview Deputy Chairman of the Executive Board, Mr. Rauf Akhundov mentioned: Their bank appreciates TA provided by PSCEP project, assistance was provided on right time and covered priority topics for them like micro lending, trade finance, leasing. Those trainings and provided consultations improved quality of loans issues and increased banks credit portfolio. Bank needs further strengthening of its training capacity, as they systematically higher new credit officers and have to improve their customer relations, financial and business sector knowledge skills. They receive TA from different international organizations, but additional, specific help from USAID project could benefit the bank. Observation conclusions: Bank is willing to improve its credit portfolio, and is ready to accept appropriate TA. Overall professional assistance from USAID project will benefit the bank as well as targeted and strategic business sectors. Bank appreciates TA received and is willing additional assistance, same time they realize need for sustaining internal training capacity and plan to develop training of trainers program. 13. Rabita Bank (Commercial Bank). During interview manager of the bank, Mr. Elchin Gadimov mentioned: Their bank appreciates TA provided by PSCEP project, assistance was provided on right time and covered priority topics (micro lending, trade finance, leasing). They like training materials provided and plan to use for future trainings also. Trainings and provided consultations improved quality of banks credit portfolio, as credit officers have skills to better evaluate loan applications. They receive TA from different international organizations and can not think about any additional need. Observation conclusions: Bank appreciates TA received from PECEP and likes quality of training materials. At this stage they have no determined additional TA needs from USAID but raised interest in discussing DCA possibility. For Public Use [33] 14. Access Bank, Lankaran Branch (Commercial Bank). During interview deputy manager of the bank branch, Mr. Rahman Aliyev mentioned: They don’t recall if their branch directly received any TA from PSCEP project, but recognize that could have been relationship with bank headquarter. Several project consultants visited them and asked specific questions about practices used during agriculture lending. They partner with Lankaran BDC, specifically: present bank products to BDC’s customers; accept better quality business plans and loan applications developed by BDC; discuss specific business sector details when lack information. Bank has some internal training system for their credit officers and partners with different training providers. Observation conclusions: Bank recognizes benefits from their interactions with BDC. They have not raised any interest with future TA, but inviting their credit officers on agricultural sector trainings could increase their interest in funding agribusinesses. Business Development Organizations 15. Sheki BDS – Sheki Business Innovation Center. During interview director of the center, Mr. Bahram Osman mentioned: They partner with PSCEP since 2009. Project’s experts provided substantial TA to BDS and to their customers. Center received different manuals and trainings materials; and was assisted to better understand local business needs. As a results center has more customers then had before; has lot to do; and will continue active work after PSCEP ends. Together with project they organized aquaculture conference; facilitated participation of several fish farmers in Turkish aquaculture exhibition; imported trout eyed-eggs and provided to several fish farmers; and provided substantial TA to assisted farmers. Despite their satisfaction by project approach would be better to have more practical workshops and study tours. Observation conclusions: Sheki BDS seems is working properly, was able to improve capacity thanks to partnering with project; better understands their customer needs; has plans for future activities and was able to increase number of customers served and type of services provided. 16. Guba BDS – Guba Marketing Center. For Public Use [34] During interview director of BDS, Mr. Kheyraddin Jabarov mentioned: Their center was established in 2005 by previous USAID project assistance; They partner with PSCEP since 2009. Project’s experts provided TA to BDS and to their customers. Center received different manuals and trainings materials; was provided small grant and is in process of web site development. Despite their satisfaction by project approach thinks more international study tours would be beneficial for businesses. Center had some operation difficulties when PSECP decreased level of funding. Observation conclusions: Guba BDS seems is working on a satisfactory level, they provide different services to wide range of customers and has development plans, like: opening regional branch in neighbor area; creation of web site to promote center customers; purchase of publishing equipment to provide promotional and labeling service to local businesses. 17. Jalilabad BDS – Jalilabad Agribusiness Company. During interview director of BDS, Mr. Adalat Nehmatov mentioned: Their center was established in 2002 under previews USAID project; They partner with PSCEP since 2009. Project’s experts provided TA to BDS and to their customers. They received different manuals and trainings materials. International experts visited his BDS and delivered one day training for his customer farmers. He attended specific trainings organized in Baku targeted to strengthen BDS operations. Overall he is satisfied by PSECP project and TA delivered. Has no development plan and after project ends will have almost now income to cover center’s operation. Many of his customers are not ready to pay for type of services he provides. Observation conclusions: Jalilabad BDS provides more extensions type of services; has no sustainability plan; and could have operations difficulties without follow up assistance. 18. Lankaran BDS – Janub Agro-Business Center. During interview director of BDS, Mr. Mobil Penjaliyev mentioned: The center was established in 2004 under previews USAID project; They partner with PSCEP since 2009. Their center works with different donors and has income from commercial customers also. Only part of his costumers is ready to pay for consultations and trainings received. He expects that USAID and projects’ like PECEP should develop better linkage with big companies like Pal Sud to encourage them and create stronger PPP. For Public Use [35] Project’s experts provided TA to BDS and to their customers. They received different manuals and trainings materials. International experts visited his BDS, consulted farmers and delivered trainings. Not every international expert was knowledgeable about subject. He was offered to attend specific BDS training conducted by PSECP in Baku, but as this training was not useful for him decided not to attend. Project funding was much less then they deserved, they performed huge volume of work in dairy and fruit and vegetable field. Observation conclusions: Lankaran BDS understands his customer needs; is ready to improve and develop services; has diversified income sources. Overall center is somewhat satisfied by PSECP project. Center manager raised several quite important issues in projects approach and management practices. Other Businesses and Institutions 19. Golden Feed company (feed for livestock) During interview owner of the company, Mr. Alovsat Huseynov mentioned: During operation they mix different ingredients purchased locally and in Baku and produces feed for chicken and livestock. He is aware of USAID projects and partners with them from 2002. Projects before PSECP were much more useful. He likes Ganja BDS, but his relationship with them primarily ends with sharing information and bringing international visitors to his facility. PSECP staff visited him twice; they brought some flyers and never showed up after. He was willing to develop feed testing lab, but project used his idea and provided grant for lab to other organization. He is interested in producing fish feed, but needs TA and help. Observation conclusions: Overall he is not happy how project treated him. He claims that PSECP used his idea and provided grant for lab to another organization. He can independently (primarily from internet) find information he needs for his business, but showed interest for specific TA needed for fish feed production. 20. Mahmudavar village Milk Collection Point During interview manager of the center, Mr. Elchin Aliyev mentioned: Mahmudavar milk collection center daily purchases milk from almost 3,000 households. 2.5 years ago when they started business, milk supplied by farmers was of terrible quality and was rejected. Farmer needs 5-12 months to understand and improve produced milk quality. For Public Use [36] Lankaran BDS provided proper information and TA to farmers and quality of milk supplied increased faster then expected. They significantly benefited from BDS activities. Presence of their owner company - Pal Sud and network of milk collection centers motivates famers to increase their farm sizes; attract finance and investments; and improve livestock feeding and care practices. Despite the fact that their partnership with BDS was very successful. He is not authorized to say that are ready to pay for similar assistance. Observation conclusions: This milk collection centers play crucial role in working with livestock farmers. They are satisfied by improved volume of quality milk and recognize BDS’s proper role in it. Despite their successful partnership for future progress is recommended to work with headquarters of Pal Sud and other milk processing factories to encourage them to partner with BDS’s on commercial bases. GDA’s and PPP’s could be used to formalize those partnerships and linkages. 21. Davachi Broiler LLC (poultry). During interview General Director, Mr. Azimov Zakir mentioned: TA provided by PSCEP project was valuable, and they already implemented some of recommendations provided by project consultants. During last two years they purchased new equipment and substantially upgraded their production capacity and technology. Company heavily invests in its current production development and TA for determining and planning following actions and investment programs could be useful (for example, they are willing to create new feed mill, start production of fish feed, etc). They understand potential value of future TA, but at this stage are not ready to pay for it. Observation conclusions: Company understands value of improved production technology, has interest to continue development of its business and is willing to partner with future USAID projects. 22. Assena (Equipment Producer). During interview the owner of the company, Vahid Hummatov mentioned: Two years ago PSCEP project approached his company based on good feedback from his customers. They produce feed mills and other equipments for local and international agribusinesses, primarily from CIS countries. Last year project co financed organization of third conference on “Modern methods in feed production and feeding”, which his company organizes for last three years. This conference is excellent instrument for them to communicate with their current and future customers; and to introduce different technology innovations in feed industry. Project was not able to assist in organizing the conference this year. For Public Use [37] He requested project to help in finding US produced fish feed mill. Project was not able to assist. Overall he rates the project as an average. Observation conclusions: This company plays important role in developing local agribusiness sector. They produce good quality equipment with reasonable price. They can benefit from targeted and specific TA. 23. AIM – lab (feed lab) During interview manager of the center, Mr. Fazil Jafarov mentioned: PSCEP Project and AIM started partnership several months ago. From project AIM received grant for developing livestock feed lab. This lab will benefit their extension service consultants, their customers and other farmers. Recently they serve almost 30,000 farmers in different agricultural sectors, primarily from central part of Azerbaijan. The feed lab is not installed yet and AIM was not able to say what demand and income they expect from that business. During last two years they already operate soil testing lab donated by German development organization, and have proper experience in running lab business. AIM also benefited from PESEC TA in area of strawberry production and aquaculture. Observation conclusions: AIM seems to have positive image and important role in agriculture extension business. They provided wide specter of services to targeted farmers. The grant for feed testing lab will benefit livestock, chicken and aquaculture farmers and feed producers from different regions of Azerbaijan. But as project shortly ends, without continued TA, AIM could have difficulties during initial operation and marketing of feed lab services. 24. Executive Power (Mayor) of Sheki. During interview head of the executive power, Mr. Murad Jabbarli mentioned: His office is aware about PSCEP and other USAID projects active in Sheki rayon. The priority business sectors in Sheki are agriculture (fruits and vegetables, processing, milk and meat production, aquaculture), tourism and construction. Project correctly realized importance of aquaculture for the rayon. In additional to business growth, development of local infrastructure and ecology is a priority. They have development plans and appropriate funding for priority infrastructure projects. They also have several SME development and support programs, but businesses are not using those possibilities and USAID projects can help in promotion and utilization of those programs. Observation conclusions: For Public Use [38] Sheki Executive power had information about PESEP project and its local partner - Sheki BDS, he showed satisfaction by project achievements and raised interest to improve and increase future partnership with other USAID projects. 25. Softline (Microsoft Anti-Piracy GDA). During interview the Managing Director, Alim Salahov mentioned: PSCEP initiated relationship with Softline and after several meetings was decided to develop GDA in the area of Anti Piracy. Under this GDA company trained financial auditors of National Bank of Azerbaijan on how to detect existence of illegal software in local financial institutions. After those trainings and consultations overall share of licensed software in banking sector substantially increased. Company provided trainings of local ICT companies and encouraged them to promote licensed software to their customers. Overall as a result of this partnership with PSECP, company managed to substantially increase its efforts against anti piracy. Observation conclusions: It is obvious that this partnership is very successful and beneficial for development of ICT industry in Azerbaijan. 26. Barama Innovation Center. During interview the Manager of the Center, Gahraman Kazimov mentioned: Barama Innovation Center was created by mobile company – Azercell, as its Social Responsibility action. Center exists for last several years and attracts different ICT professionals with commercially viable business ideas. Those professionals benefit from office space and services offered by center and are assisted in developing and commercialization of their ideas. From September, 2010 center implements project which is co-financed by PSCEP and includes development of Cisco innovation center, purchase of needed equipment and small grants to successful entrepreneurs attracted by center. Observation conclusions: Partnership with Azercell and Barama Innovation Center is a good model for assisting development of startup ICT entrepreneurs. The concern with this activity is that the grant was provided to center few months ago and is unclear how it will be spent before end of the PSECP project, also center was not provided any TA and center’s sustainability as well as long term funding is unclear. For Public Use [39]