EVALUATION Mid-Term Performance Evaluation of the Restoring Efficiency to Agriculture Production Project (REAP) in Georgia [November 2015] This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Mendez England & Associates. MID-TERM PERFORMANCE EVALUATION OF THE RESTORING EFFICIENCY TO AGRICULTURE PRODUCTION PROJECT (REAP) Final Report November 2015 Prepared under RFTOP No. SOL-114-15-000005 Submitted to: USAID/Georgia Submitted by: Colin Maclean, Team Leader Dr. Gary Woller, International Evaluation Expert Dr. Gaga Nikabadze, Local Expert Contractor: Mendez England & Associates 4300 Montgomery Avenue, Suite 103 Bethesda, MD 20814 Tel: 301- 652 -4334 www.mendezengland.com DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government CONTENTS EXECUTIVE SUMMARY .................................................................................................. 2 Evaluation Background ........................................................................................................................................ 2 Project Purpose .................................................................................................................................................... 2 Evaluation Design, Methods and Limitations .................................................................................................. 2 Findings, Conclusions and Recommendations ............................................................................................... 3 1.0 EVALUATION BACKGROUND, PURPOSE AND QUESTIONS ..................... 7 1.1 Evaluation Background ............................................................................................................... 7 1.2 Evaluation Purpose ..................................................................................................................................... 7 1.3 Evaluation Questions ................................................................................................................................. 7 2.0 PROJECT BACKGROUND .................................................................................... 8 3.0 EVALUATION METHODS AND LIMITATIONS ............................................... 9 3.1 Evaluation Methodology ............................................................................................................. 9 3.1.1 Quantitative Research and Analysis ........................................................................................ 9 3.1.2 Qualitative Research and Analysis ........................................................................................... 9 3.2 Evaluation Limitations ............................................................................................................................. 10 4.0 FINDINGS, CONCLUSIONS AND RECOMMENDATIONS ........................... 11 4.1 Are the REAP’s approaches necessary and sufficient for developing competitive and commercially sustainable agribusinesses, and increasing incomes and employment in rural areas in Georgia? ...................................................................................................................................... 11 4.1.1 What are the major constraints / challenges / outstanding needs that inhibit the project’s contribution toward achieving the results during the remaining term of the project? ................................................................................................................................. 11 4.1.2 Is technical and/or business training provided to assisted primary producers and SMEs sufficient to improve skills, yields and incomes of farmers? ................................. 12 4.1.3 Is the provided and intended TA specific and tailor made to each of the grantees needs? Are the quality of topics and instructors of the training appropriate for Georgian agriculture as defined by REAP beneficiaries? .................................................. 14 4.1.4 Are the grantees that are selected a good and fair representation to sustainably increase the availability and utilization of agricultural inputs, services and technical information among small-holder and emerging commercial farmers? Were the requirements selected in a fair and transparent manner as described by initials project design documentation? Was the grant selection process fair and transparent? ................................................................................................................................ 16 4.1.5 Has the program been successful in integrating the cross-cutting issues (gender, environment, workforce development and access to finance) into the program implementation, when appropriate? ..................................................................................... 19 4.2 How is the program perceived by beneficiaries? ............................................................................. 21 4.3 What, if any, corrections are needed to REAP’s implementation approaches during the remaining term of the project? ............................................................................................................. 23 ANNEXES Annex A: Statement of Work Annex B: List of Interviewees Annex C: List of Documents Reviewed Annex D: Mini-Survey Questionnaires Annex E: Results Chart from Evaluation Team’s Investigations TABLES Table 1: Direct Beneficiaries – Interview Categories FIGURES Figure 1: Influence of REAP Technical Trainings Figure 2: Question from Mini-Survey on TA Impact Figure 3: Expectations of Beneficiaries Regarding Future REAP Trainings Figure 4: Question from Mini-Survey on TA Relevance i ACRONYMS AMP APMA Access to Mechanization Project Agriculture Projects Management Agency BSP Business Service Provider CFF Commercial Finance Fund CNFA Cultivating New Frontiers in Agriculture COP Chief of Party COR Contracting Officer’s Representative FGD Focus Group Discussion FSC Farm Service Centre GoG HACCP Government of Georgia Hazard Analysis Critical Control Point KIIs Key Informant Interviews LOP Life of Project MSC Mechanization Service Centre M&E Monitoring and Evaluation ME&A Mendez England and Associates MFI Micro-Finance Institution MOU Memorandum of Understanding MSME Micro, Small and Medium Enterprises NGO PERSUAP Non-Governmental Organization Pesticide Evaluation Report and Safer Use Action PHP Post-Harvest Producers PMP Performance Monitoring Plan PP Primary Producer REAP Restoring Efficiency to Agriculture Production RFA Request for Application SME Small and Medium Enterprise SOW Statement of Work TA Technical Assistance TOR Terms of Reference USAID United States Agency for International Development USG United States Government WB World Bank 2 EXECUTIVE SUMMARY EVALUATION BACKGROUND This is a report on the Mid-Term Performance Evaluation of the Restoring Efficiency to Agricultural Production (REAP) Project funded by the United States Agency for International Development (USAID) Mission in Georgia. REAP is being implemented by Cultivating New Frontiers in Agriculture (CNFA) from September 2013 through September 2018. The evaluation was conducted during September - October 2015 by a team assembled by Mendez England & Associates (ME&A), which included one local and two international experts. IRMS, a local organization, was contracted to conduct a survey of grantee and non￾grantee enterprises as well as technical assistance (TA) recipients. The findings, conclusions and recommendations in this report emanate from the collective efforts of the above-mentioned team. PROJECT PURPOSE According to the Statement of Work (SOW) for this assignment (see Annex A) the main purpose of the evaluation was “to determine the relevance of REAP’s programmatic approaches in achieving intended life-of￾project results and to provide recommendations and corrective actions and new directions for the remaining years of the project implementation and beyond.” In specific terms, the evaluation would “examine the relevance of REAP’s approaches in achieving the intended results of developing competitive and commercially sustainable agribusinesses, and increasing incomes and employment in rural areas.” EVALUATION DESIGN, METHODS AND LIMITATIONS Quantitative Research and Analysis The Evaluation Team reviewed a wide range of project-related materials from various sources, including those received from USAID/Georgia at the pre-mobilization stage, and from the REAP project throughout the course of the evaluation mission. These documents included, among others, the original contract between USAID Caucasus and CNFA1 and three subsequent contract modifications, Annual Reports2, Annual Work Plans, Performance Monitoring Plans (PMPs), and Quarterly Reports. These reports constituted the main reference point for all relevant statistical data, including financial budgets, from September 2013 through September 2015. From this material, the Evaluation Team was able to track REAP’s performance over time with respect to the various pre-determined indicator targets established at the start of the project. The verification of statistical results actually achieved by REAP at the time of the evaluation, however, did not fall within the Evaluation Team’s remit3. The nature and extent of REAP’s activities in terms of reaching and providing assistance and support to its targeted beneficiaries was reviewed by cross-referencing quantitative achievements with evidence gathered from its qualitative investigations. The Evaluation Team worked with the local firm, IRMS, to conduct a mini-survey of grantees and non￾grantees4 in order to consolidate findings through cross-correlation of results from the team’s face-to-face interviews and focus group discussions (FGDs). A total of 113 respondents (81 males and 32 females) were interviewed in eight regions (Adjara, Guria, Imereti, Kakheti, Kvemo-Kartli, Samegrelo-Zemo Svaneti, Samtskhe-Javakheti, Shida-Kartli) and in Tbilisi, through the use of a structured questionnaire (see Annex D for full list of questions). Data gathered was disaggregated by gender, age of respondents, and beneficiary category - primary producers (PPs), post-harvest handling enterprises (PHPs), and Farm Service Centers / Mechanization Service Centers (FSCs/MSCs). Accordingly, three questionnaires5 were developed and piloted for each category of business prior to the main field work. Questionnaires included modules on business characteristics, technical assistance, other grant / donor-funded programs, and the grant application process. A total of 19 interviewers were trained to conduct the survey (face-to-face interviews) and assigned to specific regions / districts (based on the number of applicants per region / district). In some instances, following requests from out-of-country respondents, an e-version of the questionnaire was sent for completion and returned for analysis. 1 Signed 30 September 2013 2 Only year 1 Annual Report was available for review at the time of the evaluation 3 The Evaluation Team has assumed that results reported by REAP and approved by USAID have been verified by both parties 4 Beneficiaries who only requested technical assistance not a grant 5 See Annex E for full list 3 Qualitative Research and Analysis The Evaluation Team elicited feedback and commentary from as representative a body as possible with direct or indirect knowledge and/or experience of REAP throughout the period in question (see Annex B for full list of interviewees). In short, the Evaluation Team met with 57% of all REAP’s grantees, 25% of whom were women, which provided sufficient information and feedback to inform the team’s findings. In addition, the team interviewed four non-grantee TA recipients from whom additional comment was elicited, as well as a cross-section of key stakeholders from both the public and private sector. Evaluation Limitations Although the Evaluation Team initially anticipated the possibility of recall or halo6 bias from potential interviewees, this proved not to be an issue with either one-on-one interviews with direct beneficiaries (grantees and TA recipients) and stakeholders or failed and/or rejected REAP grant applicants who attended any of the regional FGDs. FINDINGS, CONCLUSIONS AND RECOMMENDATIONS Question 1: Are REAP’s approaches necessary and sufficient for developing competitive and commercially sustainable agribusinesses and increasing incomes and employment in rural areas in Georgia? See sub-questions 1.1to 1.5 below for specific responses. Question 1.1: What are the major constraints/challenges/outstanding needs that inhibit the project’s contribution toward achieving the results during the remaining term of the project? Findings There is clear evidence that, to date, the project has met (and continues to meet) the expectations of direct beneficiaries (grantees) with respect to facilitating business expansion and job creation opportunities. Feedback from the 40 interviewees indicated that as a result of REAP’s interventions a total of 712 jobs had been created: 94 full time, 301 temporary, and 207 seasonal (see Annex E for detailed analysis of the Evaluation Team’s findings). The only limitations noted by the team referred to TA training modules, which 72% of respondents suggested could be more practical focused. Conclusions From the available evidence gathered by the Evaluation Team it is possible to conclude that REAP does not appear to be experiencing any serious impediments to the delivery of its two main components, matching￾grant disbursements and TA, with the exception of some suggested re-adjustment to the format and structure of the latter. Recommendations There are no specific recommendations for Component 1 as the grant allocation has already been fully committed. For Component 2 only a review of the content and duration of TA training sessions is advised in line with the above findings and conclusions. Question 1.2: Is technical and/or business training provided to assisted primary producers and small medium enterprises (SMEs) sufficient to improve skills, yields, and incomes of farmers? Findings Feedback from interviewees indicated that REAP’s training programs (business and technical) were well designed and, in most cases, in line with the expectations of the applicants. The structure and content of the technical training was deemed sufficient to improve skills, help increase productivity, and contribute towards increasing incomes. However, 75% of interviewees stated a clear preference for more practical TA modules. Conclusions It can be concluded that REAP has met its obligations in terms of contributing towards improved business performance but that its existing training modules may need to be re-visited in order to determine the optimum format needed to satisfy clients, especially with regard to agricultural technical issues. Recommendations REAP should consider re-designing its training modules to focus more on ‘learning-by-doing’ approach - cycle process training mainly through international experts then later by local consultants. A major advantage in adopting this approach is that it will serve to ensure that client expectations are adequately 6 Halo bias is the extent to which respondents are prepared to reveal their true views i.e. perhaps not wanting to be critical of benefactors or people they may depend upon for the provision of services 4 met in line with their specific needs rather than through a ‘broad brush’ approach. The Evaluation Team acknowledges that the project has already begun the process of engaging international consultants to facilitate knowledge transfer and innovative approaches to its beneficiaries, and recommends that this should continue in line with client needs and requirements. Question 1.3: Is the provided and intended TA specific and tailor-made to each of the grantees’ needs? Are the quality of topics and instructors of the training appropriate for Georgian agriculture as defined by REAP beneficiaries? Findings Data gathered from grantee interviews indicated that in 80% of cases, the TA was fully in line with their expectations and relevant within the Georgian context. The high level of competency of the instructors was also greatly appreciated by most of the interviewees. Their only reservation was the weak linkage that existed between REAP’s training initiatives and local knowledge institutions (vocational colleges, universities, extension centres), which could impact their ability to access TA expertise post-project. Conclusions REAP appears to be adequately responding to the specific training needs of the grantees, in particular the FSCs/MSCs, in line with their expectations (demand driven). Recommendations REAP should continue with its current TA initiatives and working closely with beneficiaries to develop specific and practical technical training sessions. REAP could also consider fostering linkages or synergies with local academic or vocational institutions for knowledge distribution and access to REAP materials for interested parties. Question 1.4: Are the grantees that are selected a good and fair representation to sustainably increase the availability and utilization of agricultural inputs, services and technical information among small-holder and emerging commercial farmers? Were the requirements for grantees selected in a fair and transparent manner as described by initial project design documentation? Was the grant selection process fair and transparent? Findings Grantees who participated in the interviews exhibited characteristics commonly associated with entrepreneurship and ambition to operate a successful business in the long-term. Feedback from respondents suggests that the application process appears to have been successful in screening applicants and in selecting ‘winners’. Unsuccessful applicants also appeared to support this view even when rejected or failed although several advised that they were never informed about the outcome of their applications. Conclusions REAP’s grant application and screening processes worked effectively to further program objectives of supporting agricultural entrepreneurs with potential to help catalyse growth and development in Georgia’s agricultural sector. However, a question remains as to whether some of the unsuccessful applicants did in fact receive notification of their applications. Recommendations No specific recommendation as the matching grant component has already been implemented effectively, with funds disbursed or committed. However, perhaps a review of the project’s communication procedures may be worthwhile to verify that all unsuccessful applicants actually received notification of the result of their applications. Question 1.5: Has the program been successful in integrating the cross-cutting issues (gender, environment, workforce development, and access to finance) into the program implementation, when appropriate? Findings From the data and information gathered during the evaluation mission, both reported by REAP and from interviewee comments, it is apparent that cross-cutting issues are well integrated in REAP’s implementation plans. Various project initiatives have been designed to reach out and support women grantees, advising commercial banks on lending strategies for agriculture, and developing a workforce internship program for students. With regard to environmental matters, in the second year of the project, REAP conducted a Pesticide Evaluation Report and Safer Use Action (PERSUAP) for USAID/Georgia Mission and monitored environmental aspects of all operational grantees. Conclusions There is clear evidence that REAP has acknowledged the importance of cross-cutting issues as an integral part of its mandate. This is manifested in a number of on-going initiatives and reported accordingly in various quarterly reports. 5 Recommendations No specific recommendations for the duration of the project. However, some suggestions are made in Section 4.3.3 in the main body of the report regarding possible future options for a USAID-supported program not necessarily based on matching grants, which could help address the access to finance issues that continue to be a constraint to growth in Georgia, and especially for women. Question 2: How is the program perceived by beneficiaries? Findings Almost without exception REAP’s direct beneficiaries (grantees and TA recipients) as well key stakeholders from the public and private sectors, acknowledged the contribution that the project’s interventions had made (and are continuing to make) in advancing the cause of rural agricultural communities in Georgia. In addition to this, interviewees nearly universally complimented the quality of the assistance they received from REAP staff, local Business Service Providers (BSPs), and international experts. The only criticism that the grantees had for the project was that the 70% cost-share for them was too high. Nevertheless, in the absence of alternative sources of finance, it was still deemed the best option open to them if they were to develop their businesses as planned. More practical TA sessions and advice on food safety standards was also mentioned as needing to be addressed in any future training program. Conclusions There is no doubt that the REAP program is overwhelmingly perceived in positive terms by grantees, TA recipients, and implementing partners. According to them, the benefits acquired by REAP’s clients through participation in the program are tangible and are making a valuable contribution to improving business performance. However, comments was also made that perhaps REAP could focus more effort on designing more practical training workshops and demonstrations together with advice on food safety issues. Recommendations The primary recommendation in response to this question is one already offered above: to shift the TA increasingly towards a ‘learning-by-doing’ approach - cycle process training in such a way that it guides grantees and TA recipients through the process of implementing the new technologies and business practices in which they have been instructed by the program. In addition, while REAP has conducted a number of trainings on food safety issues for grantees and non-grantees in Year 2, this should be actively pursued in line with its current 50:50 cost-share initiatives with new grant recipients on, for example, Global GAP, Hazard Analysis Critical Control Point (HACCP), and ISO22000. Question 3: What, if any, corrections are needed to REAP’s implementation approaches during the remaining term of the project? Findings It has already been noted in responses to the above questions that only few corrections are needed to REAP’s implementation activities or delivering on its mandate and these are largely associated with TA training modules. Indicator targets have been met or are in the process of being achieved. Beneficiaries have largely expressed their satisfaction with the project’s support subject to the issues raised above with respect to TA training modules, 70% cost-share contribution, food safety issues, and access to finance, particularly for women. No further comment is therefore needed within this finding in terms of anticipated corrections to the project between now and September 2018, when REAP ends. Conclusions Based on the findings to the above-mentioned questions, the Evaluation Team can conclude that with the project’s grant funds already committed, REAP has already satisfied its mandate in this respect. No corrections to REAP’s implementation modalities are therefore envisaged here other than the need for effective planning of the expenditure of the $2.5m available for TA support to REAP’s beneficiaries, in line with client expectations and demands. Recommendations There are no recommendations for the remainder of the program, as per the above findings and conclusions. However, an option for consideration is the following scenario: Instead of grants for targeted beneficiaries - SMEs, entrepreneurs including women and young people, primary producers, processors, and/or minority groups, all of whom may be considered as riskier clients for traditional finanical institutions - consider the creation of a Commerical Finance Fund (CFF) funded by USAID in collaboration with a private sector investor, which would make funds available to targeted clients at competitive rates. Such initiative can be managed by a private contractor and run effectively as a revolving fund. Although further 6 research is needed to determine how this could be accomplished within the Georgian context, the potential for reaching a wider client base in agriculture/agribusinesses, with a view to improving job creation opportunities and income generating initiatives for rural farming communities, remains a prospect worth examining in more detail. 7 1.0EVALUATION BACKGROUND, PURPOSE AND QUESTIONS 1.1 EVALUATION BACKGROUND This is a report on the Mid-Term Performance Evaluation of the Restoring Efficiency to Agricultural Production (REAP) Project funded by the United States Agency for International Development (USAID) Mission in Georgia. REAP is being implemented by Cultivating New Frontiers in Agriculture (CNFA) from September 2013 through September 2018. The project funding for REAP was $19,589,463. The evaluation of REAP was conducted during the period September and October 2015 by a team assembled by Mendez England & Associates (ME&A) located in Bethesda, Maryland. The Evaluation Team comprised three key experts: Mr. Colin Maclean (Team Leader), Dr. Gary Woller (Evaluation Expert), and Dr. Gaga Nikabadze (Local Expert). In addition, the Evaluation Team was supported by IRMS, a local organization contracted to conduct a survey of grantee and non-grantee enterprises as well as technical assistance (TA) recipients. The findings, conclusions and recommendations in this report emanate from the collective efforts of the above-mentioned team. 1.2 EVALUATION PURPOSE According to the Statement of Work (SOW) for this assignment (see Annex A) the main purpose of the evaluation was “to determine the relevance of REAP’s programmatic approaches in achieving intended life-of￾project results and to provide recommendations and corrective actions and new directions for the remaining years of the project implementation and beyond.” In specific terms, the evaluation would “examine the relevance of REAP’s approaches in achieving the intended results of developing competitive and commercially sustainable agribusinesses, and increasing incomes and employment in rural areas.” 1.3 EVALUATION QUESTIONS In line with the Scope of Work (SOW) requirements, the Evaluation Team was tasked with answering a specific set of questions. These are highlighted below: 1. Are REAP’s approaches necessary and sufficient for developing competitive and commercially sustainable agribusinesses and increasing incomes and employment in rural areas in Georgia? 1.1 What are the major constraints/challenges/outstanding needs that inhibit the project’s contribution toward achieving the results during the remaining term of the project? 1.2 Is technical and/or business training provided to assisted primary producers and SMEs sufficient to improve skills, yields and incomes of farmers? 1.3 Is the provided and intended TA specific and tailor made to each of the grantees’ needs? Are the quality of topics and instructors of the training appropriate for Georgian agriculture as defined by REAP beneficiaries? 1.4 Are the grantees that are selected a good and fair representation to sustainably increase the availability and utilization of agricultural inputs, services and technical information among small-holder and emerging commercial farmers? Were the requirements selected in a fair and transparent manner as described by initial project design documentation? Was the grant selection process fair and transparent? 1.5 Has the program been successful in integrating the cross-cutting issues (gender, environment, workforce development, and access to finance) into the program implementation, when appropriate? 2. How is the program perceived by beneficiaries? 3. What, if any, corrections are needed to REAP’s implementation approaches during the remaining term of the project? The Evaluation Team’s response to these questions is elaborated in the main body of the report in the following sections and is supported, where appropriate, by tables, figures and charts. 8 2.0PROJECT BACKGROUND In designing the original SOW for the REAP project, USAID considered the ongoing constraints faced by rural farming communities in Georgia in developing profitable and sustainable levels of production capable of competing in domestic as well as global markets. USAID’s research identified a serious lack of competitively priced production inputs (seeds, breeding stock, pesticides, veterinary medicines, machinery and equipment) as well as professional services (testing laboratories, veterinarians, commercial pesticide applicators, private technical consultants) as the main inhibiting factors adversely affecting agricultural growth and development in Georgia. To compound matters, before 2012, the Government of Georgia (GoG) did not view agriculture as a priority in economic terms, allocating only 0.44% of its entire budget to the agriculture sector in 2010. Instead, GoG’s focus was socially-orientated, providing hand-outs of flour, food and fuel. In real terms, investment in agriculture in Georgia from both the public and private sectors in recent years has been limited. Although USAID’s Access to Mechanization Project7 (AMP) addressed some of these concerns, most notably contributing towards increasing productivity and incomes of smallholder farmers through improving access to machinery services8, significant constraints remained that clearly needed to be confronted. Some of them included: high costs of machinery services (often prohibitive to small farmers); intermittent supply of high quality seeds and saplings; limited knowledge and skills of modern, sustainable agricultural practices; and insufficient supply and utilization of blended fertilizers and other agro-inputs. In addition to these deficiencies, USAID acknowledged other equally pressing issues that needed to be addressed, including dealing with the country’s prevailing lack of post-harvest handling infrastructure (storage, packing, grading, sorting) facilities; inability of producers and processors to meet domestic and international food safety standards; lack of a functioning market information system; and limited number of wholesale market consolidation centers. The impact of the above constraints and deficiencies in agricultural production is that farmers in Georgia remain unable to effectively compete in domestic or international markets. They are either faced with cheap foreign imports entering the domestic market, or are unable to supply overseas markets (where trade agreements allow) due to their inability of meeting the required quality or food safety standards9. The REAP project was developed against this background. Its primary objective is to increase employment and incomes in rural areas, building on AMP’s earlier contribution and achievements, while acknowledging that more needed to be done if progress was to be made in the agricultural sector in Georgia. The focus of REAP is the provision of assistance to small agribusinesses in the form of matching grants, as well as demand-driven TA to those enterprises that meet the required qualifications. The rationale for this approach was that jobs would be created, the availability of high quality inputs and machinery services would be improved, and markets for agricultural goods and services would be strengthened in the country. In addition, REAP was required to identify and address cross-cutting issues such as gender inequalities and relationships between REAP beneficiaries and financial institutions. Furthermore, REAP needed to include initiatives to support workforce development largely through the creation of formal linkages between the private sector and academic/workforce development institutions, and also to support environmental best practices in production, processing, and marketing in order to minimize pollution, ecosystem loss, and resource depletion or degradation. To monitor the project’s progress and contribution towards achieving these aims, REAP’s SOW included a number of quantifiable target indicators, which would be reported at regular intervals for review and scrutiny by USAID. The Evaluation Team acknowledges this historical background and rationale for REAP, including its expected results. The above has served to inform the design and structure of the content of this report. 7 Implemented between September 2009 and December 2012 8 Now through Mechanization Service Centers / Farm Service Centers (FSCs) 9 Although the European Union (EU) and Georgia have a Deep and Comprehensive Free Trade Agreement (DCFTA) this does not in itself guarantee Georgian businesses being able to comply with the required EU standards and thus be able to supply EU markets. 9 3.0EVALUATION METHODS AND LIMITATIONS 3.1 EVALUATION METHODOLOGY The evaluation methodology was carefully designed to respond to the three main evaluation questions, and to determine whether REAP has made progress towards its initial objectives and is on track to deliver the expected results. The Evaluation Team collected quantitative and qualitative data from a broad range of stakeholders and beneficiaries to ensure independence of the evaluation process, as well as accuracy and completeness of the subsequent conclusions and recommendations. 3.1.1 Quantitative Research and Analysis The Evaluation Team reviewed a wide range of project-related materials from various sources, including those received from USAID/Georgia and REAP throughout the course of the evaluation mission. These documents included, among others, the original Request for Application (RFA) including SOW for REAP10, original contract between USAID Caucasus and CNFA11 and three subsequent contract modifications, Annual Reports12, Annual Work Plans, Performance Monitoring Plans (PMPs), and Quarterly Reports. These reports constituted the main reference point for all relevant statistical data, including financial budgets, from September 2013 through September 2015. From this material, the Evaluation Team was able to track REAP’s performance over time with respect to the various pre-determined indicator targets established at the start of the project. The verification of statistical results actually achieved by REAP at the time of the evaluation, however, did not fall within the Evaluation Team’s remit13. The nature and extent of REAP’s activities in terms of reaching and providing assistance and support to its targeted beneficiaries was reviewed by cross-referencing quantitative achievements with evidence gathered from its qualitative investigations. 3.1.1.1 Mini-Survey The Evaluation Team worked closely with IRMS, a Georgian company, to conduct a mini-survey of grantees and non-grantees14. Results of the survey were cross-checked with results from the team’s face-to-face interviews and FGDs in order to consolidate findings. Overall, a total of 113 respondents (81 males and 32 females) were interviewed in eight regions (Adjara, Guria, Imereti, Kakheti, Kvemo-Kartli, Samegrelo-Zemo Svaneti, Samtskhe-Javakheti, Shida-Kartli) and in Tbilisi. Data gathered by the mini-survey was disaggregated by gender, age of respondents, and beneficiary category - PPs, PHPs and FSCs/MSCs. Accordingly, three questionnaires15 (see Annex E for full list of questions) were developed for each category of business and piloted for testing purposes prior to the main field work. Questionnaires included modules on business characteristics, TA, other grant /donor-funded programs, and the grant application process. A total of 19 interviewers were trained to conduct the survey (face-to-face interviews) and assigned to specific regions/districts (based on the number of applicants per region/district). In some instances, following requests from out-of-country respondents, an e-version of the questionnaire was sent for completion and returned for analysis by IRMS. Following completion of the mini-survey, IRMS prepared a number of tables and charts, which served as a source of important information for the Evaluation Team in its understanding of beneficiary attitudes and perceptions with regard to REAP and its implementation activities. Where deemed relevant, extracts from the mini-survey are presented in tabular or graphic format within the context of specific questions, described in detail in the Section 4.0 below. 3.1.2 Qualitative Research and Analysis The Evaluation Team elicited feedback and commentary from as representative a body as possible with direct or indirect knowledge and/or experience of REAP throughout the period in question (see Annex B 10 AID-114-C-13-00002 11 Signed 30 September 2013 12 Only year 1 Annual Report was available for review at the time of the evaluation 13 The Evaluation Team has assumed that results reported by REAP and approved by USAID have been verified by both parties 14 Beneficiaries who only requested technical assistance not a grant 15 See Annex E for full list 10 for full list of interviewees). Below is a brief summary of the various categories of direct beneficiary interviewees with whom the Evaluation Team met both in Tbilisi and on field trips in the regions: Table 1: Direct Beneficiaries - Interview Categories Grantees Total Project Target Total Met Total Women Businesses Primary Producers 6 3 2 Post-Harvest Handling Enterprises 34 20 12 Information Service Providers 1 1 0 MSCs/FSCs 29 16 3 Total 70 40 17 Source: Data gathered and collated by the Evaluation Team during the in-country mission In short, the Evaluation Team met with 57% of all REAP’s grantees, 14% of whom were women, which provided sufficient information and feedback to inform the Team’s findings. In addition, the Team interviewed four non-grantee TA recipients from whom additional comments were elicited, as well as a cross-section of key stakeholders from both the public and private sector. Focus Group Discussions (FGDs) In order to ascertain the reasons why individuals/businesses did not progress beyond the application stage, the Evaluation Team held a total of 12 FGDs in the regions with failed and rejected16 REAP grant candidates. The focus group sessions were attended by a total of 154 participants – 113 failed and 41 rejected applicants. To date, REAP has received 231 grant applications from its various RFAs, which means that two thirds of applicants were in fact unsuccessful. Separate FGDs were held for each of the two categories of applicants. These FGDs provided significant amount of information on attitudes towards REAP together with views on the implications of failing or being rejected outright. The findings from these FGDs were cross-correlated with the results of the mini-survey, in both quantitative and qualitative terms, which ultimately corroborated the Evaluation Team’s own findings with respect to each of the evaluation questions. 3.2 EVALUATION LIMITATIONS Although at the start of the evaluation the Team anticipated the possibility of recall or halo17 bias from potential interviewees, this proved not to be an issue with either one-on-one interviews with direct beneficiaries (grantees and non-grantees) and stakeholders or failed and/or rejected REAP grant applicants who attended any of the regional FGDs. All such sessions were conducted respecting full confidentiality and anonymity of responses, which ultimately provided the Evaluation Team with sufficient information and data to inform its findings, conclusions and, in some cases, recommendations, for the remainder of the project. In short, by adopting an open and transparent approach to fact-finding and data gathering, the quality of feedback from respondents was an important feature of the entire evaluation mission. 16 Rejected applicants are those who did not satisfy the requisite number of points needed to proceed. Failed applicants are those whose applications needed further revision or upgrading before final approval. 17 Halo bias is the extent to which respondents are prepared to reveal their true views i.e. perhaps not wanting to be critical of benefactors or people they may depend upon for the provision of services 11 4.0 FINDINGS, CONCLUSIONS & RECOMMENDATIONS 4.1 ARE THE REAP’S APPROACHES NECESSARY AND SUFFICIENT FOR DEVELOPING COMPETITIVE AND COMMERCIALLY SUSTAINABLE AGRIBUSINESSES, AND INCREASING INCOMES AND EMPLOYMENT IN RURAL AREAS IN GEORGIA? 4.1.1 What are the major constraints/challenges/outstanding needs that inhibit the project’s contribution toward achieving the results during the remaining term of the project? SUMMARY OF EVALUATION DESIGN AND METHODS Evaluation Question Type of Analysis Conducted Data Sources Data Collection Method Limitations What are the major constraints challenges/ outstanding needs that inhibit the project’s contribution toward achieving the results during the remaining term of the project? Description – based on content analysis of expert opinions & review of relevant documentation  Project reports (annual, quarterly), PMP, work plans  Project staff, REAP grantees, TA recipients, unsuccessful grant applicants, key stakeholders  Submitted by REAP and reviewed by ET18  Feedback from KIIs19  Feedback from FGDs in regions None encountered FINDINGS Following an in-depth review of REAP’s reported results, especially those recorded in its Quarterly Reports (only Fiscal Year (FY) 1 Annual Report was available for review at the time of the evaluation), and from feedback gathered from the various data sources highlighted in Section 4.1.1 above, there is clear evidence that the project has met (and continues to meet) the expectations of direct beneficiaries (grantees) with respect to facilitating business expansion and job creation opportunities. The Evaluation Team interviewed 40 of the 70 grantees of the project, 80% of whom were satisfied with REAP’s support on their behalf with respect to matching grants. Half of those interviewed were post-harvest handling enterprises while 16 were FSCs/MSCs. The remainder were primary producers (6) and one information service provider. In terms of coverage of the total population of 70 grantees, as mentioned above, the Evaluation Team met with 57% of them, representing a significant sample size by any statistical standards. With respect to TA recipients, the Evaluation Team met with 4 of REAP’s 10 clients. This allowed them to gather sufficient data to inform their findings and subsequent conclusions and recommendations. As a result of these investigations, the Team found no major constraints to REAP’s implementation strategy in either component, other than some limitations regarding the delivery of TA to beneficiaries in terms of content and duration of training sessions. This issue is addressed in more detail in Section 4.1.2 below. CONCLUSIONS From the evidence made available to the Evaluation Team, and from data gathered from its in-country investigations, it can be concluded that the project is on the right track in meeting its contractual obligations without fear of serious hindrance or constraints. This is particularly true of Component 1 where a total of $6.0m has now been committed to grantees, namely, either disbursed already or nearing contractual finalization. With regard to Component 2, the only issue that appears to have a degree of significance concerns the delivery of TA training modules. However, in principle, beneficiary level of satisfaction remains high as to what has been delivered to date, with the provision that any future training modules/workshops be more focused on practical demonstrations on how implementation of what has been learned can be more readily replicated in the field. In short, the Evaluation Team encountered no ‘red flags’ that might have signalled urgency in dealing with outstanding issues during the remaining three years of the project. 18 ET – Evaluation Team 19 KIIs – Key Informant Interviews 12 RECOMMENDATIONS No recommendation is envisaged for Component 1 regarding this specific question. A re-alignment of training content and delivery mechanisms under Component 2 is recommended (see 4.1.2 below for more detail). 4.1.2 Is technical and/or business training provided to assisted primary producers and SMEs sufficient to improve skills, yields and incomes of farmers? SUMMARY OF EVALUATION DESIGN AND METHODS Evaluation Question Type of Analysis Conducted Data Sources Data Collection Method Limitations Is technical and/or business training provided to assist primary producers and SMEs sufficient to improve skills, yields and incomes? Description – based on content analysis of expert opinions & review of relevant documentation  Project documents: weekly & quarterly reports, M&E plan, work plans, PMPs. Sub-grant documents: solicitations, applications, selection memos  Primary producers & post-harvest handling companies  Supplied by REAP project for ET review  KIIs with selected beneficiaries  FGDs in regions  Data extracted from mini-survey findings None encountered FINDINGS Feedback from interviewees indicated that REAP’s trainings (business and technical) had a positive influence on their performance; however, some issues regarding agronomy for FSCs/MSCs and innovation and technology for post-harvest handling enterprises, still needed to be addressed. The technical trainings provided by the project were well-designed and, in most cases, in line with the expectations of the applicants. According to the beneficiaries, the structure and content of the technical trainings was sufficient to improve their skills, help increase productivity, and contribute towards increasing incomes. Feedback from the countrywide field trip interviews with grantees confirms this assertion, as noted in Figure 1 below: Figure 1: Influence of REAP Technical Trainings Influence of Reap Technical Trainings Category Increase % 0-10% 10-35% 35-60% 60-100% Increase in Productivity ( % of total respondents) 5% 15% 25% 55% Increase in Income ( % of total respondents) 3% 22% 30% 45% Source: Prepared by the Evaluation Team from data gathered from the field trips According to those beneficiaries who participated in the mini-survey, 46% rated TA as a significant contributor to improving the performance of their businesses. Less than 10% stated that TA had no impact on their business operations (see Figure 2, below). Figure 2: Question from Mini-Survey on TA Impact Source: Mini-Survey conducted by IRMS for Evaluation Team 13 During the Evaluation Team’s regional field visits, one of the beneficiaries - the owner of a post-harvest storage unit - pointed out that the main advantage of REAP’s technical training sessions was obtaining theoretical knowledge about modern agricultural practices in agronomy. As a direct result, he managed to double his productivity levels and increase his income by 45%. However, while he expressed overall satisfaction with this development, he also suggested that it would have been even more useful for him if REAP’s training modules also included practical activities from which he could learn first-hand how, for example, particular pieces of machinery or equipment operated. The duration of TA was an additional concern that the majority of beneficiaries expressed about technical trainings. In most cases, REAP organized short-term training (2-3 days) with more focus on theoretical aspects, usually within a classroom environment. While it was agreed that this support helped lay the foundations in terms of upgrading technical skills, it was felt that without follow-up on a longer term basis, this newly acquired knowledge may, over time, not be implemented in practical terms due to possible lack of interest or in owners (managers) passing on these skills to relevant employees. A case in point was the owner of a FSC in Western Georgia who, during the interview, spoke about the importance of having more long-term, small group, and practice-oriented technical trainings rather than having short-term trainings in a class of 30 participants. This view was reiterated on a number of occasions by grantees suggesting that the issue of theory versus practical application remains a concern, in that they still lack hands-on training in a number of areas including, in some instances, greenhouse management especially with regard to irrigation. When asked about their future expectations as to what REAP should deliver in terms of TA, 85% of those interviewed indicated a high level of expectation that such training would be more technically-orientated on a demand-led basis, as highlighted in Figure 3 below: Figure 3: Expectations of Beneficiaries Regarding Future REAP Trainings Type of Training Expectation For Future Training Low Average High Technical 0% 15% 85% Business 45% 35% 20% Source: Prepared by the Evaluation Team from data gathered from the field trips On the other hand, unlike technical training, there was little enthousiam for business training delivered by REAP, which interviewees felt did not significantly impact the performance of their respective businesses. The majority of them are in fact existing businesses that already possess sufficient experience in business￾related issues to be largely operational. Although there is an opportunity to further improve their managerial skills, the marginal rate of benefit of one additional training is much lower than is the case for technical trainings. In any event, there was little demand or requests from grantees for further support in business planning or management as most interviewees felt they already had the relevant functions in place to move forward. An additional view expressed was that FCS/MSC clients (mainly local farmers) are also more interested in practical advice and demonstrations, and it is their expectation that these centers offer such support. This makes it a more compelling argument for the focus to be on the practical as opposed to the theoretic, not least, as farmers often have to make decisions about inputs and equipment as the occasion demands (which could be influenced by weather conditions, pest infestations, machinery maintenance requirements, among others). CONCLUSIONS The structure of REAP’s training sessions played a useful and important role in addressing some of the technical needs of beneficiaries. For the most part, existing (and often basic) knowledge has clearly been upgraded for most of these businesses, especially in agronomy related matters and, in particular, with respect to post-harvest management. It can therefore be concluded that REAP has met its obligations in terms of contributing towards improved business performance but that its existing training modules may need to be re-visited in order to determine the optimum format needed to satisfy client demand. According to the beneficiary in West Georgia referred to above (greenhouse vegetable and herb producer), his expectation was to receive more practical-oriented training, delivered by an international expert who would be available to work with him on a day-to-day basis, say, intermittently over a 3-month period, during which he and his employees would benefit from the expert’s experience and expertise in 14 dealing with matters as they arose or advising on preventative action where potential problems were anticipated. RECOMMENDATIONS REAP should consider re-designing its training modules to focus more on ‘learning-by-doing’ approach - cycle process training (mainly through international experts then later by local consultants). A major advantage in adopting this approach is that it will serve to ensure that client expectations are adequately met in line with their specific needs. In support of this initiative, REAP beneficiaries should also be involved in the preparation of Terms of Reference (ToR) for international experts, including design of the expected deliverables following such interventions. At the same time, local experts could be engaged under the auspices of the project to work alongside contracted international experts with a view to learning best practices from them and, just as importantly, how to advise clients on addressing pre-existing conditions or specific agricultural-related matters as they arise. In the future, as international short-term expertise (STTA) is ramped up, pairing of local experts with international consultants should become common practice within the project’s implementation approach. In this way, the capacity of local experts will be increased due to the knowledge and experience gained by effectively ‘shadowing’ foreign experts as they support REAP’s own clients. This approach would also contribute towards the sustainability of agribusinesses if local expertise becomes available (and affordable) to them in addressing their immediate or anticipated needs. In support of its initiative, REAP should consider reducing any future planned business training in favor of the above-mentioned technical support, which is more in demand for the reasons elaborated. Another possible practical action might be supporting the installation of video demonstration units in farm centers, which could run on a non-stop basis, allowing customers to view those running or to request other videos that may be available on a particular agricultural topic. 4.1.3 Is the provided and intended TA specific and tailor made to each of the grantees needs? Are the quality of topics and instructors of the training appropriate for Georgian agriculture as defined by REAP beneficiaries? SUMMARY OF EVALUATION DESIGN AND METHODS Evaluation Question Type of Analysis Conducted Data Sources Data Collection Method & Sample Size Limitations Is the provided and intended TA specific and tailor made to each of the grantees needs? Are the quality of topics and instructors of the training appropriate for Georgian agriculture as defined by REAP beneficiaries Description – based on content analysis of expert opinions & review of relevant documentation  Project documents: weekly & quarterly reports, M & E plan, work plans, PMPs. Sub-grant documents: solicitations, applications, selection memos  Primary producers & post-harvest handling companies  Supplied by REAP project for ET review  KIIs with selected beneficiaries  FGDs in regions Data extracted from mini-survey findings None encountered FINDINGS The Evaluation Team visited 40% of REAP supported FSC/MSCs (12 out of 29) 80% of whom indicated that they were satisfied with the project’s TA interventions. In the Team’s assessment, the training materials appear relevant and appropriate for the local agricultural practices and conditions. According to the feedback obtained from the beneficiaries, the TA was in most cases fully in line with their expectations and relevant within the Georgian context. For instance, an FSC director in Racha region mentioned that the REAP project’s TA was a good opportunity for his staff to increase skills and knowledge in modern agricultural practices. In particular, he highlighted the high level of competency of the instructors whose participatory approach in the training workshops was greatly appreciated by the audience. This response appears to be corroborated by the mini-survey results conducted by IRMS – (see Figure 4 below) - which indicate that 55% of respondents (REAP grantees and non-grantees) rated the topics covered by the training as relevant or very 15 close to their expectations. Interestingly, the survey encountered no dissenting voices regarding REAP’s training programs, meaning that all respondents were satisfied with the training sessions. Figure 4: Question from Mini-Survey on TA Relevance Source: Mini-Survey conducted by IRMS for Evaluation Team However, the Evaluation Team noted an area of common concern regarding TA resulting from interviews with project beneficiaries held during the field trips, namely, the weak linkage between REAP’s training initiatives and local knowledge institutions (vocational colleges, universities, extension centers). For example, the director of a FSC/MSC in Western Georgia pointed out the importance of strengthening the capacities of these institutions so that businesses like his would have access to TA expertise and knowledge at a local level once REAP’s activities ended in September 2018. While this is not currently part of REAP’s mandate, it was felt that the project could play an active role in addressing this issue in the remaining years. Regarding this matter, the Evaluation Team acknowledges that in its first year of implementation, following a mapping of educational institutions at the start of the project, REAP organized a theory and practice training program in post-harvest handling and refrigeration technologies (through its sub-contractor WFLO) for professors and instructors from Akaki Tsereteli State University, Shota Rustaveli State University, and Agricultural University of Georgia. In addition, the universities were provided post-harvest handling toolkits and modern resource materials translated by REAP into the Georgian language that could be used for teaching. As a result, Akaki Tsereteli State University and Shota Rustaveli State University revised and updated curricula of their courses in post-harvest handling and refrigeration that are being offered to more than 300 students. CONCLUSIONS REAP appears to be adequately responding to the specific training needs of the FSCs/MSCs in particular, in line with their expectations (demand driven). The TA Component delivered by REAP was well-designed and the instructors of the trainings were, in most cases, high level professionals. The topics selected for the specific training modules were in line with the expectations of the beneficiaries and very closely aligned with the Georgian context. TA provided by the project consisted of both theoretical and practical modules. These were participatory events with the beneficiaries able to interact with instructors and obtain more detailed information about the discussed topics. In summary, REAP’s beneficiaries expressed a keen interest in collaborating further with the project to develop bespoke training modules that are aimed at addressing specific needs, and which are more of a technical nature than business-related. According to beneficiaries (and also acknowledged by the Ministry of Agriculture), it can further be concluded that local knowledge institutions do not have enough capacity to provide good quality training to REAP’s project beneficiaries after the project ends in September 2018. This can therefore be viewed as a possible detriment to longer term sustainability of some of these grantees who until now have largely relied on REAP to propose training modules and workshops, usually for groups of participants in selected countrywide locations. RECOMMENDATIONS REAP should continue with current TA initiatives but should also consider linkages or synergies with local academic or vocational institutions for knowledge distribution and access to REAP materials for interested parties. The project could also consider building the capacity of these institutions to conduct training which 16 would serve to improve local access to expertise, especially once REAP has ended. It is also recommended that the project exerts more effort in providing the above-mentioned technical trainings for the beneficiaries. In particular, it is important that training modules be more oriented on practical exercises. However, the importance of practical exercise should not totally disregard the theoretical component in the module. In order to achieve a wider scope of knowledge distribution, it would be beneficial if REAP could develop a web platform where all relevant materials could be uploaded for access by interested parties. It is vital that such a web portal continues to operate after the project is finished and thus worthwhile for the project to review and consider establishing linkages with other similar existing web portals for future collaboration. 4.1.4 Are the grantees that are selected a good and fair representation to sustainably increase the availability and utilization of agricultural inputs, services and technical information among small-holder and emerging commercial farmers? Were the requirements selected in a fair and transparent manner as described by initials project design documentation? Was the grant selection process fair and transparent? FINDINGS REAP’s Work Plan for Year 2 includes the following language (with the critical passage italicized): “In PY2, and in succeeding years, it is critical that the focus remain on the central figure in the program, the agricultural entrepreneur who receives a grant. REAP procedures, processes and programs all must support this central figure, the grant recipient. Unless he and his fellow grantees are successful in their individual investments, REAP cannot succeed as a project. Thus, REAP will ensure that cultivating entrepreneurship remains its overarching thrust (p. 7).” Toward this end, REAP instituted a rigorous application and screening process, as described in its Year 1 Work Plan: “Successful applicants will demonstrate market demand and profitability for their project, present market description, including potential clients and competition analysis, and provide financial projections (p. 6).” The Evaluation Team found that REAP’s implementation has remained faithful to the principles expressed in the above quotes. In the first place, those grantees who participated in the interviews consistently exhibited characteristics commonly associated with entrepreneurship. They were, as a whole, ambitious and growth￾oriented; willing to take financial risks, offer new products/services, and adopt new modes of working; looking to enter new markets; forward thinking; and focused on customer service with a good awareness of their customers and their needs. Moreover, the application process, including the application form itself and the internal vetting process, appears to have done a good job screening in applicants possessing these characteristics and screening out applicants who did not. On juxtaposing the successful applicants participating in KIIs and the unsuccessful applicants participating in the FGDs, the Evaluation Team observed the following key differences between the two groups: SUMMARY OF EVALUATION DESIGN AND METHODS Evaluation Question Type of Analysis Conducted Data Sources Data Collection Method Limitations Are the grantees that are selected a good and fair representation to sustainably increase the availability and utilization of agricultural inputs, services and technical information among small-holder and emerging commercial farmers? Were the requirements selected in a fair and transparent manner as described by initial project design documentation? Was the grant selection process fair and transparent?  Description – based on content analysis of expert opinions & review of relevant documentation  Project documents: weekly & quarterly reports, M & E plan, work plans, PMPs, sub￾grant documents, solicitations, applications, selection memos  Primary producers & post-harvest handling companies  Failed and rejected grantee applicants  Submitted by REAP prior to mobilization and during the evaluation mission  One-to-one interviews with selected grantees in the regions  FGDs in the regions with selected individuals / company representatives  None encountered  None encountered  None encountered 17 Successful Applicants (Grantees)  Existing business, typically with several years of operation and looking to expand existing lines of business or enter into new lines of business.  Sufficient resources to meet the 70% cost-share requirement in the form of cash savings, land, building and equipment, investors, bank loans or other sources.  Had existing bank loan or had borrowed from commercial banks in the past. The majority of existing loans were secured under Agriculture Projects Management Agency (APMA) preferred lending program.  Aware of and desirous to achieve international quality and/or safety/hygiene standards.  Often involved in their own on-farm production activities in addition to the supported business. In some cases, this on-farm production was separate from their supported business but in most cases, it played a supporting role to the supported business. Consequently, grantees understood farming, agronomic practices, and the situation of small farmers in the catchment areas served by their business.  Bought into the idea of providing ongoing TA to small farmer customers or raw materials suppliers as a means to: a) add value to the services they provided; b) position themselves in the marketplace; and c) create greater customer loyalty. When asked, for example, what distinguished him from his competitors, one FSC/MSC grantee replied, “I give better service. I have demonstration fields where I give farmers advice and I give farmers trainings in my shop as well. Farmers come to me because I help them and they trust me.”  In many cases, the grantee was already providing TA to small farmers before receiving the REAP grant. As one FSC/MSC grantee noted, “I have been an agronomist for 12 years and am very experienced in agriculture. I decided to go into this business because other farmers were constantly asking me for advice on how to farm. They would even stop me on the road to ask for advice. Based on this, I determined that there was substantial demand for these types of services, and I decided to open this farm service center.”  Well-articulated business ideas.  Existing commercial relationships with hundreds or even thousands of small farmers as customers of products or services or as raw materials suppliers, but also looking to expand those commercial relationships.  Little difficulty filling out the REAP application form. Perhaps the one quote that best demonstrates the mindset of many (although not necessarily all) of REAP’s grantees is the following, offered by one of the program’s FSC/MSC grantees, “I spend a great deal of time thinking about my relationship with my customers and with the farmers in this area. I think a lot about their dignity and that they should be happy when they do business with me, and that they should also be prosperous on their farms. One of my motivations for building this new facility with a conference room is the large number of farmers coming to me for advice and how I’m currently not able to meet their needs.” Unsuccessful Applicants  Start-up business or businesses with fewer years of operation.  Fewer resources to meet the 70% cost-share requirement. (Perhaps related to this, the Evaluation Team’s casual observation was that the grant amounts requested by unsuccessful applicants tended to be smaller than the grant amounts received by successful applicants.)  No existing bank loan or previous loans from commercial banks, even under the APMA preferred lending program.  Less well-articulated business ideas.  Difficulty filling out REAP application form. 18 The grantee interviews and non-grantee FGDs did not, with one exception, uncover evidence that members in either group had legitimate criticisms related to the grant application and selection process, including the transparency of this process. Rather, the general consensus was that this process was both fair and transparent. While there were scattered complaints about the difficulty of the application form, the fact that so many businesses were able to complete the form (albeit some with program support) indicates that this complaint was not generally shared. The one exception to the above finding that arose in the FGDs involved evidence of a potential lapse in transparency in that a number of rejected applicants reported either: a) waiting a long time (up to several months) to be informed that their grant proposal had been denied; b) found out their application had been denied after they contacted the program to determine its status; or c) never received notice that their application had been denied. The most extreme example of this was one discussion participant who, after several months of waiting, had yet to receive word on his application and only realized that it had been denied mid-way through the discussion, “I thought you came here to talk to me about the grant I was getting, but then I realized after talking to you a bit that I didn’t get it after all.” Along the same lines, the Evaluation Team asked rejected applicants if they knew the reasons their application was denied but in several instances they could not cite a specific reason. In some of these cases, they did mention that they were denied because they failed to achieve the minimum score of 70, but they could not say why they failed to meet the minimum score. Given the non-representative nature of the FGDs, however, it is not possible to determine just how common such lapses in transparency were. One benefit of the grant process that needs to be highlighted, and which has implications for both the sustainability of the grantees’ businesses and the on-farm production and income of their small farmer customers and suppliers, is the training infrastructure that the program has helped build. This includes not only the BSPs that the program has worked through but also the extensive network of training facilities covering much of the country together with dozens of businesses committed to the value proposition of providing training as a means to customer loyalty and business growth. The existence of this training infrastructure takes on added relevance when considering the numerous and significant deficiencies that exist among small Georgian farmers and within Georgian agricultural markets. During grantee interviews, the Evaluation Team asked the grantees for their views on the major constraints facing small farmers. Grantees consistently mentioned the following constraints: 1. Low use of productive inputs 2. Limited access to mechanization 3. Limited access to cold storage 4. Limited knowledge about modern agronomic practices 5. Low yields and poor quality 6. Lack of integration into markets - most farmers sell at the farm gate to distributors and are price￾takers without the ability to capture higher value-added from production. Due to low prices paid at the farm gate, farmers often do not bother to harvest their crop and bring it to market. Combined with this, heavy competition from imported crops drives the farm gate price for crops down even further. REAP is directly addressing the first three of the above constraints (with the exception of irrigation) and is working indirectly (for the most part) to address the remaining ones. Significant gaps remain, however, in addressing the final three which, in turn, threatens the sustainability of REAP’s grant investments. Specifically, the success of REAP-assisted FSCs/MSCs depend to a large degree on the on-going demand from small farmers for the products and services provided by the FSCs/MSCs, while the success of the processors and post-harvest handlers depends on the on-going supply of sufficient numbers of quality raw materials and final agricultural products. Given the current state of the Georgian agricultural sector, there is a risk that small farmers may not be capable to provide this demand or supply. CONCLUSIONS The REAP grant application process successfully selected grant recipients who: a) are true agricultural entrepreneurs that are offering vital services to small farmers and other agricultural market actors; b) demonstrate high potential for continued long-term business growth; and c) have substantial outreach to small farmers, whether as customers or suppliers of raw materials. The grant application itself was rigorous but fair, and was effective in screening in true entrepreneurs and screening out less evident entrepreneurs. 19 In a qualitative sense, successful grant applicants were distinguishable from non-successful grant applicants in several respects thus reinforcing the Evaluation Team’s conclusion that REAP’s grant application and screening processes worked effectively to further program objectives of supporting agricultural entrepreneurs with potential to help catalyze growth and development in Georgia’s agricultural sector. The grant application process was, moreover, both fair and, on the whole, transparent, although with possible lapses in transparency in terms of informing rejected applicants that their application had been denied and the reasons for its denial. Many of the grantees selected for program participation brought with them a commitment to providing TA and training to their small farmer customers and suppliers as a means to deliver quality service, increase customer loyalty, and better position themselves within the marketplace. This, together with the program’s focus on training, emphasis on creating training facilities at grantees’ place of business, and on-going work with local BSPs, have contributed to the creation of a nation-wide training infrastructure. This infrastructure offers intriguing opportunities to leverage it so as to provide TA and training to the small farmers who are either customers of REAP-supported FCS/MSC grantees, or raw materials providers for REAP-supported processors and post-harvest handlers. RECOMMENDATIONS The nation-wide training infrastructure that REAP has helped build opens up a number of potential opportunities to deliver TA and training to REAP’s diverse stakeholders and small farmers working in grantee serviced catchment areas over the remaining life of the program and even possibly thereafter as part of a sustainable agricultural training network. REAP would be well served to investigate strategies for leveraging this training infrastructure. Although grantees had few complaints about the fairness and transparency of the grant application process, the long lag time between the grant application and notice of denial and, in some cases, the failure to communicate this at all (as cited by several FGD participants) indicates a possible important lapse in transparency. While the Evaluation Team does not have evidence to conclude that such lapses were systemic, they appear to have occurred with sufficient frequency that it may be worthwhile for REAP to review its other communications systems so as to determine whether such breakdowns are occurring in other areas. Finally, one way to address the risk found by the Evaluation Team with respect to the demand of small farmers for FCS/MSC services and supply issues associated with PHPs, is to provide TA and training to the farmers living in the catchment areas served by REAP’s grantees. The training infrastructure created through REAP’s various interventions offers an institutional structure and network through which this TA and training might be provided to REAP grantees and non-grantee firms. 4.1.5 Has the program been successful in integrating the cross-cutting issues (gender, environment, workforce development and access to finance) into the program implementation, when appropriate? SUMMARY OF EVALUATION DESIGN AND METHODS Evaluation Question Type of Analysis Conducted Data Sources Data Collection Method Limitations Has the program been successful in integrating the cross-cutting issues (gender, environment, workforce development and access to finance) into the program implementation, when appropriate?  Description – based on content analysis of expert opinions & review of relevant documentation  Project documents: weekly & quarterly reports, M & E plan, work plans, PMPs, sub-grant documents, solicitations, applications, selection memos; project staff  Primary producers & post-harvest handling companies  Key stakeholders  Submitted by REAP prior to mobilization and during the evaluation mission  One-to-one interviews with selected grantees in the regions  KIIs with financial institutions, women’s group, GoG  None encountered  None encountered  None encountered FINDINGS The Evaluation Team’s investigations indicated that cross-cutting issues are well integrated in REAP’s implementation plans. However, access to finance still remains a challenge for agricultural businesses. For 20 example, without the Government of Georgia’s (GoG) APMA20 cheap loan program, most grantees would have found difficulty in accessing their contribution towards the matching grant. However, the Ministry of Agriculture indicated that it is unlikely that this form of subsidy would be sustainable from a budgetary point of view in the long-term and that, therefore, other means of credit for rural agriculture/ agribusinesses would need to be developed in time, although no limit was mentioned. Discussions with REAP indicate that the project team recognizes the importance and significance of GoG’s cheap loan program in stimulating finance to the agriculture sector while accepting that APMA will not exist in perpetuity. REAP has therefore begun the process of working with commercial banks to train loan officers on agriculture lending. At the same time, the project works with all grantees and specific non￾grantees on developing business plans and proper financial record-keeping to assist in obtaining a loan. REAP has also successfully managed to achieve the targets set by USAID related to women-managed businesses within the grant component of the project. Moreover, it was evident that women beneficiaries were not simply selected in order to achieve those targets but on the basis that they were able to demonstrate they were running a sustainable business with a clear vision of where they wanted to be in the future. Nevertheless, business development opportunities for women entrepreneurs and owners of SMEs remain limited in terms of accessing finance, especially in the agriculture sector, due to the current ownership rights issue affecting women in Georgia21. According to interviewees, commercial banks remain reluctant to lend to women although there is some evidence from the financial sector itself that this state of affairs is slowly changing. If women can prove ownership then lines of credit appear more likely than before. Several banks interviewed confirmed this view. Regarding environmental issues, the Evaluation Team found no reference to these in REAP reports. In any event, it is too early to comment at this stage as to what may be the case in several years’ time for grantee enterprises, especially with regard to, for example, the processing industry, with respect to recycling or waste disposal issues and their possible effects on the environment. In all probability, an end-of-project evaluation would be more likely able to shed light on these issues, including the extent to which REAP has been involved. With regard to workforce development, most grantees appeared to be taking steps to ensure their existing staff had the necessary skills to contribute towards their respective businesses. Staff were encouraged to attend REAP training sessions delivered either on site or at other locations. Brief discussions with staff themselves at interview sessions confirmed that owners/managers regularly encouraged them to upgrade their knowledge and skills. The Evaluation Team also met with a BSP, which had been commissioned to supply business training to REAP’s clients on request. The company’s first session was in September 2015 in which the staff of selected FSCs/MSCs was trained in customer service. These were 2-day courses for five participants, based on 2 x 4 hour modules. According to the Director of the BSP, her post-training client satisfaction survey indicated a positive response to the training received. In addition, she advised that REAP’s application processes and procedures were uncomplicated and further commented that she noticed a considerable improvement in the professionalism of these centers in recent years. REAP’s internship program also appears to be playing an active role in its workforce development initiatives through engaging 17 students as interns in the project’s Tbilisi, Telavi and Kutaisi offices, with a view to enhancing their learning experience in these environments. Interns are involved in practical field days and demonstrations so that they can learn about modern agricultural practices from both local and international experts. CONCLUSIONS There is clear evidence from REAP’s reporting documentation and from data gathered from primary sources around the country, that cross-cutting issues are well reflected in the project’s implementation activities. No serious anomalies were encountered during the Team’s investigations and thus it can be concluded that REAP’s mandate is being respected to date. 20 In 2013 GoG announced two agriculture support schemes: Cheap Agro Credit Program in cooperation with 11 banks in Georgia and Grants for Agro-Processing Companies 21 Women have the same legal ownership rights as men regarding land and property in Georgia – the challenge for them mainly concerns cultural and/or traditional barriers that they face on an on-going basis 21 RECOMMENDATIONS Based on data and information accessed and available to the Evaluation Team at the time of the evaluation mission, there are no specific recommendations at this time with respect to the remainder of the project. However, with regard to the access to finance issue and how best a future USAID-funded program might reach and support entrepreneurs and SMEs in the agriculture sector, see Section 4.3.3 below for comment on how this might be achieved. 4.2 HOW IS THE PROGRAM PERCEIVED BY BENEFICIARIES? SUMMARY OF EVALUATION DESIGN AND METHODS Evaluation Question Type of Analysis Conducted Data Sources Data Collection Method Limitations How is the program perceived by beneficiaries?  Description – based on content analysis of expert opinions & review of relevant documentation  Project documents: weekly & quarterly reports, M&E plan, work plans, PMPs, sub￾grant documents, solicitations, applications, selection memos; project staff  Primary producers & post-harvest handling companies  Key stakeholders  Submitted by REAP prior to mobilization and during the evaluation mission  One-to-one interviews with selected grantees & TA recipients in the regions & Tbilisi  KIIs with cross￾section of private & public sector organizations / institutions  None encountered  None encountered  None encountered FINDINGS During each of the interviews with the different program stakeholders (grantees, TA recipients, implementing partners), the Evaluation Team asked a series of probing questions trying to ascertain how the stakeholders perceived the program in terms of the success of the collaboration, the quality of the assistance provided, and areas in which the program could improve. Almost without exception, the responses were favorable in each case. Each interviewee was able to identify one or more specific and tangible benefit that his/her business or organization had received as a result of the collaboration - moreover that this benefit either would not have occurred, or would have occurred later, or in a more haphazard fashion, without the program’s assistance. In addition to this, interviewees nearly universally complimented the quality of the assistance they received from REAP staff, local BSPs, and international experts. Finally, interviewees offered few criticisms, or at least few criticisms that they expressed with any frequency. Some highlights of these perceptions include:  Program assistance created significant ‘additionality,’ in that it helped catalyze investments and business expansion that either would not have occurred otherwise (according to 80% of the interviewees) or would have occurred over a longer period of time and/or in a less well-thought out or haphazard, fashion.  TA was provided in response to specific needs identified by the grantees/organizations and was, for the most part, highly useful and delivered by experts with the requisite technical and managerial/business knowledge. However, a number of grantees also mentioned that, at times, the TA could have/should have been more practical and that short-term training sessions were not really sufficient. Interviewees stated that they were in need of on-going, or longer-term TA, that would guide them in implementing the concepts they learned through the one-off or short-term TA. This comment was particularly prevalent with regards to training related to meeting international quality or safety/hygiene standards, such as the Hazard Analysis Critical Control Point (HACCP) standards. According to one interviewee, for example, “It is a very different thing to learn about the [HACCP] standards and to implement them. I am not confident in my ability to implement the standards and I need more help to do it.”  REAP staff were responsive and attentive; maintained an on-going, collaborative relationship with grantees; and provided useful and practical advice related to diverse aspects of the business and the 22 grant implementation process. According to one grantee, “The project is very responsive and always gives good and timely answers to my questions. As time goes on, the project is in regular contact with me, and they are able to resolve all issues in a timely manner.”  REAP’s procedures were simple, straightforward, and transparent and the project consistently conducted its business in accordance with these procedures. For example, when one interviewee was asked about his opinion of the program, he commented, “So far so good. Everything is written and they go by the book. They do their part and we do ours.”  Materials and equipment procured through the REAP procurement process were consistently high quality, or of sufficient quality to perform their function and, for the most part, operated in a timely manner. Moreover, REAP staff played an active and helpful advisory role throughout the procurement process. Given the early difficulties REAP had with procurement related to USAID procurement policies, the Evaluation Team expected procurement to be an area in which grantees would express a relatively higher level of dissatisfaction, yet the Team did not find any evidence that this was the case, but rather found a consistently high level of satisfaction among grantees with the procurement process.  Most of the grantees interviewed were able to complete the grant application on their own. However, there were some grantee interviewees who did receive assistance in completing their grant application, a good share of those whose proposals were initially declined but who were then asked to resubmit. These grantees expressed strong gratitude for the program, indicating that assistance with preparing the grant proposal was instrumental in all cases for their eventual success in securing the grant. The one most consistent criticism of the program among grantees was the 70% grantee cost-share requirement, although in complaining about it, they also acknowledged that it was the best option available and, ultimately, still a reasonable deal. The majority opinion among grantees was that the 70% cost-share requirement was overly large and burdensome. Yet, when asked why they went ahead with the grant given the large cost-share requirement, grantees consistently said that they did it because it was still the best option available. According to one grantee, for example, “I need to get going with my business, and this was the best way to do it. If I didn’t do it now, when was I going to do it? This was the best option I had.” That is, the 70% cost-share requirement was, nonetheless, still a good deal on balance, and it was the most viable option available to them to finance their planned business expansion. It is worth noting here that unsuccessful applicants, not surprisingly, also complained about the 70% cost￾share requirement with several pointing out that such a high-cost share requirement effectively ruled out start-up businesses from grant consideration, as few start-ups have the cash or non-cash resources to meet this requirement. Start-ups also face difficulties securing bank loans, which might be used to meet their cost-share requirement, because they lack the collateral to qualify for loans. According to them, business lending in Georgia is largely collateral-based. This is true even for the preferred lending program being administered by APMA. Other FGD participants argued that such a large cost-share requirement also disadvantages women-owned businesses as they are even less likely to have such resources on hand than male-owned businesses and, when they do, male relatives often control these resources. CONCLUSIONS The REAP program is overwhelmingly perceived as positive by grantees, TA recipients, and implementing partners. The benefits received from participation in the program are tangible and are abetted in turn by a consistently high quality of assistance from a responsive and attentive program staff that is involved in all stages of the grant application and implementation process, equipment and materials procurement, and TA provision. In addition to this, the program operated under a set of transparent and straightforward procedures to which it adhered consistently. Complaints were relatively rare and focused primarily on the 70% cost-share requirement and on the need for more practically-oriented TA in certain cases and, at times, on the limited usefulness of on-off or short￾term TA and the need for on-on-going follow-up TA that guides the grantee through the implementation process, particularly with regards to meeting international quality or safety/hygiene standards. RECOMMENDATIONS The primary recommendation emerging from this evaluation question is one already offered above: to shift the TA increasingly towards a ‘learning-by-doing’ approach - cycle process training in such a way that it 23 guides grantees and TA recipients through the process of implementing the new technologies and business practices in which they have been instructed by the program. 4.3 WHAT, IF ANY, CORRECTIONS ARE NEEDED TO REAP’S IMPLEMENTATION APPROACHES DURING THE REMAINING TERM OF THE PROJECT? SUMMARY OF EVALUATION DESIGN AND METHODS Evaluation Question Type of Analysis Conducted Data Sources Data Collection Method Limitations What, if any, corrections are needed to REAP’s implementation approaches during the remaining term of the project?  Description – based on content analysis of expert opinions & review of relevant documentation  Evaluation Team’s assessment of findings and conclusions from all of above questions  Project documents: weekly & quarterly reports, M&E plan, work plans, PMPs, sub￾grant documents: solicitations, applications, selection memos; project staff  Primary producers & post-harvest handling companies  Key stakeholders  Analytical results of Evaluation Team’s investigations collated by team members  Submitted by REAP prior to mobilization and during the evaluation mission  One-to-one interviews with selected grantees & TA recipients in the regions & Tbilisi  KIIs with cross￾section of private & public sector organizations / institutions  Evaluation Team’s findings following KIIs, FGDs both in Tbilisi and in the field  None encountered  None encountered  None encountered  None encountered FINDINGS As a result of its analysis and subsequent responses to USAID’s questions highlighted and answered above in some detail, the Evaluation Team has proposed some suggestions as to how REAP might re-align its TA interventions for the remaining three years of the project, in terms of addressing the demands of its main beneficiaries including grantees and non-grantees. It has already been noted that few corrections are in fact needed in the way in which REAP has to date been implementing its activities or delivering on its mandate. Indicator targets have in general been met or in the process of being achieved. Beneficiaries have largely expressed their satisfaction with the project’s support interventions subject to the issues raised above with respect to training modules. As REAP has already committed its $6million in matching grants, its primary focus for the remaining three years will be on the provision of TA to grantees and non-grantees on a demand-led basis. With approximately $2.5million of REAP’s budget available for this activity, this means that the project will effectively be focusing its support on 70 grantees and over 100 non-grantees until September 2018. As any future TA interventions will largely depend on client requests for specific types of assistance, delivery of this support will effectively involve the preparation of a careful planning schedule to ensure clients’ needs are met on time and within the scope of the project’s budget. CONCLUSIONS Based on the above findings, the Evaluation Team is only in a position to conclude that with grant funds already committed REAP has satisfied its mandate in this respect. No corrections to the project’s implementation are therefore envisaged here other than the need for effective planning of the expenditure of the $2.5million available for TA support to REAP’s beneficiaries subject to demand. In the absence of a second year Annual Report, which would have allowed a year-on-year comparative analysis of reported results, the Evaluation Team is only in a position to confirm its earlier findings relative to the previous questions highlighted above, in reference to suggested recommendations for the remainder of the project. In short, at this early stage in the project’s implementation, it can be concluded that REAP has in a sense leapt out of the starting blocks in pursuit of reaching the finishing line, with only a few anticipated hurdles in the way of achieving its goal. 24 Whether this strategy is the most effective utilization of project funds is open to debate given the rather limited target audience (70 grantees and 100 plus non-grantee beneficiaries). The question therefore remains as to whether the same or a similar amount of money for a future such project could reach and serve a wider client base, including potential entrepreneurs and start-ups in agriculture/agribusiness sectors, not least in addressing the still problematic issue of access to finance in Georgia, which remains a hindrance to development and growth in agriculture as acknowledged in the GoG’s Strategy for Agricultural Development in Georgia 2015-2020 (Measure 3.1.4 – Developing specific tools to strengthen the agricultural credit and leasing system). As observed in the above findings, matching grants clearly play an important role in fostering business development in agriculture and in improving much-needed technical skills for beneficiary companies to increase both domestic and export opportunities. However, as has been noted, REAP’s matching grants have to some degree encouraged clients to borrow funds from the commercial sector, family, friends, or partners in order to meet their (in this case) 70% contribution, as few actually have the required funds themselves. These borrowed funds need to be repaid in due course so the question remains as to whether and to what extent the project has the resources to follow up clients’ progress in order to determine in the months/years ahead their true operational status. In other words, does the availability of a matching grant (from an international donor) ensure or go some way towards the sustainability of the grantee? If the definition of sustainability rests with the assumption that an organization has the tools in place to function profitably for the foreseeable future, then it can be concluded that in one respect REAP has indeed made a contribution towards this eventuality i.e. by encouraging a carefully selected group of enterprises to invest in their businesses, by offering an alternative channel through which to acquire funds to support their development plans and, as noted above, without which a significant number of enterprises may not have proceeded at all. Therefore, REAP’s interventions through grants and TA have been important for the rural agriculture sector in Georgia as envisaged in the project’s SOW. However, below is an alternative approach that USAID may wish to consider for any future planned agriculture project aimed at developing the country’s agriculture sector. RECOMMENDATIONS The Evaluation Team does not have any recommendations with respect to the remainder of the project as per the above findings and conclusions in terms of fine-tuning existing delivery mechanisms regarding TA for beneficiaries. However, an option for consideration is the following scenario. Instead of grants for targeted beneficiaries - SMEs, entrepreneurs including women and young people, primary producers, processors and/or minority groups, all of whom may be considered as riskier clients for traditional finanical institutions - consider the creation of a Commerical Finance Fund (CFF) funded by USAID in collaboration with a private sector investor, which would make funds available to targeted clients at competitive rates. Such an initiative would be managed by the private contractor and run effectively as a revolving fund. Once USAID’s initial investment was made, no further cost would be involved and, if successful, the fund could run in perpetuity22. There would be no limit to the number of beneficiary clients other than the speed with which funds were repaid once borrowed. In addition, a client could receive credit on more than one occasion, subject to meeting the loan requirements. The concept behind this type of initiative is that it widens the scope for potential clients who otherwise would not have access to funds for expansion or starting up. The rationale for this approach is that once businesses become more established, they will become ‘bankable’ in the commercial banking sector. This, in turn, supports both client and the financial sector. The more profitable clients become, the more likely they are to continue to invest and grow with the potential for increased incomes and employment opportunities in rural communities. As growth will be dependent on enterprises adopting sound business principles, the finance fund will offer support and guidance on how to manage businesses successfully. The technical aspects of running the business will result from the growing confidence of the company to recognize its needs and to acquire technical expertise through its own offices – i.e contracting either local service providers or international experts on a needs-only basis. There is no reliance in this perspective on donor funds other than the original investment in the finance fund which would in effect be registered locally as a non-banking financial institution subject to the laws of 22 See USAID’s Commerical Finance Fund – managed by Crimson Capital – as a reference – www.crimsoncapital.org 25 Georgia in accordance with its Central Bank requirements. As this appears to be largely in line with GoG’s vision for the next 5 years for the agriculture sector, it may well be worth investigating further by USAID. 1 ANNEXES 2 ANNEX A: STATEMENT OF WORK 3 STATEMENT OF WORK MID-TERM EVALUATION OF THE USAID RESTORING EFFICIENCY TO AGRICULTURAL PRODUCTION (REAP) PROGRAM May 2015 1. EVALUATION PURPOSE AND USE Project Numbers: Contract No. AID-114-C-13-00002 Project Dates: 09/27/13-09/26/2018 Project Funding: $19,589,463.00 Implementing organization: CNFA Contracts Officer's Representative (COR): Shamenna K. Gall Alternate Contract Officer's Representative (Alternate COR): David Tsiklauri The evaluation Contractor must provide non-personal services for a mid-term performance evaluation23 of the Restoring Efficiency to Agricultural Production (REAP) activity. The evaluation Contractor must assess: 1) whether REAP's approaches are necessary and sufficient for developing competitive and commercially sustainable agribusinesses and increasing incomes and employment in rural areas in Georgia; 2) how the project is perceived by beneficiaries, i.e., grantees, technical assistance recipients, Government stakeholders and others who have used REAP services; and 3) whether and what corrections are needed in REAP's implementation approaches. The evaluation findings will be used to inform implementation approaches for the remaining years of project implementation and beyond. 2. SUMMARY OF SPECIFIC TECHNICAL REQUIREMENTS The evaluation Contractor must ensure that the evaluation team will complete the following tasks and provide the following deliverables within the terms defined by the contract: - Telecon with USAID/Caucasus to discuss the upcoming work. - Provide a draft evaluation design and work plan to USAID for review and comment prior to arrival in country. - Incoming briefing with USAID management to present the detailed evaluation design. - Conduct an evaluation in accordance with the USAID-approved evaluation design and work plan. - Outgoing briefing with USAID management to present the preliminary findings of the Evaluation. - Provide an evaluation report to USAID in accordance with Reporting Guidelines under Section 9 - Deliverables. The evaluation report should follow the "Criteria to Ensure the Quality of the Evaluation Report" of the USAID Evaluation Policy. - Submit USAID-approved evaluation report to Development Experience Clearinghouse (DEC) within 30 calendar days following the acceptance of the report by the USAID Task Order Contracting Officer's Representative (TOCOR.) - Submit quantitative dataset in a machine-readable format to the Development Data Library (DDL) as part of the Open Data initiative. 3. PURPOSE OF THE EVALUATION AND KEY EVALUATION QUESTIONS The purpose of this mid-term performance evaluation is to determine the relevance of REAP’s programmatic approaches in achieving intended life-of-project results and to provide recommendations on corrective actions and new directions for the remaining years of project implementation and beyond. The Contractor must review actual progress toward achieving key expected results and identify accomplishments, delays, challenges, and their impact on achieving REAP's life-of-project results. Specifically, on a broad scale, this evaluation will examine the relevance of REAP's approaches in achieving the intended results of developing competitive and commercially sustainable agribusinesses and increasing incomes and employment in rural areas. On a narrower scale, the evaluation will examine REAP's progress and 23 As per the 2011 USAID Evaluation Policy, "Performance evaluations focus on descriptive and normative questions: what a particular project or program has achieved (either at an intermediate point in execution or at the conclusion of an implementation period); how it is being implemented; how it is perceived and valued; whether expected results are occurring; and other questions that are pertinent to program design, management and operational decision making. Performance evaluations often incorporate before-after comparisons, but generally lack a rigorously defined counterfactual." 4 approaches in selection of grantees and awarding grants to SME, and in delivering technical assistance (TA) and training for grantees and non-grantees. The evaluation will also gauge the perception of the program by its beneficiaries, grantees and non-grantees, Ministry of Agriculture, and other stakeholders related to agriculture. Finally, it will determine whether and what course corrections are needed for the remaining term of the project. Question #1: Are REAP's approaches necessary and sufficient for developing competitive and commercially sustainable agribusinesses and increasing incomes and employment in rural areas in Georgia? - What are the major constraints/challenges/outstanding needs that inhibit the projects contribution toward achieving the results during the remaining term of the project? - Is technical and/or business training provided by REAP to assisted primary producers and SMEs sufficient to improve skills, yields, and incomes of farmers? - Is provided and intended TA specific and tailor made to each of the grantees needs? Are the quality, topics and instructors of the training appropriate for Georgian agriculture as defined by REAP beneficiaries? - Are the grantees that are selected a good and fair representation to sustainably increase the availability and utilization of agricultural inputs, services, and technical information among small-holder and emerging commercial farmers? Were the requirements for grantees selected in a fair and transparent manner as described by initial project design documentation? Was the grant selection process fair and transparent? - Has the program been successful in integrating the cross-cutting issues (gender, environment, workforce development, and access to finance) into the program implementation, when appropriate? Question #2: How is the program perceived by beneficiaries? - In this case the program can be defined as the activities, the actors, the results, etc. Beneficiaries include grantees, non-grantees, TA/grant recipients, Ministry of Agriculture, and other stakeholders related to agriculture (e.g. Georgian SMEs and agribusinesses, including primary producers, post-harvest handling enterprises, information service providers, processors, etc.) Specific sub-questions may include but not be limited to: how the GOG and other sector stakeholders perceive the program and its impact to date; how the grantees feel about the grant selection process; and what grantee applicants who were not awarded grants feel about the project. Question #3: What if any, corrections are needed to REAP's implementation approaches during the remaining term of the project? - Based on the findings and conclusions related to questions #l and question #2, the evaluation team must make specific recommendations with regard to REAP's implementation approaches. For example, the team may identify opportunities to improve the implementation of future similar grant programs, or suggest additional interventions to advance REAP's objectives. 4. METHODOLOGY The Mission is looking for the Contractor to suggest the best methods that minimize bias and provide strong evidence. The Contractor is expected to suggest the use of various data collection and analysis methods, both quantitative and qualitative, including document review, key informant interviews with project beneficiaries, the GOG, private sector, focus group discussions, survey instruments, and others. The methodology for any evaluation process that involves the selection of participants (e.g. surveys, focus groups, interviews) must be clearly explained and justified. For example, for a survey or mini-survey (if proposed), the number of respondents and their selection process should be explained and justified. The same is true for key informants, focus group discussions, and other methods as well. Selected respondents should be representative of women, youth, and vulnerable groups, where appropriate. The Contractor must conduct a desk review for REAP related documents, and will help identify areas that merit closer attention once the team begins its fieldwork. Reading materials will be available to the team prior to their arrival. The Contractor must develop a detailed evaluation design and a workplan, including data collection plan and drafts of data collection tools. A draft of the workplan and evaluation design must be shared with USAID TOCOR for review and comment prior to the evaluation team's in country arrival. The plan will then be presented to the Mission during the in brief in more detail. The evaluation design must include the evaluation matrix (an illustrative evaluation matrix for this study is given below). The evaluation design must explain how the evaluation Contractor intends to conduct the study in detail, including a detailed description of one or more proposed methodologies as well as limitations of proposed methodologies. It must explain in detail what 5 methods will be used to obtain answers for each evaluation question. The design must also explain how the proposed methodology (mix of methods) to conduct the study generate evidence to ensure rigor and reliability of results; and how and why the proposed methodology will minimize bias. The evaluation design must also include the data analysis plan for each question, draft questionnaires (to be included as an attachment), and other data collection instruments or their main features, criteria for assessing responses to evaluation questions, known limitations, and a dissemination plan. The evaluation design must also include specific sub-questions for each evaluation question, where needed. Again, the methods described herein are only illustrative and USAID expects that the Contractor will suggest the best methods that would generate most reliable and evidence-based answers to the key evaluation questions. 5. PERFORMANCE PERIOD The following levels of effort are illustrative and should serve only as an example of the staff which may be mobilized under this Task Order. These levels may not reflect the actual level of effort contracted, and the Contractor will be expected to submit its own estimate of the level of effort needed to fulfill the objectives. Total No of Days in Country/ Consultant including travel days No of Work Days in Country/ Consultant No of Days for Preparation and Report Writing Total No of Work Days/ Consultant International Technical Expert - Team Leader 23 19 14 33 (plus 2 travel days) Evaluation Expert 23 19 14 33 (plus 2 travel days) Local Consultant 19 19 4 23 A six-day work week will be authorized in Georgia with no premium pay. 6. PROJECTS DOCUMENTS FOR REVIEW AND LOGISTICS The USAID REAP COR, through the Mission's Economic Growth Office (EG), will put the Contractor in contact with REAP implementing partner and may provide help with organizing a small number of meetings as needed. Relevant reports and other project documentation will be provided by the Mission to the Contractor prior to travel to Georgia. These documents are:  Statement of Work as is stated in the REAP award;  IR 2.2 Project Appraisal Document (PAD);  REAP Work Plan(s);  REAP Quarterly Reports;  Initial list of in-country contacts;  M&E plans submitted and approved by USAID and performance data tables;  Various program documentation (grants manual, RFAs, grants selection memos, grants applications, monitoring reports, etc.)  Other deliverables (expert reports, publications) produced by various stakeholders. Prior to arriving to Georgia, the Contractor may decide to interview USAID/E&E officials in the United States, in addition to any other Washington-based experts as appropriate. The USAID/E&E M&E POC based in D.C. will assist in arranging these meetings, as necessary. While in Georgia, the Contractor will conduct meetings in Tbilisi. Some meetings will require traveling to regions outside Tbilisi to meet with REAP grant recipients, agribusinesses, primary producers, processors, information service providers and municipal government authorities. REAP program implementing partner may assist with setting those meetings. 7. DELIVERABLES 6 1. Detailed evaluation design and the work plan: The evaluation design must explain in details methodologies that will be used to collect required information. The design must outline in details what methods the Contractor will use to get answers for each evaluation question. The evaluation design must include a detailed evaluation matrix (including the key questions, methods and data sources used to address each question and the data analysis plan for each question), draft questionnaires and other data collection instruments or their main features, known limitations to the evaluation design, a work plan, and a dissemination plan. The refined design must be sent to the contract TOCOR three days prior to research team's arrival in country. This information together with the Mission's comments will be discussed in detail during the in-brief meeting with USAID and will be finalized per task order requirements. The work plan must include the anticipated schedule and logistical arrangements and delineate the roles and responsibilities of members of the evaluation team. 2. In brief with the mission: Within three (3) days of arrival in country, the Contractor must present a design plan and a work plan to USAID officials. This will be a maximum of 30 minute presentation of how the questions asked in SOW will be answered. Prior to in brief research teams may have working meeting/s with the evaluation TOCOR and REAP COR to agree all the details of the design. 3. Out brief: Prior to departure, the Contractor must present to USAID officials an outline (in bullets, possibly in PowerPoint or as a handout) of the evaluation report with general findings, conclusions, and anticipated recommendations. This will be a maximum one hour presentation. 4. Draft Report: The Contractor must submit a draft report within twenty (20) working days of completing the out brief with USAID. This document must explicitly respond to the requirements of the SOW, must answer the evaluation questions, be logically structured, and adhere to the standards of the USAID Evaluation Policy of January 2011 and the criteria to ensure the quality of the evaluation report. 5. Final Report: The Contractor must incorporate USAID's comments and submit the final report to USAID/Georgia within five (5) working days following receipt of final batch of comments on the draft report. Final evaluation report should follow USAID's template, and should not exceed 25 pages, excluding executive summary and annexes. The Contractor will make the final evaluation reports publicly available through the Development Experience Clearinghouse at http://dec.usaid.gov within 30 calendar days of final approval of the formatted report with USAID consent. In case it is determined that the full report includes sensitive information, sanitized version will be produced and submitted to DEC. 6. All records from the evaluation (e.g. interview transcripts and summaries, focus group transcripts, code books, etc.) must be provided to the evaluation TOCOR as requested. All quantitative data collected by the evaluation team must be provided in an electronic file in easily, machine readable format agreed upon with the TOCOR. The data should be organized and fully documented for use by those not fully familiar with the project or the evaluation. USAID will retain ownership of the survey and all datasets developed. In addition, the dataset must be submitted to the Development Data Library (DDL) as part of the Open Data initiative. Per the USAID evaluation policy, draft and final evaluation reports will be evaluated against the following criteria to ensure the quality of the evaluation report.  The evaluation report should represent a thoughtful, well-researched and well organized effort to objectively evaluate what worked in the projects, what did not and why.  Evaluation reports shall address all evaluation questions included in the statement of work.  The evaluation report should include the statement of work as an annex.  Evaluation methodology shall be explained in detail, and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides will be included in an Annex in the final report.  Evaluation findings will assess outcomes and impact on males and females.  Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.).  Evaluation findings should be presented as analyzed facts, evidence and data and not based on anecdotes, hearsay or the compilation of people's opinions. Findings should be specific, concise and supported by strong quantitative or qualitative evidence.  Sources of information shall be properly identified and listed in an annex.  Recommendations shall be supported by a specific set of findings.  Recommendations shall be action-oriented, practical and specific, with defined responsibility for the action. 7 8. OTHER REQUIREMENTS The evaluation team must be familiar with USAID's Human Subject Protection Policy and USAID's Evaluation Policy (http://www.usaid.gov/evaluation). The evaluation team must provide adequate training for its survey staff on survey methodology, USAID's survey regulations, other relevant regulations, and the data collection plan. The Contractor has the responsibility to safeguard the rights and welfare of human subjects involved in the survey research supported by USAID. USAID has adopted the Common Federal Policy for the Protection of Human Subjects, Part 225 of Title 22 of the Code of Federal Regulations (http://www.usaid.gov/policy/ads/200/200mbe.pdf). Recipient organizations must familiarize themselves with the USAID policy and provide "assurance" that they will follow and abide by the procedures of the Policy. 8 ANNEX B: LIST OF INTERVIEWEES 9 LIST OF MEETINGS AND FGD’S BY REAP EVALUATION TEAM CONTACT PERSON POSITION ORGANISATION LOCATION DATE Keti Chogovadze Program and Project Support Office, Program Development Specialist USAID US Mission,Tbilisi 29/09/15 Lela Kerashvili Program and Project Support Office, Program Development Specialist USAID US Mission,Tbilisi 29/09/15 Shamenna Gall Economic Growth Office, Agriculltural Development Officer (REAP COR) USAID US Mission,Tbilisi 29/09/15 Veronica Lee Economic Growth Office, Director USAID US Mission,Tbilisi 29/09/15 Nino Kumsishvili Economic Growth Office, Project Management Sppecialist USAID US Mission,Tbilisi 29/09/15 David Tsiklauri Economic Growth Office, Project Management Specialist USAID US Mission,Tbilisi 29/09/15 Howard H. Handler Program Officer USAID US Mission,Tbilisi 29/09/15 Michael Rossman Regional Contracting Officer USAID US Mission,Tbilisi 29/09/15 Nino Zambakhidze Chairwoman Georgian Farmers Association Tbilisi 30/09/15 Teo Urushadze Head of Department Agricultural University of Georgia Tbilisi 30/09/15 Dodi Malazonia Retail Segment Coordinator Bank Republic Tbilisi 30/09/15 Giorgi Gamkrelidze Micro and Small Business Segment Coordinator Bank Republic Tbilisi 30/09/15 Dimitri Dekanozishvil Product Development Officer Bank Republic Tbilisi 30/09/15 Sergi Baramidze Director Georgian Business Development Center (BSP) Tbilisi 30/09/15 10 LIST OF MEETINGS AND FGD’S BY REAP EVALUATION TEAM CONTACT PERSON POSITION ORGANISATION LOCATION DATE Mikheil Chelidze Executive President Georgian Small & medium Enterprises Association Tbilisi 01/10/15 Dr. Elguja Meladze President Georgian Employers Association Tbilisi 01/10/15 Mariam Rusishvili Head of Micro Business New Initiatives TCB Bank Tbilisi 02/10/15 Vakhtang Kopaleishvili Partner Stimor Associates (Non-Grantee) Tbilisi 02/10/15 Levan Davitashvili Deputy Minister of Agriculture Ministry of Agriculture Tbilisi 02/10/15 Khatia Tsilosani Head of International Relations Ministry of Agriculture Tbilisi 02/10/15 Irine Mekerishvili Senior Business Consultant Association of Business Consulting Organizations - Georgia Tbilisi 02/10/15 Zaza Gorozia Consultant in Agriculture Individual Consultant Tbilisi 02/10/15 Maia Tsereteli Executive Director Key Management Solutions Tbilisi 03/10/15 FGD with Failed Grant Applicants Telavi (REAP Office) 05/10/15 Izoldi Kitesashvili Member Cooperative Gile Gurjaani 05/10/15 Kakha Khutsishvili Grantee (Owner) Agrovita Gurjaani 05/10/15 Givi Meterveli Dep. Governor Kakheti Region Kakheti 05/10/15 Kakhi Mesablishvili Grantee (Owner) Agroservice Telavi 05/10/15 Natia Senior Specialist Ministry of Agriculture Telavi 05/10/15 Givi Mindiashvili Grantee (Owner) Mindia 2011 Ltd Kvareli 06/10/15 11 LIST OF MEETINGS AND FGD’S BY REAP EVALUATION TEAM CONTACT PERSON POSITION ORGANISATION LOCATION DATE Marina Akolashvili Owner Independent Enterprise (IE) Gurjaani 06/10/15 FGD with Failed Grant Applicants Telavi (REAP Office) 06/10/15 Ambrosi Macharashveli Grantee (Owner) Independent Enterprise (IE) Lagodekhi 06/10/15 Tamaz Naskidashvili Grantee (Owner) Luk-Agro Sighnaghi 07/10/15 Pridon Digmelashvili Grantee (Owner) Independent Enterprise (IE) Sagarejo 07/10/15 Mamuka Alpaidze Grantee (Owner) Farkoni Ltd Kutaisi 08/10/15 Zura Janelidze Grantee (Owner) Herbia Ltd Tskaltubo 08/10/15 FGD with Failed Grant Applicants Kutaisi (REAP Office) 08/10/15 FGD with Rejected Grant Applicants Kutaisi (REAP Office) 08/10/15 Mindia Khurtsidze Grantee (Owner) Geoflower Ltd Ambrolauri 09/10/15 Archil Elbakidze Grantee (Owner) Akhali Fermeri Ltd Ambrolauri 09/10/15 Merab Chitanava Grantee (Owner) Agrigeorgia (Ferrero) Kutaisi 09/10/15 Darejan Kanteladze Director Santa LTD Tsalka 09/10/15 Malkhaz Zoidze Director Orientali 2013 LTD Tsalka 09/10/15 FGD with Failed Grant Applicants Marneuli Centre for Civic 09/10/15 12 LIST OF MEETINGS AND FGD’S BY REAP EVALUATION TEAM CONTACT PERSON POSITION ORGANISATION LOCATION DATE Engagement Tamar Gugushvili Director Aromaproduct LTD Tbilisi 09/10/15 Koba Gvazava Grantee (Owner) Gvaza Ltd Senaki 10/10/15 Mamuka Tsikoridze Grantee (Owner) Independent Enterprise (IE) Tskaltubo 10/10/15 FGD with Failed Grant Applicants Zugdidi (REAP Office) 10/10/15 FGD with Rejected Grant Applicants Zugdidi (REAP Office) 10/10/15 Goga Simonishvili Director Agro Kartli LTD Gori 10/10/15 FGD with Rejected Grant Applicants Agro Kartli Ltd - Gori 10/10/15 Dimitri Dzirkbilashvili Director Eleniksta LTD Kareli 10/10/15 Goga Kvitinadze Director Georgian Fruit Company Cooperative Gori 10/10/15 Andro Khetereli Grantee (Owner) Agropharm Plus Ltd Ozurgeti / Anaseuli 12/10/15 Gia Khuchua Director Anaseuli Tea Factory Ozurgeti / Anaseuli 12/10/15 Aleko Mameshvili Project Co-ordinator Guria ABC (BSP) Ozurgeti / Anaseuli 12/10/15 Ioseb Erkomaishvili Dep. Governor Guria Region Ozurgeti 12/10/15 13 LIST OF MEETINGS AND FGD’S BY REAP EVALUATION TEAM CONTACT PERSON POSITION ORGANISATION LOCATION DATE Lika Glonti Regional Project Co-ordinator Guria Region Ozurgeti 12/10/15 Giorgi Stepniashvili Director I/E Giorgi Stepniashvili Mtskheta 12/10/15 FGD with Failed Grant Applicants Agro Kartli Ltd - Gori 12/10/15 Nugzar Papunashvili Director I/E Nugzar Papunashvili Shindisi 12/10/15 Avto Javakhishvili Director Iveria LTD Dzevera 12/10/15 Zaur Putkaradze Regional Minister of Agriculture GoG (Autonomous region) Batumi 13/10/15 MerabTavdgiridze Grantee (Owner) Farmer House Ltd Khulo 13/10/15 Gia Dzirkvadze Member Dignisi Co-operative Khulo 13/10/15 Rezo Gogoladze Director Zoreti LTD Borjomi 13/10/15 Beso Babunashvili Director GBZ LTD Akhaltsikhe 13/10/15 FGD with Rejected Grant Applicants Akhaltsikhe Centre for Civic Engagement 13/10/15 FGD with Failed Grant Applicants Akhaltsikhe Centre for Civic Engagement 13/10/15 Vaja Mkurnalidze Grantee (Owner) Agroinvestservice Ltd Kobuleti 14/10/15 Nona Tordia Grantee (Owner) Gemuani Ltd Zugdidi 14/10/15 Edisher Sanikidze Grantee (Owner) Gejeti Ltd Senaki 14/10/15 14 LIST OF MEETINGS AND FGD’S BY REAP EVALUATION TEAM CONTACT PERSON POSITION ORGANISATION LOCATION DATE Zurab Sadatierashvili Director Rural Advisory Service LTD Akhaltsikhe 14/10/15 Meruzhan Ezoyan Director I/E Seriozha Ezoyan Akhalkalaki 14/10/15 Armen Amirkhanian Director I/E Sosiko Amirkhanian Ninotsminda 14/10/15 Lasha Khonelidze Exec Director (TA Recipient) Georgian Mushrooms Tbilisi 15/10/15 Tintatin Chikvaldze Project Manager (TA Recipient) Olive Green Tbilisi 15/10/15 15 ANNEX C: LIST OF DOCUMENTS REVIEWED 16 REAP Documentation and Other Literature Name of Document Submission Date Annual Report:- Sept 13 – Sept 14 12 September 20 PMP – PY1 Not known Quarterly Report – Y1 Q1 (October – December 2013) 17 January 2014 Quarterly Report – Y1 Q2 (January – March 2014) 20 May 2014 Quarterly Report – Y1 Q3 (April – June 2014) 14 July 2014 Quarterly Report – Y! Q4 (July – September 2014) n/a Quarterly Report – Y2 Q1 (September – December 2014) 14 January 2015 Quarterly Report – Y2 Q2 (January – March 2015) 15 April 2015 Quarterly Report – Y2 Q3 (April – June 2015) n/a Quarterly Report – Y2 Q4 (July - September 2015) n/a Social & Behaviour Change Communication Strategy September 2012 Voluntary, Education and target Savings Product Development: Pub. Duterimbere IMF April 2013 Mapping the Financial Services Landscape in Rwanda: Pub. AMIR / REMC April 2012 Poverty Assessment Tools Survey: Pub. Research Moguls Ltd October 2013 Ejo Heza: Livelihood Baselines: Southern & Western Provinces Not Known Knowledge, Attitudes and Practices of USAID-EJO HEZA beneficiaries: Pub. Emmanuel Rigira August 2012 USAID/Ejo Heza Value Chain Development Strategy: Pub. Noel Ujeneza August 2012 USAID/Ejo Heza – Yr 13 Narrative Report December 2013 IILP 2014 Indicators Report Not Known Yr13 Indicator Report December 2013 Ministry of Agriculture – Annual 2014 n/a Ministry of Agriculture – Georgian Agro-Food Sector for your Investment n/a Ministry of Agriculture – Strategy for Agricultural Development in Georgia – 2015- 2020 n/a Agriculture Lending Training Status Report and recommendations – Daniel M. Gies September 2014 DCA Leasing Status and Recommendations – Daniel M. Gies September 2014 USAID/REAP Agriculture Lending Training Report November 2014 Deep and Comprehensive Free Trade Area Agreement (DCFTA) opportunities for agribusinesses: Needs and Recommendations June 2015 TBC Constanta Bank: Loan Product Development July 2015 BDS Market Analysis & Capacity Building Plan March 2014 Post Harvest and Training Needs Assessment June 2014 Integrating Cost Benefit Analysis (CBA) Indicators with the M & E Framework April 2014 17 ANNEX D: MINI-SURVEY QUESTIONNAIRES 18 REAP Primary Producers (PP) Questionnaire Questionnaire Number Interviewer code: Date of interview: |___|___| |___|___| Day Month Region Settlement Respondent is Grantee 1 – Yes ; 0 - No Application ID Business Legal Name Respondent information Name: Telephone: Respondent role in the enterprise 1. Owner 2. Shareholder 3. Employee 4. Family member/relative 5. Other (please specify) ______________________________________________ Interview Result 1. Completed 2. Refusal 3. Other (please specify) _________________________________ Other Notes: 19 Interview Start Time_________________ DEMOGRAPHICS Q1.[Don’t ask]Gender of Respondent Male 1 Female 2 Q2.Age of the respondent __________________ Q3.Education level of the respondent [Only one answer] No formal education 1 Kindergarten 2 Elementary school (4-5 classes) 3 Incomplete secondary (5-9 classes) 4 Secondary (10-12 classes including general education, lyceum, gymnasium) 5 Secondary vocational (technical or college) 6 Higher education diploma (Bachelor, Master) 7 Advanced higher education 8 (Don’t know) -1 (Refuse to answer) -2 Q4.Please Specify the Top 3 source/s for your income indicating the importance Q4a Q4b Yes No Do not Know Refuse to Answer Rank 1 Jobs (excluding agricultural activity) 1 0 -1 -2 2 Agricultural activities and goods 1 0 -1 -2 3 Business activity 1 0 -1 -2 4 Interest 1 0 -1 -2 5 Property income (e.g. income gained from leasing space, garage or tractor) 1 0 -1 -2 6 Relatives 1 0 -1 -2 7 Family member working abroad 1 0 -1 -2 20 8 Family member working in another city in Georgia 1 0 -1 -2 9 Inheritance 1 0 -1 -2 10 Gifts 1 0 -1 -2 11 Government aid, pensions, scholarship, etc 1 0 -1 -2 12 Aid from international organizations 1 0 -1 -2 13 Loan 1 0 -1 -2 14 Other (Specify _____________________) 1 0 -1 -2 BUSINESS SPECIFICATIONS Q5.Please specify the total land in hectares OR square meters that you use for your business? [INTERVIEWER! ONLY USE ONE MEASURE! 1 Hectare is 10,000 square meters. If the respondent says only full hectares, write 0000 in the Square Meters field. If the respondent only says Square Meters, write 0000 in the Hectares field. ] Hectares_____________________ Sq.M_______________________ Q6.How long has your business been operational? Less than 1 year 1 1 to 2 years 2 3 to 5 years 3 More than 5 years 4 Don’t know -1 Refuse -2 Q7.What type of agricultural business are you engaged in? (Check all that apply) Primary agricultural production 1 Process of agricultural production 2 Agricultural Service 3 Other (Please specify________________) 4 Refuse -2 Q8.Your business activities Products 8a Major activity 8b Other activity 8c Activity applied for grant Melons and gourds 1 Water melon 1 1 1 2 Melon 2 2 2 3 Pumpkin 3 3 3 Other gourds (please specify) 21 Products 8a Major activity 8b Other activity 8c Activity applied for grant Vegetables 4 Potatoes 4 4 4 5 Carrots 5 5 5 6 Beet-roots 6 6 6 7 Beans 7 7 7 8 Cucumbers 8 8 8 9 Tomatoes 9 9 9 10 Cabbage 10 10 10 11 Aubergine 11 11 11 Other vegetables (Please specify) Fruits 12 Apples 12 12 12 13 Pears 13 13 13 14 Plums 14 14 14 15 Cherry – plum (Alucha) 15 15 15 16 Tkemali 16 16 16 17 Cherries, black cherries 17 17 17 18 Peaches 18 18 18 19 Persimmon 19 19 19 20 Figs 20 20 20 21 Pomegranate 21 21 21 Other fruits (please specify) Nuts 22 Almond 22 22 22 23 Hazel-nut 23 23 23 24 Pea nut 24 24 24 25 Walnut 25 25 25 26 Chestnut 26 26 26 Other nuts (please specify) Citrus 27 Lemon 27 27 27 28 Orange 28 28 28 29 Tangerines 29 29 29 30 Kiwi 30 30 30 Other Citrus (please specify) Crop 31 Wheat 31 31 31 32 Maize 32 32 32 33 Sunflower 33 33 33 Other crop (please specify) Other Products 34 Cattle – meat 34 34 34 35 Cattle – dairy 35 35 35 36 Poultry – meat 36 36 36 37 Poultry – egg 37 37 37 38 Fish 38 38 38 39 Pork 39 39 39 40 Sheep 40 40 40 22 Products 8a Major activity 8b Other activity 8c Activity applied for grant 41 Goat 41 41 41 42 Honey production 42 42 42 43 Tea 43 43 43 44 Flower 44 44 44 Other agriculture production (please specify) _______________ _______________ ________________ 46 Process of agricultural production 45 45 45 47 Agricultural Service 46 46 46 Q9.Can you please tell me approximate turnover of your agribusiness for the following years: 2013 2014 2015 GEL Don’t know -1 -1 -1 Refuse -2 -2 -2 Did not operate -8 -8 -8 Q10. Can you please tell me approximate costs of your agribusiness for the following years: 2013 2014 2015 GEL Don’t know -1 -1 -1 Refuse -2 -2 -2 Did not operate -8 -8 -8 Q11. Payment of wages and salaries for the following years 2013 2014 2015 GEL Of which employees: male female male female male female permanent seasonal temporary Don’t know -1 -1 -1 Refuse -2 -2 -2 Did not operate -8 -8 -8 Q12. Can you please tell me approximate net income of your agribusiness for the following years: 2013 2014 2015 GEL Don’t know -1 -1 -1 Refuse -2 -2 -2 Did not operate -8 -8 -8 23 Q13. What kind of the following agricultural inputs and machinery do you require for your business activity (or activities)? (Check all that apply) Q14. Provider of the raw materials for each option selected in Q13 Q13 Q14 FSC Market Direct purchase from producer Distributor Other Refuse 1. Seeds 1 1 2 3 4 -2 2. Fertilizers 2 1 2 3 4 -2 3. Plant protection 3 1 2 3 4 -2 4. Veterinary medicines 4 1 2 3 4 -2 5. Feed materials/supplies 5 1 2 3 4 -2 6. Premixes 6 1 2 3 4 -2 7. Combined feed 7 1 2 3 4 -2 8. Seedlings/Plantings 8 1 2 3 4 -2 9. Livestock 9 1 2 3 4 -2 10. Fertilized eggs 10 1 2 3 4 -2 11. Fingerlings 11 1 2 3 4 -2 12. Deep frozen semen 12 1 2 3 4 -2 13. Machinery and equipment 13 1 2 3 4 -2 14. Irrigation system 14 1 2 3 4 -2 15. Fuel for Agriculture machinery 15 1 2 3 4 -2 16. Storage 16 1 2 3 4 -2 17. Consultation/training costs 18 1 2 3 4 -2 Other ___________________ 20 1 2 3 4 -2 Q15. What is the value of the produced products? 2013 2014 2015 Value Value Value 24 Major Activity 1 Secondary Activity 2 3 4 5 6 7 Q16. Distribution Channels for the produced products Products Wholesaler Collection point Farm market Processor Super market Other 2013 2014 2015 Value Value Value 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 Q17. Exported products (GEL) Products 2013 2014 2015 code Value Value Value 1 2 3 4 5 6 Q18. Number of delivery channels 1 wholesalers 2 Collection points 3 Farm market/s 4 Processor/s 25 5 Export to other countries 6 Other/s specify__________ 7 Refuse -2 Q19. Delivery Channel number Change compared to 2014 Increase 1 Decrease 2 No change 3 Q20. Please tell me how well do you manage to sell your products on the markets listed above I do not 1 Continue I managed to sell my products partially 2 I manage to sell my products completely 3 Go to Q22Error! Reference source not found. I manage to sell my products completely, but there’s higher demand (I can’t produce more) 4 Q21. You mentioned that you do not manage or just partially manage to sell the products on the market with profit for you. Can you tell me the reasons for that? (check all that apply) Transportation problems 1 Delivery terms (non-profitable intermediary prices, timeline, etc.) 2 Not sufficient amount of harvest 3 Unhealthy competition/monopoly 4 High registration tax(es) on the market 5 Low Demand 6 Insufficient quality 7 Imported products cost less 8 Other (specify)____________________ 9 Q22. What can be done in order to increase production, sell more products on market? Increase Production Increase Sales Access More Markets 1 Buy new equipment 1 2 3 2 Renovate the premises 1 2 3 3 Buy/rent additional land 1 2 3 4 Better access to get loans from financial institutions 1 2 3 5 Inputs 1 2 3 6 Technical assistance 1 2 3 26 7 irrigation 1 2 3 8 Other (please specify) 1 2 3 OTHER DONORS & GRANTS Q23. Have you participated in another donor/grant or loan program other than USAID REAP? Yes 1 Continue No 2 Go to Don’t Know/can’t remember -1 Q28 Q24. Did you receive Financing or assistance from that donor or loan program? Yes 1 Continue Yes, but not received yet 2 No 3 Go to Don’t Know/can’t remember -1 Q28 Q25. From which organization did or will you receive grant? 1 2 3 Q26. What type of financing or assistance was or will it be? (Check all that apply) Grant with matching contribution 1 Grant without matching contribution 2 Technical Assistance 3 Preferential credit or loan 4 Other (please specify_____________) 5 Q27. When did you receive the financing or assistance (From the donor, or loan program other than USAID REAP) YEAR & Month Monetary Aid Value 1 2 3 TECHNICAL ASSISTANCE RECIEVERS Q28. Did you receive technical assistance through REAP Project Yes 1 Continue No 2 Go to Don’t Know/can’t remember -1 Q33 Q29. Please tell me if you have participated in any of the named trainings. If yes, in which one? (check all that apply) Marketing 1 Agriculture modern technologies 2 International Standards implementation (ISO, HACEP) 3 27 Post-Harvest Processing 4 Business management 5 The training course in veterinary 6 Other (please specify) ______________________________ 7 Q30. How would you rate TA provided by REAP project? (cycle each row) 1 Impact on your business 1 – No Impact 2 3 4 5 – significant impact 2 Relevance 1 – non relevant at all 2 3 4 5 – relevant 3 Relevance of topics covered 1 – non relevant at all 2 3 4 5 – relevant 4 Duration of Training 1 – non sufficient 2 3 4 5 – sufficient 5 Other(specify) ____________________ 1 2 3 4 5 Q31. Please tell me if the acquired knowledge has helped you in increasing the productivity of your agricultural activities No change insignificantly significantly 1 Increase productivity 1 2 3 2 Increase sales 1 2 3 3 Increase export 1 2 3 4 Improve the quality 1 2 3 5 More access to markets 1 2 3 6 Reduce post-harvest loses 1 2 3 7 Employment increase 1 2 3 8 Other(Specify) _________________ 1 2 3 If respondents answered "3" on every question at Q31, skip question Q32 Q32. Could you tell me the reasons why you couldn’t manage to increase productivity or reduce post-harvest loses (check all that apply) 1 Can’t apply modern technologies due to the money required 1 2 Machinery not applicable or can’t afford 2 3 Not enough agriculture land 3 4 Not enough demand 4 5 Other(Specify) 5 ASK ONLY TO REAP GRANT RECIPIENTS Q33. Why did you apply for REAP grant?[check all that apply] To expand a pre-existing business 1 Continue To start up business 2 Go to Q35 To sustain the existing business 3 Other (specify) ________________________ 4 Don’t know -1 Refuse -2 28 Q34. For what type of expansion did you apply for? [Check all that apply] To improve enterprise infrastructure 1 To purchase machinery 2 To purchase transportation means 3 To purchase real estate 4 Other (specify) ________________________ 5 Q35. What source of funding did you use for your matching contribution? (check all that apply) Personal savings 1 Borrowing money from informal source money lender, pawn shop, etc 2 Borrowing money from informal source – relatives/friends, etc 3 Loan from Bank 4 Loan or grant from another source (another donor program, another lending organization) 5 Real estate 6 Other (Please specify) __________________ 7 Q36. In general how satisfied are you with the grant/process/handling? Scale (1 – very dissatisfied; 5 – very satisfied) Application submission Process 1 2 3 4 5 Questions answered in timely manner 1 2 3 4 5 Application Processing time 1 2 3 4 5 Grant Release process (ask only if Respondent is Grantee ) 1 2 3 4 5 Grant value (ask only if Respondent is Grantee ) 1 2 3 4 5 [Ask only grantees] How would you rate overall impact of the grant on your business? (1=no impact; 3=significant impact) 1 2 3 ADDITIONAL COMMENTS ____________________________________________________________________________________________ ____________________________________________________________________________________________ ____________________________________________________________________________________________ ____________________________________________________________________________________________ ____________________________________________________________________________________________ ____________________________________________________________________________________________ ____________________________________________________________________________________________ __________________ Interview end time ___________ THANK THE RESPONDENT 29 30 REAP Post-Harvest Processor (PHP) Questionnaire Interview Start Time_________________ Questionnaire Number Interviewer code: Date of interview: |___|___| |___|___| Day Month Region Settlement Respondent is Grantee 1 – Yes ; 0 - No Application ID Business Legal Name Respondent information Name: Telephone: Respondent role in the enterprise 1. Owner 2. Shareholder 3. Employee 4. Family member/relative 5. Other (please specify) ______________________________________________ Interview Result 4. Completed 5. Refusal 6. Other (please specify) _________________________________ Other Notes: 31 DEMOGRAPHICS Q1.[Don’t ask]Gender of Respondent Male 1 Female 2 Q2.Age of the respondent __________________ Q3.Education level of the respondent [Only one answer] No formal education 1 Kindergarten 2 Elementary school (4-5 classes) 3 Incomplete secondary (5-9 classes) 4 Secondary (10-12 classes including general education, lyceum, gymnasium) 5 Secondary vocational (technical or college) 6 Higher education diploma (Bachelor, Master) 7 Advanced higher education 8 (Don’t know) -1 (Refuse to answer) -2 Q4.Please Specify the Top 3 source/s for your income indicating the importance in Q4b Q4a Q4b Yes No Do not Know Refuse to Answer Rank 1 Jobs (excluding agricultural activity) 1 0 -1 -2 2 Agricultural activities and goods 1 0 -1 -2 3 Business activity 1 0 -1 -2 4 Interest 1 0 -1 -2 5 Property income 1 0 -1 -2 6 Relatives 1 0 -1 -2 7 Family member working abroad 1 0 -1 -2 8 Family member working in another city in Georgia 1 0 -1 -2 9 Inheritance 1 0 -1 -2 10 Gifts 1 0 -1 -2 11 Government aid, pensions, scholarship, etc 1 0 -1 -2 12 Aid from international organizations 1 0 -1 -2 13 Loan 1 0 -1 -2 14 Other (Specify _____________________) 1 0 -1 -2 32 BUSINESS SPECIFICATIONS Q5.Please specify the total land in hectares OR square meters that you use for your business? [INTERVIEWER! ONLY USE ONE MEASURE! 1 Hectare is 10,000 square meters. If the respondent says only full hectares, write 0000 in the Square Meters field. If the respondent only says Square Meters, write 0000 in the Hectares field. ] Hectares_____________________ Sq.M_______________________ Q6.How long has your business been operational? Less than 1 year 1 1 to 2 years 2 3 to 5 years 3 More than 5 years 4 Don’t know -1 Refuse -2 Q7.What type of agricultural business are you engaged in? (Check all that apply) Production 1 Product collection 2 Processing 3 Service 4 Other (Please specify________________) 5 Refuse -2 Q8.Your business activities Products 8a Major activity 8b Other activity 8c Activity applied for grant Melons and gourds 1 Water melon 1 1 1 2 Melon 2 2 2 3 Pumpkin 3 3 3 Other gourds (please specify) Vegetables 4 Potatoes 4 4 4 5 Carrots 5 5 5 6 Beet-roots 6 6 6 33 Products 8a Major activity 8b Other activity 8c Activity applied for grant 7 Beans 7 7 7 8 Cucumbers 8 8 8 9 Tomatoes 9 9 9 10 Cabbage 10 10 10 11 Aubergine 11 11 11 Other vegetables (Please specify) Fruits 12 Apples 12 12 12 13 Pears 13 13 13 14 Plums 14 14 14 15 Cherry – plum (Alucha) 15 15 15 16 Tkemali 16 16 16 17 Cherries, black cherries 17 17 17 18 Peaches 18 18 18 19 Persimmon 19 19 19 20 Figs 20 20 20 21 Pomegranate 21 21 21 Other fruits (please specify) Nuts 22 Almond 22 22 22 23 Hazel-nut 23 23 23 24 Pea nut 24 24 24 25 Walnut 25 25 25 26 Chestnut 26 26 26 Other nuts (please specify) Citrus 27 Lemon 27 27 27 28 Orange 28 28 28 29 Tangerines 29 29 29 30 Kiwi 30 30 30 Crop 31 Wheat 31 31 31 34 Products 8a Major activity 8b Other activity 8c Activity applied for grant 32 Maize 32 32 32 33 Sunflower 33 33 33 Other crop (please specify) Other Products 34 Cattle – meat 34 34 34 35 Cattle – dairy 35 35 35 36 Poultry – meat 36 36 36 37 Poultry – egg 37 37 37 38 Fish 38 38 38 39 Pork 39 39 39 40 Sheep 40 40 40 41 Goat 41 41 41 42 Honey production 42 42 42 43 Tea 43 43 43 44 Flower 44 44 44 Other agriculture production (please specify) _______________ _______________ ________________ 45 Product collection 45 45 45 46 Processing 46 46 46 47 Agricultural service provision 47 47 47 Q9.Please indicate the amount of Raw materials/Inputs/packaging business use according to the years. (Please use the following codes: “-2 “-refuse to answer; “-1” don’t know/can’t remember”; “-8” business doesn’t operate) 9a. before grant(ask only grantees) 9 b. Amount (GEL) Value in 2013 (GEL) Value in 2014 (GEL) Value in 2015 (GEL 1 Vegetable 1 2 Meat 2 3 Milk 3 4 Fruits 4 35 5 Nuts 5 6 Eggs 6 7 Crops/Cereals 7 8 Fish 8 9 Pork 9 10 Tea 10 11 Citrus 11 12 Honey 12 13 Berries 13 14 Herbs & Greens 14 15 Seeds 15 16 Fertilizers 16 17 Plant protection means 17 18 Veterinary medicines 18 19 Feed supplies, premixes, combined feed 19 20 Seedlings/Plantings 20 21 Livestock, fertilized eggs, fingerlings, deep-frozen semen etc. 21 22 Machinery and equipment 22 23 Irrigation systems 23 24 Packaging materials 24 25 Labels 25 26 Other (specify) ____________ 26 27 Other (specify) ____________ 27 28 Total 28 Q10. Number of suppliers 10a.Currently 10b. Before Grant/2014 PP FSC Total (including other types) Q11. What is the value of the produced products? 36 2013 2014 2015 Value Value Value Major Activity 1 Secondary Activity 2 3 4 5 6 7 [Note: If respondent doesn’t remember the exact amount of turnover, profit, cost and wages &salaries, ask them, that you will contact them later after he/she will check the exact numbers] Q12. Can you please tell me approximate turnover of your business for the following years: 2013 2014 2015 GEL Don’t know -1 -1 -1 Refuse -2 -2 -2 Did not operate -8 -8 -8 Q13. Can you please tell me approximate costs of your business for the following years: 2013 2014 2015 GEL Don’t know -1 -1 -1 Refuse -2 -2 -2 Did not operate -8 -8 -8 Q14. Payment of wages and salaries for the following years 2013 2014 2015 GEL Of which employees: male female male female male female permanent seasonal temporary 37 Don’t know -1 -1 -1 Refuse -2 -2 -2 Did not operate -8 -8 -8 Q15. Can you please tell me approximate net income of your business for the following years: 2013 2014 2015 GEL Don’t know -1 -1 -1 Refuse -2 -2 -2 Did not operate -8 -8 -8 Q16. Exported products (GEL) Products 2013 2014 2015 code Value Value Value 1 2 3 4 5 6 Q17. Number of distribution channels 1 Distributors 2 wholesalers 3 Stores 4 Supermarkets 5 Grocery 6 Export/Import company 7 Other/s specify__________ 8 Refuse -2 Q18. Distribution Channel number Change compared to 2014 38 Increase 1 Decrease 2 No change 3 Q19. Please tell me how well do you manage to sell/export your products using distribution channels listed above I do not 1 ContinueError! Reference source I managed to sell/export my products partially 2 not found. I manage to sell/export my products completely 3 Go to 21 I manage to sell my products completely, but there’s higher demand (I can’t produce more) 4 Q20. You mentioned that you do not manage or just partially manage to sell the products on the market with profit for you, or you don’t produce enough. Can you tell me the reasons for that? (check all that apply) Local Distribution Export Transportation problems 1 1 Delivery terms (non-profitable intermediary prices, timeline, etc.) 2 2 Not sufficient amount of raw materials 3 3 Unhealthy competition/monopoly 4 4 High registration tax(es) on the market 5 5 Low Demand 6 6 Insufficient quality 7 7 Imported products cost less 8 8 Non-compliance with standards 9 9 Other (specify)____________________ 10 10 Q21. What can be done in order to increase production, sell more products on market? [check all that apply] Increase Production Increase Sales Access More Markets Export 1 Buy new equipment 1 2 3 4 2 Renovate the premises 1 2 3 4 3 Buy/rent more land 1 2 3 4 4 Better access to get loans from financial institutions 1 2 3 4 39 5 Raw materials 1 2 3 4 6 Technical assistance 1 2 3 4 7 irrigation 1 2 3 4 8 Other (please specify) 1 2 3 4 OTHER DONORS & GRANTS Q22. Have you participated in another donor/grant or loan program other than USAID REAP? Yes 1 Continue No 2 Go to Q27 Don’t Know/can’t remember -1 Q23. Did you receive Financing or assistance from that donor or loan program? Yes 1 Continue Yes, but not received yet 2 No 3 Go to Q27 Don’t Know/can’t remember -1 Q24. From which organization did or will you receive grant? 1 2 3 Q25. What type of financing or assistance was or will it be? (Check all that apply) Grant with matching contribution 1 Grant without matching contribution 2 Technical Assistance 3 Preferential credit or loan 4 Other (please specify_____________) 5 Q26. When did you receive the financing or assistance (From the donor, or loan program other than USAID REAP) YEAR & Month Monetary Aid Value 1 2 40 3 TECHNICAL ASSISTANCE RECIEVERS Q27. Did you receive technical assistance through REAP Project Yes 1 Continue No 2 Go to Q332 Don’t Know/can’t remember -1 Q28. Please tell me if you have participated in any of the named trainings. If yes, in which one? (check all that apply) Marketing 1 Agriculture modern technologies 2 International Standards implementation (ISO, HACEP) 3 Post-Harvest Processing 4 Business management 5 Veterinary trainings 6 Other (please specify) ______________________________ 7 Q29. How would you rate TA provided by REAP project? (cycle each row) 1 Impact on your business 1 – No Impact 2 3 4 5 – significant impact 2 Relevance 1 – non relevant at all 2 3 4 5 – relevant 3 Relevance of topics covered 1 – non relevant at all 2 3 4 5 – relevant 4 Duration of Training 1 – non sufficient 2 3 4 5 – sufficient 5 Other(specify) ____________________ 1 – non sufficient 2 3 4 5 – sufficient Q30. Please tell me if the acquired knowledge has helped you in increasing the productivity of your agricultural activities No change insignificantly significantly 1 Increase productivity 1 2 3 2 Increase sales 1 2 3 3 Export increased 4 Improve the quality 1 2 3 5 More access to markets 1 2 3 41 6 Employment increase 1 2 3 7 Other(Specify) _________________ 1 2 3 Q31. Could you tell me the reasons why you couldn’t manage to increase productivity or reduce post-harvest loses (check all that apply) 1 Can’t apply modern technologies due to the money required 1 2 Machinery not applicable or can’t afford 2 3 Not enough agriculture land 3 4 Not enough demand 4 5 Other(Specify) 5 Q32. Why did you apply for REAP grant?[check all that apply] To expand a pre-existing business 1 Continue To start up business 2 Go to Q34 To sustain the existing business 3 Other (specify) ________________________ 4 Don’t know -1 Refuse -2 Q33. For what type of expansion did you apply for? To improve enterprise infrastructure 1 To purchase machinery 2 To purchase transportation means 3 To purchase real estate 4 Other (specify) ________________________ 5 Q34. What source of funding did you use for your matching contribution? (check all that apply) Personal savings 1 Borrowing money from informal source money lender, pawn shop, etc 2 Borrowing money from informal source – relatives/friends, etc 3 Loan from Bank 4 Loan or grant from another source (another donor program, another lending organization) 5 42 Real estate 6 Other (Please specify) __________________ 7 Q35. In general how satisfied are you with the grant/process/handling? Scale (1 – very dissatisfied; 5 – very satisfied) Application submission Process 1 2 3 4 5 Questions answered in timely manner 1 2 3 4 5 Application Processing time 1 2 3 4 5 Grant Release process (Ask only grant recipients) 1 2 3 4 5 Grant value (Ask only grant recipients) 1 2 3 4 5 ASK ONLY TO REAP GRANT RECIPIENTS How would you rate overall impact of the grant on your business? (1=no impact; 3=significant impact) 1 2 3 ADDITIONAL COMMENTS ____________________________________________________________________________________________ ____________________________________________________________________________________________ ____________________________________________________________________________________________ ____________________________________________________________________________________________ ____________________________________________________________________________________________ ____________________________________________________________________________________________ ________________________________________________________________ Interview end time ___________ THANK THE RESPONDENT 43 REAP Farm Service Centers (FSC) Questionnaire Intervie w Start Time__ _______ _______ _ DEMO GRAPH ICS Q1.[ D o n ’ t a s k ] G e n d e r o f R e s p o n d e n t Questionnaire Number Interviewer code: Date of interview: |___|___| |___|___| Day Month Region Settlement Respondent is Grantee 1 – Yes ; 0 - No Application ID Business Legal Name Respondent information Name: Telephone: Respondent role in the enterprise 1. Owner 2. Shareholder 3. Employee 4. Family member/relative 5. Other (please specify) ______________________________________________ Interview Result 7. Completed 8. Refusal 9. Other (please specify) _________________________________ Other Notes: 44 Male 1 Female 2 Q2.Age of the respondent __________________ Q3.Education level of the respondent [Only one answer] No formal education 1 Kindergarten 2 Elementary school (4-5 classes) 3 Incomplete secondary (5-9 classes) 4 Secondary (10-12 classes including general education, lyceum, gymnasium) 5 Secondary vocational (technical or college) 6 Higher education diploma (Bachelor, Master) 7 Advanced higher education 8 (Don’t know) -1 (Refuse to answer) -2 Q4.Please Specify the Top 3 source/s for your income indicating the importance Q4a Q4b Yes No Do not Know Refuse to Answer Rank 1 Jobs (excluding agricultural activity) 1 0 -1 -2 2 Agricultural activities and goods 1 0 -1 -2 3 Business activity 1 0 -1 -2 4 Interest 1 0 -1 -2 5 Property income (e.g. income gained from leasing space, garage or tractor) 1 0 -1 -2 6 Relatives 1 0 -1 -2 7 Family member working abroad 1 0 -1 -2 8 Family member working in another city in Georgia 1 0 -1 -2 9 Inheritance 1 0 -1 -2 10 Gifts 1 0 -1 -2 11 Government aid, pensions, scholarship, etc 1 0 -1 -2 12 Aid from international organizations 1 0 -1 -2 45 13 Loan 1 0 -1 -2 14 Other (Specify _____________________) 1 0 -1 -2 BUSINESS SPECIFICATIONS Q5.Please specify the total land in hectares OR square meters that you use for your business? [INTERVIEWER! ONLY USE ONE MEASURE! 1 Hectare is 10,000 square meters. If the respondent says only full hectares, write 0000 in the Square Meters field. If the respondent only says Square Meters, write 0000 in the Hectares field. ] Hectares_____________________ Sq.M_______________________ Q6.How long has your business been operational? Less than 1 year 1 1 to 2 years 2 3 to 5 years 3 More than 5 years 4 Don’t know -1 Refuse -2 Q7.What type of agricultural business are you engaged in? (Check all that apply) Primary agricultural production 1 Process of agricultural production 2 Agricultural Service 3 Other (Please specify________________) 4 Refuse -2 Q8.Please indicate your company's product or service provided for customers and their total value by years. [Check 8a only if the respondent is grantee and the product / service sales began after receiving the grant. 8b filled in any case] 8a. Provided before grant (ask to grantees only) 8b. Value in GEL 2013 Year 2014 Year 2015 Year 1 Chemicals 1 2 Fertilizers 2 3 Seeds 3 4 Equipment/small tools 4 5 Machinery 5 6 Consulting 6 7 Veterinary service 7 8 Veterinary medicines 8 9 Other (Specify) _______________ 9 Q9.Number of customers (write “-2” refuse; “-1” don’t know/can’t remember; “-8” if business was not operational) 2013 2014 2014 46 PP PHP Total (including other types) Q10. Can you please tell me approximate turnover of your business for the following years: 2013 2014 2015 GEL Don’t know -1 -1 -1 Refuse -2 -2 -2 Did not operate -8 -8 -8 Q11. Can you please tell me approximate costs of your business for the following years: 2013 2014 2015 GEL Don’t know -1 -1 -1 Refuse -2 -2 -2 Did not operate -8 -8 -8 Q12. Payment of wages and salaries for the following years 2013 2014 2015 GEL Of which employees: male female male female male female permanent seasonal temporary Don’t know -1 -1 -1 Refuse -2 -2 -2 Did not operate -8 -8 -8 Q13. Can you please tell me approximate net income of your business for the following years: 2013 2014 2015 GEL Don’t know -1 -1 -1 Refuse -2 -2 -2 Did not operate -8 -8 -8 Q14. How would you rate the demand on the products/services you provide from the local farmers/businesses No demand 1 ContinueError! Reference source not found. Poor Demand 2 Moderate Demand 3 High Demand 4 Go to Q17 Q15. You mentioned that there is no or moderate demand for your products. Can you tell me the reasons for that? (check all that apply) 47 Farmers Can’t afford the products offered (seeds, fertilizers etc) 1 Farmers Can’t afford the services offered (machinery, consulting etc) 2 Machinery not needed, most of the farmers have small land 3 Unaware of such type of service (uninformed) 4 Location isn’t accessible for all 5 Other (specify)____________________ 6 Q16. What can be done in order to increase the demand for the services provided by FSCs? Increase Production 1 Add more machinery types 1 2 Renovate the premises 2 3 Offer more type of products 3 4 Other (please specify) 4 OTHER DONORS & GRANTS Q17. Have you participated in another donor/grant or loan program other than USAID REAP? Yes 1 Continue No 2 Go to Don’t Know/can’t remember -1 Q28 Q18. Did you receive Financing or assistance from that donor or loan program? Yes 1 Continue Yes, but not received yet 2 No 3 Go to Don’t Know/can’t remember -1 Q28 Q19. From which organization did or will you receive grant? 1 2 3 Q20. What type of financing or assistance was or will it be? (Check all that apply) Grant with matching contribution 1 Grant without matching contribution 2 Technical Assistance 3 Preferential credit or loan 4 Other (please specify_____________) 5 Q21. When did you receive the financing or assistance (From the donor, or loan program other than USAID REAP) 48 YEAR & Month Monetary Aid Value 1 2 3 TECHNICAL ASSISTANCE RECIEVERS Q22. Did you receive technical assistance through REAP Project Yes 1 Continue No 2 Go to Don’t Know/can’t remember -1 Q33 Q23. Please tell me if you have participated in any of the named trainings. If yes, in which one? (check all that apply) Marketing 1 Agriculture modern technologies 2 International Standards implementation (ISO, HACEP) 3 Post-Harvest Processing 4 Business management 5 Veterinary training 6 Other (please specify) ______________________________ 7 Q24. How would you rate TA provided by REAP project? (cycle each row) 1 Impact on your business 1 – No Impact 2 3 4 5 – significant impact 2 Relevance 1 – non relevant at all 2 3 4 5 – relevant 3 Relevance of topics covered 1 – non relevant at all 2 3 4 5 – relevant 4 Duration of Training 1 – non sufficient 2 3 4 5 – sufficient 5 Other(specify) ____________________ 1 2 3 4 5 Q25. Please tell me if the acquired knowledge has helped you in increasing the productivity of your agricultural activities No change insignificantly significantly 1 Increase productivity 1 2 3 2 Increase sales 1 2 3 3 Improve the quality 1 2 3 4 More access to markets 1 2 3 5 Employment increase 1 2 3 6 Other(Specify) _________________ 1 2 3 Q26. Could you tell me the reasons why you couldn’t manage to increase productivity or reduce post-harvest loses (check all that apply) 1 Can’t apply modern technologies due to the money required 1 2 Machinery not applicable or can’t afford 2 49 3 Not enough agriculture land 3 4 Not enough demand 4 5 Other(Specify) 5 Q27. Why did you apply for REAP grant?[check all that apply] To expand a pre-existing business 1 Continue To start up business 2 Go to Q35 To sustain the existing business 3 Other (specify) ________________________ 4 Don’t know -1 Refuse -2 Q28. For what type of expansion did you apply for? To improve enterprise infrastructure 1 To purchase machinery 2 To purchase transportation means 3 To purchase real estate 4 Other (specify) ________________________ 5 Q29. What source of funding did you use for your matching contribution? (check all that apply) Personal savings 1 Borrowing money from informal source money lender, pawn shop, etc 2 Borrowing money from informal source – relatives/friends, etc 3 Loan from Bank 4 Loan or grant from another source (another donor program, another lending organization) 5 Real estate 6 Other (Please specify) __________________ 7 Q30. In general how satisfied are you with the grant/process/handling? Scale (1 – very dissatisfied; 5 – very satisfied) Application submission Process 1 2 3 4 5 Questions answered in timely manner 1 2 3 4 5 Application Processing time 1 2 3 4 5 Grant Release process (ask only if Respondent is Grantee ) 1 2 3 4 5 Grant value (ask only if Respondent is Grantee ) 1 2 3 4 5 [Ask only grantees] How would you rate overall impact of the grant on your business? (1=no impact; 3=significant impact) 1 2 3 ADDITIONAL COMMENTS 50 ____________________________________________________________________________________________ ____________________________________________________________________________________________ ____________________________________________________________________________________________ ____________________________________________________________________________________________ ____________________________________________________________________________________________ ____________________________________________________________________________________________ ____________________________________________________________________________________________ __________________ Interview end time ___________ THANK THE RESPONDENT 51 ANNEX E – EVALUATION RESULTS CHART 52 6 2 214,400 55,050 159,350 142,442 16,908 4 18 43 61  34 12 11,204,391 2,634,325 8,570,066 5,862,314 2,707,752 21 98 248 346  1 0 151,585 64,245 87,340 11,340 76,000 1 4 04  70 17 23,620,358 6,000,000.00 17,620,358.00 12,836,059 4,784,299.00 38 Beneficiary Category Results Chart ‐ Evaluation Team Total Project Target Number HM L TOTAL 40 712 FSCs / MSCs 29 16 3 12,049,982 3,246,380.00 8,803,602.00 1,983,639 12 94 207  Post-Harvest Handling Enterprises 20 Information Service providers 1 Grantees Primary Producers 3 6,819,963 Jobs Created FST 301 Client Contribution (Bank Loan)* Total Number of Clients Never Received Grants Before Total Number Met Total Number of Women Business es Matching Grant Amount REAP’s Contribution Level of Sustainability (Probability)** Client Contribution ( In cash and equity) Client Contribution 53 ANNEX F- EVALUATION WORK PLAN 54 WORK PLAN & EVALUATION DESIGN For: Mid-term Performance Evaluation of the Restoring Efficiency to Agricultural Production (REAP) Project 25 September 2015 Submitted by: Error! Objects cannot be created from editing field codes. Mendez England & Associates 4300 Montgomery Ave., Suite 103 Bethesda, MD 20814-4413 Contact Person: Thomas C. England, President Tel: (301) 652-4334  Fax: (301) 652-3733  Email: tengland@engl.com 55 1. INTRODUCTION In the preparation of the Work Plan and Evaluation Design for the mid-term performance evaluation of the Restoring Efficiency to Agriculture Production (REAP), the Evaluation Team has followed the guidelines outlined in USAID’s Request for Task Order Proposal (RFTOP) and accompanying Statement of Work (SOW) (Annex 1), together with ME&A’s Technical Proposal In overall terms, according to the SOW, the purpose of this evaluation is to “determine the relevance of REAP’s programmatic approaches in achieving intended life-of-the project results and to provide recommendations on corrective actions and new directions for the remaining years of project implementation and beyond”. In determining REAP’s effectiveness, the Evaluation Team will keep in mind that the project is designed to have long-term benefits, some of which may have not been fully realized given that this is a mid-term evaluation and that social and attitudinal changes are long-term processes. The evaluation will answer a specific set of evaluation questions posed by USAID/Georgia as outlined in Annex 3 and will cover the period from September 2013 to September 2015 of REAP’s activities. 2. EVALUATION TEAM The evaluation of REAP will be conducted by a team that consists of two international experts, Mr. Colin Maclean (Team Leader) and Dr. Gary Woller (Evaluation Expert), as well as a Local Expert, Dr. Gaga Nikabadze. Mr. Maclean will assume overall responsibility for the management of the evaluation in collaboration with USAID/Georgia. In short, this encompasses all activities specified in the Evaluation Schedule (Annex 2), including pre-mobilization, on-site implementation and end-of-assignment deliverables. Dr. Woller and Dr. Nikabadze will contribute to the evaluation mission by assisting in scheduling meetings, preparing documents, conducting small sample surveys and key informant interviews, supporting the organization of focus group discussions (FGDs), and carrying out observations, site visits and additional research as identified by the Team Leader. They will further participate in the planning and implementation of regional field visits (where appropriate) as well as contributing towards the preparation of the draft and final evaluation reports. The above team will be supported by the local organization IRMS which will be responsible for conducting a survey of grantee and non-grantee businesses as well as technical assistance (TA) recipients. Finally, oversight of the evaluation mission will fall under the remit of Ms. Mirela McDonald, Evaluation IQC Manager with ME&A. 3. EVALUATION TASKS and SUB-TASKS 3.1 Pre-Mobilization Activities Communication A number of email exchanges have taken place since the Task Order was signed among the Evaluation Team members, ME&A Project Manager and Project Coordinator, and USAID staff in order to coordinate the mobilization activities, in general, and to obtain relevant documents, clarify project and evaluation approaches, identify informants, and begin scheduling meetings, in particular. 56 Desk Study Prior to mobilization, the Evaluation Team received a number of important REAP project-related documents from USAID, including Work Plans, Annual Reports, Quarterly Reports, the original Request for Applications (RFA), Contract No. AID-114-C-13-00002 signed between USAID Caucasus and CNFA, as well as subsequent contract modifications. All documents provided to the Evaluation Team have been reviewed and used as a source of reference in preparation of this Work Plan. Contact with REAP Project During the week commencing Monday, 21 September, 2015, the Team Leader established communication with Mr. Louis Faoro, Chief of Party (COP) of REAP, during which a number of important and relevant issues were raised that would serve to assist the Evaluation Team during its in￾country investigations. The Team Leader also confirmed that the Evaluation Team would meet with the COP and his colleagues immediately following the Team’s in-briefing with USAID in Tbilisi scheduled for Tuesday, 29 September, 2015. Draft Work Plan and Evaluation Design A Draft Work Plan and Evaluation Design (this document) was prepared in collaboration with ME&A staff. Following the in-briefing with USAID in Tbilisi, the Team Leader will revise and/or edit the Work Plan and Evaluation Design to accommodate any corrections or adjustments requested by USAID resulting from that meeting. 3.2 In-Country Activities Initial Meetings The Evaluation Team will meet on Monday, 28 September, 2015, to discuss the draft Work Plan and each team member’s responsibilities. As mentioned above, the Evaluation Team has scheduled a preliminary meeting with key members of REAP staff on Tuesday, 29 September 2015, immediately following the Team’s in-brief with USAID/Georgia, in order for both parties to confirm their understanding of the purpose of the evaluation together with its expectations. These discussions will provide the opportunity to clarify the proposed activities of the Evaluation Team and, at the same time, contribute towards refining the list of interviewees and planning of the evaluation schedule as outlined in the Work Plan. Data Collection As highlighted in the Evaluation Schedule (see Annex 2), immediately following the in-briefing, the Evaluation Team will begin carrying out interviews with REAP’s staff, partners, stakeholders and beneficiaries, as well as a cross-section of other relevant individuals and organizations with an interest in REAP’s activities and outcomes. The Team’s investigations will initially focus on Tbilisi and then rolled out to the regions, municipalities, and districts nationwide. This will continue until Friday, 16 October 2015, following which work will begin on preparing a summary of findings, conclusions and recommendations that will be presented to USAID at an out-briefing on Monday, 19 October, 2015. A preliminary Draft Report will subsequently be submitted to USAID/Georgia on Wednesday, 4 November, 2015. Due to the extent of REAP’s activities, the Evaluation Team has determined that in addition to face-to￾face meetings with identified stakeholders in Tbilisi and in the regions, the most effective data-gathering approach will need to encompass a range of diverse data gathering methods if USAID’s proposed questions are to be satisfactorily answered. These methods will include Focus Group Discussions (FGDs) and a mini-survey. 57 FGDs will be conducted in the municipalities (district locations to be discussed and agreed with the REAP team) with the purpose of obtaining feedback from the project’s main beneficiaries. The precise number of FGDs will be determined following discussion at the USAID in-briefing and subsequent meeting with the entire REAP team shortly thereafter. For both face-to-face interviews and FGDs, specifically designed questionnaires will be prepared (see Annex 3 for examples) to be used as aide￾memoires for note-taking and later reference when preparing the draft Final Report. Specific approaches for both qualitative and quantitative data gathering and analysis, as well as methodology for the evaluation are outlined in Section 5. Mini-survey. Given the time constraints and relatively small population size of agribusinesses who applied for grants (230), the Evaluation Team will also conduct a mini-survey which will cover both grantees and non-grantees (those who applied but failed). The mini-survey will focus primarily on process issues, but also seek to estimate the effect of receiving a grant (the intervention). Data Analysis Throughout the in-country period of the evaluation, the Evaluation Team will meet regularly to discuss the outcome of each day’s activities. As members will be attending some meetings individually, it is essential that the Team meet regularly to discuss, digest and analyze data and information gathered during the key informant interviews, FGDs and site visits. Quantitative data collected will be analyzed using established evaluation techniques and industry standard data analysis tools. For qualitative data resulting from stakeholder interviews, where much of the evidence may be anecdotal or inferred, the Team will use triangulation to identify any inconsistencies and ensure reliability. Triangulation will assist the Evaluation Team to reduce the “response bias” in which respondents tend to tell the evaluators what they want to hear. The majority of data analysis will take place in the final days that the Evaluation Team is in Georgia (14-17 October), in preparation for the out-briefing with USAID/Georgia. 3.3 End-of-Evaluation Activities Out-briefing On Monday, 19 October, 2015, the Evaluation Team will conduct an out-briefing with USAID/Georgia during which initial preliminary findings will be presented in summary format together with key issues arising from the evaluation. This will take the form of a formal PowerPoint presentation supplemented by briefing notes for reference purposes. At this stage, comments and/or suggestions offered by USAID/Georgia will be acknowledged and addressed in the draft Final Report. Completion and Submission of Draft Report to USAID On completion of the in-country mission and following the Team Leader’s return to home base, a Draft Final Report will be prepared and submitted to USAID/Georgia on Wednesday, 4 November, 2015. Final Report submitted to USAID with integrated comments By Wednesday, 11 November, 2015, the Team Leader will receive the comments of USAID/Georgia for integration into the Final Report, which will subsequently be re-submitted to USAID/Georgia no later than Friday, 20 March, 2015. 4. EVALUATION DESIGN MATRIX In preparing the evaluation design, the Evaluation Team took into consideration the various REAP￾related documents forwarded by USAID and the project COP. Following an in-depth review of this material and considering the nature of the evaluation questions, the Evaluation Team prepared the 58 following design matrix depicting each evaluation question and sub-questions. Detailed analysis of each of these questions will be presented in the main body of the Final Report. For reference purposes and for ease of understanding, the first column highlights the component(s) to which each sub-question can be attributed. 59 Research Questions Type of Analysis Data Source Data Collection Method & Sample Size Limitations 1. Are REAP’s approaches necessary and sufficient for developing competitive and commercially sustainable agribusinesses and increasing incomes and employment in rural areas in Georgia?  See sub-questions 1a – 1e for detailed information 1a COMPONENT 1 & 2  What are the major constraints/challenges/ outstanding needs that inhibit the project’s contribution toward achieving the results during the remaining term of the project?  Description – based on content analysis of expert opinions & review of relevant documentation  Project documentation: weekly and quarterly reports, M&E plan, results framework, work plans, sub-grant documentation (solicitations, applications, selection memos, etc.)  Partner documentation Project Appraisal Document  Project staff: Partner NGO staff; REAP grantees (MSCs/FSCs, primary producers, post-harvest handling enterprises, information services providers, processors); unsuccessful grant applicants; REAP’s TA and training recipients; other SMEs and agribusinesses; farmers, Partner NGOs, Ministry of Agriculture  To be submitted by REAP for review by ET  To be submitted by REAP for review by ET  KIIs with individuals / organizations in agreement with REAP and USAID  FGDs in the regions (to be decided) with representatives of all REAP beneficiary groups  None envisaged  None envisaged  No specific limitations identified at this stage other than availability considerations of potential interviewees  No specific limitations envisaged other than availability of targeted attendees 1b  Is technical and/or  Description – based  Project documentation: weekly  To be submitted by  None envisaged 60 COMPONENT 2 business training provided to assisted primary producers and SMEs sufficient to improve skills, yields, and incomes of farmers? on content analysis of expert opinions & review of relevant documentation and quarterly reports, M&E plan, results framework, work plans, sub-grant documentation (solicitations, applications, selection memos, etc.)  Feedback from REAP primary producers and SMEs REAP for review by ET  KIIs with selected enterprises to be agreed with REAP  FGDs with selected participants from both categories. REAP to assist in selection of representative sample of attendees including women (8-12 participants)  If interviews - no specific limitations expected other than availability of potential interviewees or  If FGDs – availability / willingness of targeted individuals to attend 1c COMPONENT 2  Is the provided and intended TA specific and tailor made to each of the grantees needs? Are the quality, topics, and instructors of the training appropriate for Georgian agriculture, as defined by REAP beneficiaries?  Description – based on content analysis of expert opinions & review of relevant documentation  Project documentation: weekly and quarterly reports, M&E plan, results framework, work plans, sub-grant documentation (solicitations, applications, selection memos, etc.)  Feedback from beneficiary grantees  To be submitted by REAP for review by ET  FGDs in the regions – all 10 targeted. REAP to assist in selection of representative sample of participant grantees (8-12 participants)  None envisaged  No specific limitations envisaged other than availability of targeted attendees 1d COMPONENT 1  Are the grantees that are selected a good and fair representation to sustainably increase the availability and utilization of agricultural inputs, services and technical information among small-holder and emerging commercial farmers? Were the  Description – based on content analysis of expert opinions & review of relevant documentation  Project documentation: weekly and quarterly reports, M&E plan, results framework, work plans, sub-grant documentation (solicitations, applications, selection memos, etc.)  Feedback from beneficiary grantees  To be submitted by REAP for review by ET  FGDs in the regions – all 10 targeted. REAP to assist in selection of representative sample of participant grantees (8-12 participants)  None envisaged  No specific limitations envisaged other than availability of targeted attendees 61 requirements for grantees selected in a fair and transparent manner as described by initial project design documentation? Was the grant selection process fair and transparent? 1e COMPONENT 1 & 2  Has the program been successful in integrating the cross-cutting issues (gender, environment, workforce development, and access to finance) into the program implementation, when appropriate?  Description – based on content analysis of expert opinions & review of relevant documentation  Project documentation: weekly and quarterly reports, M&E plan, results framework, work plans  Project staff, REAP grantees, REAP TA/training recipients, unsuccessful grant applicants, Ministry of Agriculture, and other stakeholders related to agriculture (e.g. Georgian SMEs, agribusinesses, primary producers, post-harvest handling enterprises, etc.)  To be submitted by REAP for review by ET  KIIs with representatives of all categories of direct / indirect beneficiaries  None envisaged  If interviews - no specific limitations expected other than availability of potential interviewees 2. How is the program perceived by beneficiaries? In this case the program can be defined as the activities, the actors, the results, etc. Beneficiaries include grantees, non￾grantees, TA/grant-  Description – based on content analysis of expert opinions & review of relevant documentation  Project documentation: weekly and quarterly reports, M&E plan, results framework, work plans, sub-grant documentation (solicitations, applications, selection memos, etc.)  Partner documentation Project Appraisal Document  To be submitted by REAP for review by ET  To be submitted by REAP for review by ET  None envisaged  None envisaged 62 COMPONENT 1 & 2 recipients, Ministry of Agriculture, and other stakeholders related to agriculture (e.g. Georgian SMEs and agribusinesses, including primary producers, post-harvest handling enterprises, information service providers, processors, etc.) Specific sub-questions may include but not be limited to: how the GOG and other sector stakeholders perceive the program and its impact to date; how the grantees feel about the grant selection process; and what grantee applicants.  Project staff: Partner NGO staff; REAP grantees (MSCs/FSCs, primary producers, post-harvest handling enterprises, information services providers, processors); unsuccessful grant applicants; REAP’s TA and training recipients; other SMEs and agribusinesses; farmers, Partner NGOs, Ministry of Agriculture  KIIs with representatives of all categories of direct / indirect beneficiaries  Mini-survey conducted to a representative sample of the targeted 36, 000 distinct clients. Design and implementation the survey (format and structure to be decided) targeted at pre-determined sample population.  No specific limitations identified at this stage other than availability considerations of potential interviewees  Time constraint considerations to gather and analyze results within the mission timeframe. 3. COMPONENT 1 & 2 What, if any, corrections are needed to REAP’s implementation approaches during the remaining term of the project? Based on findings and conclusions related to questions #1 and #2, the evaluation team must make specific recommendations with regard to REAPs implementation  Description – based on content analysis of expert opinions & review of relevant documentation  Project documentation: weekly and quarterly reports, M&E plan, results framework, work plans, sub-grant documentation (solicitations, applications, selection memos, etc.)  Partner documentation Project Appraisal Document  Project staff: Partner NGO staff; REAP grantees (MSCs/FSCs, primary producers, post-harvest handling enterprises,  To be submitted by REAP for review by ET  To be submitted by REAP for review by ET  KIIs with representatives of all categories of direct / indirect beneficiaries  None envisaged  None envisaged  No specific limitations identified at this stage other than availability considerations of potential interviewees 63 approaches. For example, the team may identify opportunities to improve the implementation of future similar grant programs, or suggest additional interventions to advance REAP’s objectives. information services providers, processors); unsuccessful grant applicants; REAP’s TA and training recipients; other SMEs and agribusinesses; farmers, Partner NGOs, Ministry of Agriculture  Review and analysis of mini- survey results to inform recommendations  None envisaged subject to the mini-survey encountering no problems or difficulties during its implementation. 64 5. METHODOLOGY The following approach further elaborates on how the Evaluation Team envisages tackling the entire evaluation process. 5.1 Quantitative Research and Analysis 5.1.1 General Approach Quantitative data (e.g. number of actions / activities to be accomplished against plan) will be sourced from REAP’s Annual Work Plans, Performance Management Plans24 (PMPs) and other project-related periodic reports. The collective outcome of this evaluation will be a thorough assessment of the performance of REAP for the period under scrutiny25 relative to the planned objectives as envisaged in its original contract and elaborated and/or revised upon in subsequent modifications / amendments. In this instance, as the project was tasked with accomplishing certain target indicators, verification of performance from a statistical point of view can essentially only focus on a review of REAP’s records which the Evaluation Team will assume accurately reflect whether a numeric indicator was achieved or not. The findings from document review will be cross-referenced with findings accumulated from our qualitative research approach to determine the extent to which evidence gathered contributes towards the Evaluation Team’s understanding of REAP’s impact on its intended beneficiaries. This, in turn, will enhance the Team’s prospect of being able to provide satisfactory and meaningful answers to USAID’s pre-determined evaluation questions outlined in the Evaluation Design Matrix, above. Where appropriate, the Evaluation Team will prepare relevant charts summarizing the outcome(s) of collective responses to questionnaires, aimed at shedding light on whether action in the field reflects the quantifiable data reported by the project and represents real progress or not in terms of REAP achieving its intended results, mainly on a year-on-year basis since inception. This exercise will function in tandem with the team’s qualitative approach (outlined below). While analysis of existing quantitative data on REAP’s activities will undoubtedly be invaluable to the Evaluation Team in terms of contributing to its understanding of the project’s performance to date, the Team proposes to add significantly to this analysis by conducting a mini-survey as mentioned above, with grantees and non-grantees. Local Georgian company IRMS will be tasked with carrying out the mini-survey from the design stage through the presentation of preliminary findings and completed analysis for inclusion in the Evaluation Team’s final report. It is important to note here that the Evaluation Team is aware that each of the project’s two components26 has specific indicator targets that have to be met in accordance with REAP’s mandate and whose performance is reflected in Annual Reports and PMPs. Analysis of this data has two purposes: 1) to determine the extent to which REAP is achieving objectives against plan from a quantitative point of view; and 2) to provide the background against which interview questionnaires can be posed to best elicit responses aimed at answering USAID’s specific evaluation questions. 24 Specific reference is made to the document, “Restoring Efficiency to Agricultural Production (REAP) Activity in Georgia – Performance Management Plan (approved 14 January 2014) 25 In this case from September 2013 to September 2015 26 Component 1 – Competitive Matching Grants Program Component 2 – Technical Assistance (TA) Program 65 Below is a brief synopsys of how the proposed mini-survey will be designed and implemented. Further elaboration can take place at the planned USAID in-briefing at the start of the evaluation mission if clarification is needed. 5.1.2 Survey and Sampling Methodology Given the time constraints and relatively small population size of agribusinesses who applied for grants (230) a mini-survey will be conducted. The survey will cover both grantees and non-grantees (those who applied but failed). It will focus primarily on process issues, but also seek to estimate the effect of receiving a grant (the intervention). As the number of grantees is relatively small (58) a census approach can be used for this group, i.e. every grantee will be surveyed (although a certain percentage may be difficult to find, be unavailable, or refuse to take part, so that the final number of observations is likely to fall below 58). A key advantage of the census survey approach is that it essentially eliminates any risk of selection bias. Since selection of grantees was not random, but based (presumably) on the quality of the application, the matching approach is a way of mimicking randomization, by selecting a counterfactual group that is similar in its observable characteristics to the treatment group. This will enable an analysis of the impact of the grant on selected areas (such as revenue, employment levels, access to the market, etc.) Preceding the survey, two FGDs will be conducted with grantees, partly in order to help refine the questions. If possible, one FGD will consist of women grantees. If this is not possible, gender questions will be included in any case. Sub-group analysis by grantee type will not be conducted, given that grantees are engaged in highly diverse areas (e.g. Farm Service Centers, Machinery Service Centers, Fruit Nursery, Bee Keeping, Cold Storage, Processing of wild fruit and medical herbs, etc.) and because the grantee population size is small: probability is low that a further breakdown would yield statistically significant results. Analysis will be primarily descriptive, focusing on response means and comparing, as appropriate, grantees and non￾grantees. Regression analysis will be used to assess what factors contribute to an applicant’s success in winning the grant and, if possible, to revenue growth over the project period. The specific survey questions will be developed during the initial phase and following Key Informant Interviews and the two FGDs. The question topics will pertain mainly to process issues; for example, respondent perceptions: concerning the grant application process, the selection process, the tailoring of assistance to the recipients, as well as respondent characteristics, such as the above mentioned indicators (number of employees, revenue, activity type, matching contribution, year of establishment/incorporation, and location). Data on applicant characteristics will be obtained from the project’s grant application database. Research on TA recipients and indirect beneficiaries will be conducted through the use of FGDs (see section above). In addition to the 2 FGDs conducted with grantees, FGDs will also be held with non￾grantees. There are fairly large numbers of direct and indirect beneficiaries mentioned in the SOW, including the 135, 000 smallholders (defined by sales to them from FSCs) and 150,000 individuals, including 37,500 women, receiving consultations and/or training. For each type (smallholders and consultation/training recipients), approximately 4-5 FGDs will be conducted in different regions, for a total of 8-10. Of these, some groups will be mixed men and women and some will be segregated in order to obtain a gendered 66 perspective. Together with the 2 grantee FGDs, approximately 80-96 participants (the average for each FGD is 6-10 members each) will be interviewed through FGDs. 5.2 Qualitative Research and Analysis The required approach under this activity assumes particular importance given the geographic spread of REAP’s beneficiaries and the time available to the Evaluation Team to conduct its enquiries. Here, the Team’s approach will be to identify, locate, and meet with as representative a body as possible with direct or indirect knowledge and/or experience of the project for the period September 2013 to September 2015. In effect, there is less concern here as to whether a particular action has been implemented or not (verifiable or not from REAP records) as opposed to whether the result of that action met the needs and aspirations of the intended recipients. In so doing, the Evaluation Team will pay particular attention to avoiding subjective opinion and hearsay as these effectively add little or no value to understanding the facts. On the other hand, perceptions are valid and will be included in the Team’s findings as they may enhance the quality of recommendations for the remainder of the project or any future planned interventions by USAID/Georgia. In order to effectively carry out the qualitative research needed, the following approach is suggested for different interviewee groups. This is not an exhaustive list - others may be identified at the start of the evaluation mission in Georgia: 1. REAP In addition to project-related documentation already received prior to the start of the evaluation, the Evaluation Team will review REAP’s reporting procedures to include data collection and analysis methods using information technology – software programs / excel sheets – where available for scrutiny. In particular, attention will be paid to Work Plans, PMPs and Annual and Quarterly Reports. Data Gathering Approach: Meetings with REAP management and staff by the Evaluation Team, initially and throughout the period of the in-country evaluation mission. 2. Program Grantees and Non- Grantees Structured interviews will take place with representatives of both parties to determine attitudes and satisfaction levels as well as FGDs in various locations around the country (to be decided following discussions with REAP COP and key team members). Attempts will be made to elicit comment on the grant application process and result of applications (successful or otherwise). Grantees will be classified as direct beneficiaries of REAP’s support. Questionnaires will be prepared specifically for the following categories of grantees: MSCs / FSCs; Process and Information Providers; Post-harvest Handling Enterprises; and Primary Production Enterprises. Non-grantees will be classified as potential but unsucessful direct beneficiaries. Data Gathering Approach: Direct interviews with key personnel from both parties. In addition, information will be gathered at FGDs and from the planned mini-survey. 3. REAP beneficiaries receiving technical assistance (TA): Structured interviews with grantees receiving TA as well as non-grantee enterprises receiving support in business administration, farm training, quality standards and post-harvest handling among others. Feedback from these businesses is going to be essential to the Evaluation Team’s understanding of REAP’s implementation modalities as well as monitoring and evaluation (M&E) activities. 67 Data Gathering Approach: Direct interviews with key personnel from both categories of grantee in receipt of TA support. 4. Financial Institutions: Due to their involvement in the demand and supply side including issues concerning access to finance, savings, loans etc., feedback from these bodies will provide the Evaluation Team with invaluable information on the financial sector in Georgia in as much as it impacts on REAP’s beneficiaries, including women. Data Gathering Approach: Direct interviews with key personnel familiar with REAP from relevant institutions to be identifed in collaboration with REAP COP and program team members as appropriate. 5. International Community: World Bank, EU, DfID: The effect and impact of REAP’s activities will be known to various members of the international donor community such as those mentioned above among others. Therefore, their views will enrich the Evaluation Team’s understanding of how REAP is currently perceived. Data Gathering Approach: Direct interviews using structured questionnaires with key personnel familiar with REAP 6. Official Bodies: Government Ministries, Local Authorities: Those associated with the project will have a vested interested in its activities and its progress / impact on intended beneficiaries. This is likely to assume an economic perspective to take account of USAD/Georgia’s strategic objective of “inclusive and sustainable economic growth” (for Georgia) and Intermediate Result IR2.2:- “increased competitiveness and employment generation in targeted sectors”. Data Gathering Approach: Direct interviews using structued questionnaires with key individuals familiar with REAP 5.3 Limitations As pointed out in the technical proposal, there are several limitations inherent to the design of this evaluation. The most serious anticipated limitations are the following although others may become apparent at the start of the evaluation mission following discussions among team members: 1. Selection Bias: As some key informants may decline to be interviewed, there is a possibility of selection bias, i.e. those respondents who choose to be interviewed might differ from those who do not in terms of their attitudes and perceptions, affiliation with government/non-government structures, and socio-demographic characteristics and experience. 2. Recall Bias: Since a number of questions raised during the interviews will deal with issues that took place in the past, recall bias cannot be excluded. As REAP project activities were launched in September 2013, some respondents may find it difficult to accurately compare situations before and after the project. 3. Halo Bias: There is a known tendency among respondents to under-report socially undesirable answers and alter their responses to approximate what they perceive as the social norm (halo bias). The extent to which respondents will be prepared to reveal their true opinions may also vary for some questions that call upon the respondents to assess the performance of their colleagues or people on whom they depend upon for the provision of services. To mitigate this limitation, ME&A will provide the respondents with confidentiality and anonymity guarantees, where possible; conduct the interviews in the settings where respondents feel comfortable; and establish rapport between the interviewer and the respondent. FGDs will be conducted among peer groups to encourage the expression and development of ideas that may not be accepted outside of subgroups. 68 6. PROPOSED SITE VISITS AND MEETINGS The Evaluation Team has already requested from REAP’s COP a potential list of interviewees – individuals and organizations. On receipt this will be open for discussion at the in-briefing with USAID on Tuesday, 29 September 2015. 69 WORK PLAN ANNEXES 70 WORK PLAN ANNEX 2 – Evaluation Schedule 71 Georgia REAP Evaluation Schedule ◄August 2015 ~ September – November 2015 ~ December 2015 ► Sun Mon Tue Wed Thu Fri Sat 13 Sept 14 15 16 17 18 19 20 21 Review materials Home 22 Review materials Home 23 Review materials Home 24 Review materials Home 25 Submit Work Plan Home 26 27 Travel to Georgia 28 Team Meeting Tbilisi 29 USAID In￾Brief Tbilisi 30 Key Informant Interview Tbilisi Oct 1 Key Informant Interviews Tbilisi 2 Revised Work Plan to USAID Tbilisi 3 Key Informant Interviews Tbilisi 4 Tbilisi 5 Interviews & FGD TBD 6 Interviews & FGD TBD 7 Interviews & FGD TBD 8 Interviews & FGD TBD 9 Interviews & FGD TBD 10 Interviews & FGD TBD 11 TBD 12 Interviews & FGD TBD 13 Interviews & FGD TBD 14 Interviews & FGD TBD 15 Interviews & FGD TBD 16 Interviews & FGD TBD 17 Prepare for Out￾Brief Tbilisi 18 Tbilisi 19 USAID-Out￾Brief Tbilisi 20 Travel Home 21 22 23 24 25 26 27 28 29 Write Draft Report Home 30 Write Draft Report Home 31 Nov 1 2 Write Draft Report Home 3 Write Draft Report Home 4 Submit Draft Report Home 5 6 7 8 9 10 11 Receive USAID Comments 12 13 14 72 16 17 Integrate Comments Home 18 Integrate Comments Home 19 Integrate Comments Home 20 Submit Final Report Home 21 73 WORK PLAN ANNEX 3 –Illustrative Questionnaires for Interviewees 74 FGD QUESTIONNAIRE GRANTEES - MScs / FSCs 1 REGION 2 DISTRICT 3 SECTOR / SUB-SECTOR 4 NO. OF PARTICIPANTS 5 DATE OF FGD SOW Evaluation Questions: 1 and 3 Q.1 How and when did you first make contact with REAP? A.1 Q.2 What is your general opinion regarding the selection procedure and awarding process of REAP’s matching grants scheme? Satisfied / unsatisfied? A.2 Q.3 What was your primary motivation for applying for a grant from REAP? A.3 Q.4 If you had not been successful in applying for and receiving a grant what impact would this have had on your business)? A.4 Q.5 How quickly was the whole process completed from initial application to receipt of funds from REAP? (Days, weeks, months?) A.5 Q.6 How satisfied are you with this timing and what impact did this have on your business(es)? Positive / negative (depending on response(s). A.6 Q.7 If the funds received was for machinery or equipment has this been available in Georgia or have you needed to import it (e.g. for specialized food processing) NB. Procurement issues A.7 Q.8 REAP also provides technical assistance (TA) and training for grantees. Have you received one or other (or both) and what impact has this / have they had on your business(es)? A.8 75 FGD QUESTIONNAIRE GRANTEES - Primary Production 1 REGION 2 DISTRICT 3 SECTOR / SUB-SECTOR 4 NO. OF PARTICIPANTS 5 DATE OF FGD SOW Evaluation Questions: 1 and 3 Q.1 How and when did you first make contact with REAP? A.1 Q.2 What is your general opinion regarding the selection procedure and awarding process of REAP’s matching grants scheme? Satisfied / unsatisfied? A.2 Q.3 What was your primary motivation for applying for a grant from REAP? A.3 Q.4 If you had not been successful in applying for and receiving a grant what impact would this have had on your business)? A.4 Q.5 How quickly was the whole process completed from initial application to receipt of funds from REAP? (Days, weeks, months?) A.5 Q.6 How satisfied are you with this timing and what impact did this have on your business(es)? Positive / negative (depending on response(s). A.6 Q.7 If the funds received was for machinery or equipment has this been available in Georgia or have you needed to import it (e.g. for specialized food processing) NB. Procurement issues A.7 Q.8 REAP also provides technical assistance (TA) and training for grantees. Have you received one or other (or both) and what impact has this / have they had on your business(es)? A.8 76 FGD QUESTIONNAIRE GRANTEES – Post-Harvest Handling Enterprises 1 REGION 2 DISTRICT 3 SECTOR / SUB-SECTOR 4 NO. OF PARTICIPANTS 5 DATE OF FGD SOW Evaluation Questions: 1 and 3 Q.1 How and when did you first make contact with REAP? A.1 Q.2 What is your general opinion regarding the selection procedure and awarding process of REAP’s matching grants scheme? Satisfied / unsatisfied? A.2 Q.3 What was your primary motivation for applying for a grant from REAP? A.3 Q.4 If you had not been successful in applying for and receiving a grant what impact would this have had on your business)? A.4 Q.5 How quickly was the whole process completed from initial application to receipt of funds from REAP? (Days, weeks, months?) A.5 Q.6 How satisfied are you with this timing and what impact did this have on your business(es)? Positive / negative (depending on response(s). A.6 Q.7 If the funds received was for machinery or equipment has this been available in Georgia or have you needed to import it (e.g. for specialized food processing) NB. Procurement issues A.7 Q.8 REAP also provides technical assistance (TA) and training for grantees. Have you received one or other (or both) and what impact has this / have they had on your business(es)? 77 A.8 FGD QUESTIONNAIRE GRANTEES - Process & Info. Service Providers 1 REGION 2 DISTRICT 3 SECTOR / SUB-SECTOR 4 NO. OF PARTICIPANTS 5 DATE OF FGD SOW Evaluation Questions: 1 and 3 Q.1 How and when did you first make contact with REAP? A.1 Q.2 What is your general opinion regarding the selection procedure and awarding process of REAP’s matching grants scheme? Satisfied / unsatisfied? A.2 Q.3 What was your primary motivation for applying for a grant from REAP? A.3 Q.4 If you had not been successful in receiving a grant what impact would this have had on business)? A.4 Q.5 How quickly was the whole process completed from initial application to receipt of funds from REAP? (Days, weeks, months?) A.5 Q.6 How satisfied are you with this timing and what impact did this have on your business(es)? Positive / negative (depending on response(s). A.6 Q.7 If the funds received was for machinery or equipment has this been available in Georgia or have you needed to import it (e.g. for specialized food processing) NB. Procurement issues A.7 Q.8 REAP also provides technical assistance (TA) and training for grantees. Have you received one or other (or both) and what impact has this / have they had on your business(es)? 78 A.8 FGD QUESTIONNAIRE NON-GRANTEES 1 REGION 2 DISTRICT 3 SECTOR / SUB-SECTOR 4 NO. OF PARTICIPANTS 5 DATE OF FGD SOW Evaluation Questions: 1 and 3 Q.1 How and when did you first make contact with REAP? A.1 Q.2 What was your primary motivation for applying for a grant from REAP? A.2 Q.3 We understand you were not successful in your application for a matching grant. Can you tell us why you were rejected? (Note list of reasons for later analysis and tabulation) A.3 Q.4 What impact did not being successful in your grant application have on your business(es) and did you then seek an alternative source of funding in place of the matching grant? A.4 Q.5 If yes, from whom did you acquire / borrow the money? A.5 Q.6 If no, why not? A.6 Q.7 REAP also provides technical assistance (TA) and training for grantees. Have you received one or other (or both) and what effect / impact has this / have they had on your business(es)? A.7 Q.8 If yes, what sort of TA and/or training did you receive? How did you rate its quality and were you able to implement practical changes to your working practices as a result? A.8 79 QUESTIONNAIRE Ministry of Agriculture 1 REGION Tbilisi 2 DISTRICT - 3 SECTOR / SUB-SECTOR Government of Georgia 4 DATE OF INTERVIEW 5 NAME OF INTERVIEWEE SOW Evaluation Questions: 2 and 3 Q.1 How and when were you first made aware of USAID’s REAP project? A.1 Q.2 What is your understanding of REAP’s activities in Georgia and the project’s relevance to the Government’s long term strategy for agriculture? A.2 Q.3 What is the Ministry’s role or active involvement with REAP (if any)? A.3 Q.4 What are the major constraints affecting entrepreneurs / SME’s in agriculture/agribusiness? A.4 Q.5 What measures is your Ministry / Government taking to address these constraints? A.5 Q.6 Access to finance remains a serious impediment to businesses in Georgia today and esp. in agriculture. Does the Ministry have a view on how this can be addressed? (if not dealt with in Q.5) A.6 Q.7 REAP includes an initiative aimed at supporting women-owned agribusinesses with a view to their participating in the project’s matching grant scheme. What are you views on this? A.7 Q.8 Another of REAP’s cross-cutting issues relates to workforce development in agriculture. Can you comment on the nature of linkages today between private sector and academic institutions? A.8 80 QUESTIONNAIRE Financial Institutions 1 REGION Tbilisi 2 DISTRICT - 3 SECTOR / SUB-SECTOR Finance 4 DATE OF INTERVIEW 5 NAME OF INTERVIEWEE SOW Evaluation Questions: 1,2 and 3 Q.1 How and when were you first made aware of USAID’s REAP project? A.1 Q.2 What is your understanding of REAP’s activities and your relationship with it (if any)? A.2 Q.3 What are your impressions / perceptions of REAP regarding its mandate to increase employment & incomes in rural areas and in particular its matching grant scheme? A.3 Q.4 The Government and the Rural Agriculture Development Fund are developing / have developed credit instruments to support farmers. What you can you tell us about this? A.4 Q.5 REAP is supposed to facilitate relationships between its grantees and financial institutions. What are your views on this and has your institution had any direct involvement in this? A.5 Q.6 One of REAP’s cross-cutting issues is aimed at supporting women-owned agribusinesses esp. with regard to facilitating access to finance. What are your views on this? A.6 Q.7 What is your view on the current level of interest rates in Georgia being charged by commercial banks and MFI’s and in particular with respect to agriculture? A.7 Q.8 What more needs to be done by the finance sector in Georgia to support entrepreneurs / SMEs develop and grow? What plans does your institution have regarding this? A.8 81 QUESTIONNAIRE Facilitate Development of Women-owned Agribusinesses (FDWA) 1 REGION Tbilisi 2 DISTRICT - 3 SECTOR / SUB-SECTOR Cross-Cutting Issues 4 DATE OF INTERVIEW 5 NAME OF INTERVIEWEE SOW Evaluation Questions: 1,2 and 3 Q.1 How and when were you first made aware of USAID’s REAP project? A.1 Q.2 What is your understanding of REAP’s activities and your relationship with it (if any)? A.2 Q.3 What are the main issues concerning women entrepreneurs / women-owned businesses? A.3 Q.4 How many women have applied for a matching grant from REAP? How many were successful? A.4 Q.5 For those who didn’t qualify for a grant what other options were open to them to meet their needs? How many took up those options? A.5 Q.6 REAP also provides TA for farmers including women farmers or women or who use mechanized services. How successful has this been as far as you are aware? A.6 Q.7 What are your views on access to finance in Georgia and particularly for women? Does the enabling environment support women or hinder their development? A.7 Q.8 If REAP project didn’t exist what would the situation be like today regarding the agriculture sector and especially agribusinesses? A.8 82 QUESTIONNAIRE STAKEHOLDERS 1 REGION Tbilisi 2 DISTRICT - 3 SECTOR / SUB-SECTOR 4 DATE OF INTERVIEW 5 NAME OF INTERVIEWEE SOW Evaluation Questions: 1,2 and 3 Q.1 How and when were you first made aware of USAID’s REAP project? A.1 Q.2 What is your understanding / impression / perception of REAP’s activities and what relationship do you have with the project (if any)? A.2 Q.3 What are the main issues / constraints affecting agriculture today? A.3 Q.4 REAP is mandated to support to offer matching grants to farmers and agribusinesses. What feedback have you had (from any source) regarding these grants & how the scheme works? A.4 Q.5 REAP provides TA to successful grantees as well as those applicants who were unsuccessful. What is your knowledge / experience of this TA? A.5 Q.6 REAP is also mandated to support women farmers and women-owned agribusinesses. Can you tell us what feedback you have had on how successful this has been (or otherwise)? A.6 Q.7 What still needs to be done today to support agriculture in Georgia? A.7 Q.8 If REAP project didn’t exist what would the situation be today regarding the agriculture sector? A.8 83 FGD QUESTIONNAIRE TA Recipients 1 REGION 2 DISTRICT 3 SECTOR / SUB-SECTOR 4 NO. OF PARTICIPANTS 5 DATE OF FGD SOW Evaluation Questions: 1and 2 Q.1 How and when did you first make contact with REAP? A.1 Q.2 What motivated you to make contact and what were your expectations? A.2 Q.3 Can you specifiy what support you received / are receiving and how it was / is being delivered? A.3 Q.4 What impact has this TA support had on your business in terms of overall performance? A.4 Q.5 How would you rate the quality of TA support provided by REAP’s experts? Was advice / training given practical and relevant to your needs? A.5 Q.6 If the TA you received had not been available from REAP what would have been the effect? A.6 Q.7 Has the TA received satisfactorily resolved / addressed problems or pressing operating issues? If not, what more still needs to be done in your view and by whom? A.7 Q.8 Has the TA received led to permanent changes in working practices for your business? A.8 84 WORK PLAN ANNEX 4 – Draft Outline of Evaluation Report 85 TABLE OF CONTENTS EXECUTIVE SUMMARY 1.0 EVALUATION PURPOSE & EVALUATION QUESTIONS 1.1 Evaluation Purpose 1.2 Evaluation Objectives 1.3 Evaluation Question 2.0 PROJECT BACKGROUND 3.0 EVALUATION METHODS & LIMITATIONS 3.1 Evaluation Methodology 3.1.1 Quantitative Research and Analysis 3.1.2 Qualitative Research and Analysis 3.2 Evaluation Limitations 4.0 FINDINGS, CONCLUSIONS & RECOMMENDATIONS 4.1 Are REAP’s approaches necessary and efficient for developing competitive and commercially sustainable agribusinesses and increasing incomes and employment in rural areas in Georgia? 4.1.1 What are the major/challenges/outstanding needs that inhibit the project’s contribution toward achieving the results during the remaining term of the project? 4.1.2 Is technical and/or business training provided to assisted primary producers and SMEs sufficient to improve skills, yields and incomes of farmers? 4.1.3 Is the provided and intended TA specific and taylor made to each of the grantees needs? Are the quality, topics and instructors of the training appropriate for Georgian agriculture, as defined by REAP beneficiaries? 4.1.4 Are the grantees that are selected a good and fair representation to sustainably increase the availability and utilization of agricultural inputs, services and technical information among smallholder and emerging commercial farmers? Were the requirements for grantees selected in a fair and transparent manner as described by initial project design documentation? Was the grant selection process fair and transparent? 4.1.5 Has the program been successful in integrating the cross-cutting issues (gender, environment, workforce development and access to finance) into the program implementation, when appropriate? 4.2 How is the program perceived by beneficiaries? 4.3 What, if any, corrections are needed to REAP’s implementation approaches during the remaining term of the project? 5.0 LESSIONS LEARNED ANNEXES A. Statement of Work B. Work Plan & Evaluation Design C. List of Interviewees D. List of Documents Reviewed 86 E. Overall Monitoring and Evaluation Matrix F. Annual Performance Matrix G. Illustrative Interviewee Questionnaires