EVALUATION MID-TERM THEORY OF CHANGE AND IMPACT LEARNING EVALUATION OF THE USAID FARMER-TO￾FARMER (F2F) PROGRAM August 2017 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Mendez England & Associates. MID-TERM THEORY OF CHANGE AND IMPACT LEARNING EVALUATION OF THE FARMER-TO￾FARMER PROGRAM August 24, 2017 This publication was produced at the request of the United States Agency for International Development. Developed under Contract/Task Order Number: AID-OAA-I-15-00024/AID￾OAA-TO-16-00008. Submitted by: Michelle Frain Muldoon, Team Leader, Agricultural Development Specialist Eric Johnson, Technical Agriculture Specialist Ron Wendt, Research Analyst Contractor: Program Evaluation for Effectiveness and Learning (PEEL) Mendez England & Associates 1020 19th Street NW, Suite 875 Washington, DC 20036 Tel: 240-762-6296 www.mendezengland.com DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. ABSTRACT The Farmer-to-Farmer (F2F) Program aims to provide volunteers to host institutions to build their capacity to develop economically and to build awareness of the U.S. Government’s role in international development. Through eight Leader With Associates Cooperative Agreements and one additional Cooperative Agreement, seven implementing partners deploy volunteers in 28 countries at present. Over the life of F2F, 17,603 volunteer assignments have been completed in 117 countries with 1.4 million farm families as direct beneficiaries, with a value of volunteer services of $105 million. During the period of FY 2014-2016 30 percent of volunteers were female and 41 percent of trainees were female. The evaluation undertook a survey of 978 respondents from implementing partners, hosts, and volunteers to ascertain stakeholder perception of the Program. Over 90 percent of respondents were satisfied with the project due to it being “needs based.” This survey was followed up with KIIs with stakeholders in nine countries. Three main questions were examined through this evaluation: the effectiveness of the theory of change; how well indicators capture the effects of F2F activities; and how well the F2F Program integrates innovation. CONTENTS EXECUTIVE SUMMARY.............................................................................................................................i Evaluation Purpose...............................................................................................................................i Program Background...........................................................................................................................i Evaluation Questions..........................................................................................................................ii Evaluation Methods.............................................................................................................................ii Evaluation Limitations.........................................................................................................................ii Main Findings, Conclusions, and Recommendations.....................................................................ii 1. INTRODUCTION: EVALUATION PURPOSE AND QUESTIONS ..........................................1 1.1. Evaluation Purpose ....................................................................................................................1 1.2. Audience and Intended Uses ...................................................................................................1 1.3 Evaluation Questions.................................................................................................................2 2. PROGRAM BACKGROUND...........................................................................................................2 3. EVALUATION METHODS AND LIMITATIONS .........................................................................4 3.1 Methodology...............................................................................................................................4 3.2 Data Analysis..............................................................................................................................7 3.3 Evaluation Limitations...............................................................................................................7 3.5 Respondent Characteristics and Demographic Overview..................................................8 4. FINDINGS, CONCLUSIONS, RECOMMENDATIONS ........................................................... 10 4.1 EQ-1: EFFECTIVENESS: How strong and effective is the F2F Program Theory of Change (ToC)? ....................................................................................................................... 10 4.1.1: How effectively do F2F Program activities (i.e., volunteer assignments) lead to desired impacts?.......................................................................................................... 10 4.1.2: How effectively are F2F interventions (country projects, hosts, host development plans, volunteer SOWs, etc.) selected and developed to lead to change? ......................................................................................................................... 13 4.1.3: How effectively are volunteers recruited, selected, matched to SOWs, motivated, and prepared for their assignments?.................................................... 17 4.1.4: How effectively and efficiently do volunteers complete SOWs and provide desired deliverables?................................................................................................... 22 4.1.5: To what extent do hosts change behaviour and/or perceptions as a result of F2F interventions?....................................................................................................... 27 4.1.6: To what extent are persons trained/directly assisted drivers of change in relevant host organizations?...................................................................................... 30 4.2 EQ-2: INDICATORS: How well do F2F standard indicators capture the effects of F2F activities? And do they provide useful information for F2F programmanagement? .... 32 4.2.1: How well do the indicators for F2F serve program management and the monitoring and evaluation process?......................................................................... 33 4.2.2: Do some indicators present particular difficulties to hosts and IPs for collection and management interpretation? ........................................................... 37 4.2.3: Are there indicators that might be simplified or eliminated and are any others needed?......................................................................................................................... 39 4.2.4: Is the F2F Program’s performance and impact on outreach/cultural exchange, capacity development, and environmental/natural resource management being captured effectively?................................................................................................... 41 4.3 EQ-3: LEARNING & INNOVATION: How well does the F2F Program learn and integrate innovation to encourage quality improvements and good practice in agricultural volunteer programs?......................................................................................... 42 4.3.1 How well do F2F IPs engage in a community of practice to improve the Program? ...................................................................................................................... 42 4.3.2 In what ways could the Program improve learning and innovation to become more efficient and effective? ..................................................................................... 44 4.3.3 What learning and innovation activities or community of practice support are particularly effective or needed and what may be of lesser effectiveness or use? ....................................................................................................................................... 46 4.3.4 Are there unaddressed needs for support to core F2F Program quality and good practice? What are key issues for future success of the F2F Program?... 48 4.3.5 How effectively have SPSP projects (including sub-awards) supported F2F learning, innovation, and community of practice; and how has that support been perceived by the F2F IPs?................................................................................. 50 4.3.6 What impacts have SPSP sub-award projects had on core F2F activities?......... 52 4.3.7 What factors might limit or facilitate the two SPSP projects’ effectiveness and impact?.......................................................................................................................... 54 4.3.8 How effective has the Leaders with Associates (LWA) mechanism been in promoting F2F objectives?......................................................................................... 55 ANNEXES ................................................................................................................................................. 57 Annex A: Expression of Interest ................................................................................................... 58 Annex B: Evaluation protocol ........................................................................................................ 76 Annex C: Web-based survey questions..................................................................................... 129 Annex D: Key informant interview (KII) guide and questions................................................ 173 Annex E: Tally sheets by country ................................................................................................ 186 Annex F: Volunteer Survey Result……….......……………………………………………198 Annex G: USAID Response to F2F Evaluation.....…………………………………………202 LIST OF TABLES Table 1: Leader with Associates and Funding........................................................................................3 Table 2: Data Source for EQs..................................................................................................................4 Table 3: Countries/Implementing Partners Visited for Qualitative Data Collection.......................6 Table 4: Qualitative Data Collection by Country and Disaggregated by Sex...................................6 Table 5: Web-Based Survey Respondent Characteristics...................................................................8 Table 6: What Was Your Primary Employment Previous to Your Most Recent Volunteer Opportunity With F2F? .............................................................................................................9 Table 7: What Type of Host Organization Do You Belong To?........................................................9 Table 8: Volunteer Satisfaction ............................................................................................................. 18 Table 9: Standard Performance and Impact Indicators in Use in the F2F Program FY 2013- FY 2018 ........................................................................................................................................... 32 LIST OF FIGURES Figure 1: F2F Program Theory of Change .............................................................................................................. i Figure 2: Flow Chart - The F2F Basic Theory of Change (ToC) .....................................................................10 Figure 3: Assistance Received By Host Organizations.......................................................................................20 Figure 4: Inhibiting Factors to Completing Deliverables...................................................................................23 Figure 5: Challenges Experienced by IPs..............................................................................................................24 Figure 6: Factors Influencing the Ability to Implement Recommendations by Host Organizations..........28 Figure 7: Indicators Reported as Difficult to Collect.........................................................................................38 ACRONYMS AET Agricultural Education and Training BFS Bureau for Food Security CD Country Director CDCS Country Development Cooperation Strategy (USG) CSI Country Strategies and Implementation CV Curriculum Vitae DDL USAID’s Development Data Library DEC USAID’s Development Experience Clearinghouse EQ F2F Evaluation Question F2F USAID Farmer-to-Farmer Program FGD Focus Group Discussion FTF USAID Feed the Future HQ Headquarters ICT Information Communication Technology IP Implementing Partner JICA Japan International Cooperation Agency KII Key Informant Interview LOP Life of Project LWA Leader with Associate M&E Monitoring and Evaluation ME&A Mendez England & Associates MERL USAID Monitoring and Evaluation Research and Learning NGO Non-Governmental Organization NRM Natural Resources Management ODI Organizational Development Index OFI Opportunities for Improvement PDP Program Development Project PEEL Program Evaluation for Effectiveness and Learning - USAID task order contract SO OR Strategic Objective (USAID) SOW Scope of Work (aka. Assignment) SPSP Special Program Support Project ToC Theory of Change TOT Training of Trainers UNDP United Nations Development Program U.S. United States USAID United States Agency for International Development USDA United States Department of Agriculture USG United States Government i EXECUTIVE SUMMARY EVALUATION PURPOSE This report provides an evaluation of the United States Agency for International Development (USAID) Farmer-to-Farmer (F2F) Program. It was commissioned by the USAID Bureau for Food Security (BFS). The evaluation was independently conducted from November 2016 through August 2017 by the Program Evaluation for Effectiveness and Learning (PEEL) team, led by Mendez England & Associates (ME&A).1 The purpose of the evaluation and its results will be used for program improvement, idea generation, and setting future strategic direction for the Program or USAID at large. The primary audience for this evaluation is the USAID F2F Management and staff, and the F2F Implementing Partners (IPs).2 PROGRAM BACKGROUND The USAID F2F Program was first authorized by the U.S. Congress in 1985, and has been re￾authorized in every subsequent Farm Bill since then. The program seeks to provide for the transfer of knowledge and expertise of U.S. agricultural producers and businesses, on a voluntary basis, to lower- and middle-income countries, as well as emerging democracies. A main goal of the F2F Program includes the generation of sustainable and broad-based food security and economic growth in country agricultural sectors. A secondary goal aims to increase the American public’s understanding of international development issues and programs, and to advance international understanding of the U.S. and of U.S. development programs. F2F’s theory of change (ToC) is presented below, Figure 1. Figure 1: F2F Program Theory of Change Input • • • Learning through training, technical assistance, coaching Source: IP Staff & US Volunteers Cross-cultural & peer-to-peer exchange Output • • • Knowledge transferred Recommendations provided to host Citizen diplomacy and expanded worldview Outcome • • • Adoption of recommendations by host Behavioral change, new management and innovation Good will between countries Impact • • Increased productivity & incomes Improved social, human, and environmental capital THEORY OF CHANGE 1 Program Evaluation for Effectiveness and Learning (PEEL) Contract/Task Order Number: AID-OAA-I-15-00024/AID￾OAA-TO-16-00008 2 USAID F2F is in the Bureau for Food Security (BFS)/Country Strategies and Implementation (CSI)/Technical Division ii EVALUATION QUESTIONS (EQS) According to the F2F evaluation scope of work (SOW), the team was required to address the following questions: • • • EQ1: How strong and effective is the F2F Program ToC? EQ 2: How well do F2F standard indicators capture the effects of F2F activities, with particular emphasis on indicators for outreach/cultural exchange, capacity development, and environmental/natural resource management? And do they provide useful information for F2F program management? EQ 3: How well does the F2F Program learn and integrate innovation to encourage quality improvements and good practice in agricultural volunteer programs? EVALUATION METHODS In its approach to the evaluation, the team applied rigorous mixed methods, which included: 1) qualitative data collection in the form of a literature review of USAID F2F project documents and performance reports, and the implementation of key informant interviews (KIIs) and focus group discussions (FGDs); and 2) quantitative data collection, which included an online web￾based survey distributed to 1,863 potential respondents. The survey was completed by 884 F2F IP staff, host organizations, and volunteers; 67 percent of those that responded were male. The team then gathered additional qualitative data to validate and supplement the survey findings. In total, the team visited eight countries in three regions globally, for the conduct of nine FGDs and 137 KIIs. Over half (56 percent) of FGD and KII respondents were male. EVALUATION LIMITATIONS Due to time, resource, and design constraints, not all F2F stakeholders are equally represented in the evaluation. This is especially true for lower-capacity host organizations, which either may have lacked the language skills, email, or Internet access to participate in the web survey. The qualitative data was limited by time and resource constraints, which prevented more countries from being included in the data collection. The evaluation team agreed to maintain the anonymity of the participants in the evaluation. It is recognized that given the nature of the Special Program Support Project (SPSP), the evaluation team had to be cautious in making recommendations regarding this one project. MAIN FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS The main findings from the evaluation are provided below. More detailed explanations are provided subsequently in the main report. iii Findings Conclusions Recommendations The majority of survey respondents are pleased with the F2F Program. Ninety percent of host organizations and 92 percent of IPs are very satisfied or satisfied with it. Seventy-five percent of volunteers reported that the host activity was improved by their work and led to desired impacts, such increased sales, jobs created, new markets, and access to credit. When asked what drives their satisfaction with the Program, interviewees stated that the Program is: 1) needs￾based, 2) market-driven, and 3) volunteer-based. F2F’s longevity has been tied to its core philosophy of peer-to-peer volunteer-based sharing of learning, and the ensuing citizen diplomacy and cultural exchange combined with economic impacts of the Program. F2F is able to quickly adapt, because adaptation is a result of its needs￾based approach. The F2F program model as currently designed and ideally implemented should be continued. The basic ToC is effective. Data reveal that a key driver in terms of the strength and effectiveness of the F2F ToC concerns the spirit of volunteerism and the overall uniqueness of the Program. The ToC could be even more effective and efficient (resulting in less human, financial, and opportunity costs) if intentionally viewed from a systems, networks, and processes perspective – where activities, people, and resources are interconnected and inter-dependent. The ToC is effective; but the causal relationship from input to impact is often nonlinear and variable depending on context and on effective and efficient activities to support the ToC. The evaluation team recommends a broader ToC that includes the processes, variables and support systems and networks to generate desired impacts working within the parameters and budget of an IP. Visualizing and mapping out the network and systems, and discussing those with IPs can be a great step toward identifying weak or missing links that if repaired or strengthened could more effectively and efficiently lead to desired impacts, while reducing costs related to inefficiencies. Ninety percent of the hosts are very satisfied or satisfied with their involvement in developing the intervention Scope of Work (SOW), indicating that the SOW takes their priorities into account. All agreed that the development of the SOW was essential and that there are different methods and best practices to develop a good SOW that both meet the needs of the host organization, and also make recruitment, M&E, and other IP functions much easier. Conducting a needs analysis before the development of the SOW was mentioned as an important factor by several key informants. Both host organizations and IPs emphasized the importance of a well-crafted SOW to address the needs of host organizations. As some host organizations told the evaluation team, “we know what our needs are,” and what is required is guidance to navigate them. Therefore, a needs “analysis” that asks thoughtful questions to better understand the situation in order to co-design a proposed solution (SOW) is important. SOW development should be: 1) a collaborative affair with as much input from the host organizations as possible; and 2) based on a highly analytical needs assessment. The needs analysis should explore the business plan of the host organization and include a SOW with multiple volunteer visits and more than one volunteer. SOWs need to have not only an implementation plan but also a monitoring plan to be able to demonstrate change. Where appropriate, such efforts by volunteers could be formalized as a deliverable in their SOWs. The team recommends sending someone who is technically familiar with the host’s work/crop/sector to conduct the needs analysis, so the IP is asking the right questions, and getting at the true need, which may be different than what the host has identified as its top priority. KII host respondents expressed opposition to “one off” assignments where they receive one volunteer a year and the SOWs are not related or Programs are more effective when F2F interventions are co-created with IPs with a long-term horizon and with sustainability in mind and in which volunteers implement A standard model of F2F program development and implementation should be developed in which programs are co-created by hosts and IPs with a long-term planning iv Findings Conclusions Recommendations connected, and there is no follow up when the volunteer leaves. Hosts that expressed the most enthusiasm and had demonstrable impacts also appreciated how effective the F2F interventions are when they are co-created, together with the IP, with the longer term economic sustainability in mind, to endure after the program is gone. SOWs that are related and there is follow-up after the volunteer leaves. One off assignments can be effective too; for example filling a gap for a USAID Mission Feed the Future project that urgently needs a particular skill set it is unable to find to address a very specific need. horizon and sustainability in mind, volunteers are brought in to implement related SOWs, and IPs follow-up with hosts to ensure that the intervention succeeded. For a one-off assignment, where a host receives one to three volunteers over the life of the host organization or project, make sure that F2F is addressing an area that can yield the highest return on investment. Although overall the recruitment process is effective according to both volunteers and most IPs, there remains some disagreement between volunteers and some IPs regarding its overall effectiveness. The large majority of volunteers reported that they are effectively recruited, selected, matched, motivated, and prepared for their assignments. Some IPs reported in KIIs, however, that there are issues with recruitment that impact volunteer relevance and performance. The recruitment process itself is uneven across IPs, depending upon their capacity. The volunteer recruitment, placement, and preparation program is not functioning as efficiently and effectively as it could. Hosts with less capacity are especially likely to be marginalized during the recruitment process, while IPs with less resources, projects or staff may struggle with recruitment. These factors are complicated as well by IP headquarters occasionally struggling to find an appropriate volunteer. These cases, in turn, can lead to poor matches, affecting the assignment’s impact. F2F should develop a comprehensive system of volunteer recruitment that allows all IPs of various capacities to have the same contact with the volunteers in order to maximize effective placement. F2F should have a grading system or other way to track volunteer performance, and require IPs to: 1) share information on poor performing volunteers and the hosts effected (where relationship repair may be needed); 2) reduce wasted time and money; and 3) lose valuable partnerships that are costly to acquire and grow. There is sometimes inadequate sharing of Curriculum Vitae (CVs) between IPs, which can result in recruitment of under-performing volunteers. Moreover, the various needs of the IPs impacts recruitment success. Communication between IPs regarding volunteers and information to prepare volunteers for their assignments is not sufficient for a consistently high-functioning recruitment process. Lack of sharing CVs can result in IPs hiring low-performing volunteers, negatively impacting their program outcomes. IPs should be encouraged to share information about volunteers to ensure that appropriate volunteers are hired by other IPs and to see where else they may have served, and with which IP. To solve the issue of confidentiality, USAID F2F should keep a central database of volunteers. F2F should then work with IPs to create a system of evaluating volunteer performance and otherwise recording volunteer profiles. Host organizations are generally happy with the work of volunteers and find that volunteers do effectively and efficiently complete their SOWs. The survey showed, however, that 16 percent of volunteers do not complete their SOWs because of time, timing, and logistics. It remains a challenge to field highly qualified and experienced volunteer consultants who can take the time to serve in the program. Logistics involves all the processes and steps involved with recruitment, management, Although the majority of host organizations report that volunteers effectively and efficiently complete their SOW, there remain several important constraints to effective completion. These include time, timing, and logistics. For cash-strapped hosts, the cost of interventions often exceeds their means. The issues of time and timing should be top priorities to maximize knowledge and learning exchange opportunities in the limited time and resources available. F2F should address them in a deliberate way by including time/timing/time allocation as a topic for future IP workshops, and discuss all the ways that time needs to be considered to improve the Program’s effectiveness. The evaluation started this by doing a process audit that is part of this evaluation. F2F should review the steps, activities, and transactions that occur, v Findings Conclusions Recommendations placement, health, safety, transportation, security, and, later, follow up with volunteers, as well as the steps in between. Cash-strapped hosts cannot cost share even basic needs (e.g. water during volunteer training). where and with whom, and map the links and processes as part of the larger ToC exercise (above). To keep morale and momentum going after a F2F intervention/assignment, we recommend that F2F not provide additional financial support to host organizations, but instead explore with IPs creative ways to address the challenges of adopting recommendations Sixty percent of hosts say they have adopted more than half of the volunteer’s recommendations. Example of behavioral changes includes the way a host conducts business: a marketing plan that a host co-designed with a volunteer led to their new ways of networking and promoting their farmer-owned village bank to a wider region, and their membership increase from 500 to 700 members. F2F Program is changing the behavior and perceptions of hosts, with regards to their work, and the evaluation team observed, toured or otherwise experienced its outcome and impact first hand. The biggest challenge appears to be less about motivation to adopt a recommendation and more about practical aspects like money and time. A timeline and plan may be necessary to help the host stay on track, to avoid losing motivation or momentum, which can happen when there is no communication from IP or volunteer and the host is unable to implement the ideas and recommendations discussed with volunteer. If a host requires funding or follow-on support and coaching, F2F volunteers and IPs can identify local networks, and create links to other international actors. A SOW to help hosts develop business plans can help them get access to finance when their plans and accounting are clear. Several KII host organization participants reported that they were now a model to others in the host organization and/or community. Some have become production and education farms, for example, a greenhouse operation that has now partnered with the local agriculture college and is charging money for tours because “we are so busy.” Others take what they have learned, such as low technology food processing, and go out to other communities to teach them. Host organizations that have directly received effective interventions, are valuable resources for F2F, in that they validate the credibility of the program, can introduce F2F to other leaders/models/drivers, and are for the most part available and willing to meet with F2F or provide other assistance to help advance the F2F ToC. Follow up and follow-on support is needed, even if it is a phone call or free What’s App message. Communication and support networks should be available and open to the host member who is trained. Volunteers can facilitate it, but follow up and a periodic communication is important. Connect hosts to other hosts, or have a small meeting of hosts. The value they bring and will generate in terms of outreach and networking support could offset the cost of such a meeting/event. Over 75 percent of IPs use the information provided by F2F indicators to make program management decisions; while 85 percent of IPs indicated that the indicators provide information that is important for the monitoring and evaluation of their program’s work. Indicators are most useful for tracking progress of their 5-year country work plan and, when reviewed against budget expenditures, can help IPs ensure that The indicators are used mostly by USAID and the F2F IPs (Headquarters and Country-level). Some host organizations track indicators, especially if they are partnering with, for example, USAID or USDA and international NGOs, which may have exposed the host to indicators and M&E. The F2F indicators serve well in terms of program management and the monitoring and evaluation process with some exceptions. They give IPs and all stakeholders a common language and common Every set of Indicator Tables should have a brief narrative, similar to budget notes, a short one to two-page summary that provides the reader with an overview of the indicators and any outliers or special considerations to provide the reader with an at-a-glance context, without having the go to the large report. IP field staff should be able to explain in relevant terms to the host why they need the data. IP field staff can also develop vi Findings Conclusions Recommendations funds are directed to maximize effectiveness of the program to reach desired impacts. ground from which to start and a common focus. simplified templates for hosts to review and fill in, as some IPs are doing with more literate hosts. The indicator “area of potential environmental/natural resource influence” was most commonly cited as being difficult to collect, with nearly one-third of IP respondents selecting this indicator. Second to this, “annual net income” was selected by 32 percent of respondents. The “area under improved environmental/natural resource management” is difficult to collect according to 24 percent of respondents, while the “value of annual gross sales” is difficult to collect according to 21 percent of respondents. It is very difficult to track the value of sales because hosts are often reluctant to provide that information, fearing questions about taxation. In addition, there is often the absence of written or other records to consult to establish a baseline and, therefore an accurate measure of economic gain. There are many ways to track environmental impact, but they are not standard, for example, improved yields (kg) thanks to F2F intervention in soil health. Value of Net Income would be a better indicator than gross value of sales, though even then there are challenges. F2F should lead a discussion on monetizing and quantifying environmental indicators, to see what might be added or changed, for example, reduction in fertilizer use (cost savings) thanks to composting, improved yields (kilos, bushels, or units) thanks to improved soils. The Organizational Development Index (ODI) was identified by 28 percent of IP respondents—more than any other indicator—as needing simplification through the evaluation team’s web survey. However, there was a significant difference of opinions from respondents concerning this indicator, as found during the evaluation’s field visits. The ODI indicator needs to be simplified. The ODI indicator should be considered for elimination, given the absence of clear, objective rubrics for measurements that can be applied across the board to all IPs, and thus contribute to reliable data collection. The evaluation team suggests exploring other tools and thinking in terms of: nonfinancial assets (land, labor, and buildings), livelihoods, and resilience. IPs felt that cultural exchange and the indicators for environmental/natural resources were particularly weak. To the first of these, 47 percent of respondents felt that the indicators for cultural exchange did not effectively capture the program’s performance. Similarly, 38 percent of respondents felt the indicators for environmental/natural resource management were not capturing the program’s performance. One-third of respondents felt the program’s indicators for outreach did not capture the program’s performance and impact. With regards to capacity development, 18 percent of respondents responded negatively. Improvements can be made to the indicators related to outreach/cultural exchange, capacity development, and environmental/natural resource management. Number of Press Releases; Number of Media Events by Implementers, Volunteers, Hosts, and Partners; and Number of Group Presentations by Implementers, Volunteers, Hosts, and Partners should be collapsed into a new indicator called “Number of Outreach Activities” as they do not significantly measure anything different. Add to reporting requirements, “How did you first hear about F2F?” as an awareness indicator. Knowing how someone heard about the F2F program is a useful indicator for what’s working, and where to amplify outreach. IPs strive to make improvements through regular virtual and in-person meetings. These meetings provide the chance for: 1) peer-to-peer exchange How well IPs engage as a Community of Practice (CoP) varies by IP and IP staff type. Among the core group of IP HQ staff management, the IPs engage well, but IP field To help IPs better engage in a CoP to improve the program, F2F should first examine and agree on the goals and objectives of each learning group, whether CoP, committee, vii Findings Conclusions Recommendations with fellow IPs working in other regions of the world; 2) cross training through discussion and analysis of innovations and sectors; and 3) learning about interventions by fellow IPs that may be new to another IP. However, 15 percent of IPs are dissatisfied with the level of collaboration and information sharing among implementers. staff beyond the Country Director are disconnected from the learning that takes place at the IP HQ level. This exclusion is not intentional. Information sharing faces obstacles due to competition among IPs as bidders, different experience levels among participants, and a lack of dedicated resources. or something else; then ensure that the appropriate people with the right skill set are in that group to provide the best value, or to ensure that there are numerous viewpoints represented, not just those of IP HQ management. The survey findings show that 73 percent of IPs felt that “extension and advisory services” should be prioritized to improve the effectiveness of the F2F Program, while 68 percent felt “agribusiness/value chains” should be prioritized. Additionally, the most popular answer among IPs, when asked for the key issues for the future success of F2F, was improvements to M&E systems/indicators (42 percent) and recruitment of volunteers with more specialized experience (34 percent). Increasing public knowledge of F2F can help in the recruitment of volunteers and support for the program in general, key to its future success. There exist unexplored opportunities to leverage the tremendous interest in the F2F Program by people outside of program IPs, and also opportunities to promote the program in ways that could yield multiple benefits. Volunteers are a low-cost resource for promotion of the F2F Program. The M&E system/indicators are a key component to the Program’s future success. F2F should make more use of volunteers to carry out more public awareness. The Program should develop a process to monitor and follow up with volunteers after their service is complete in order to facilitate outreach to the general public. Revisions to M&E systems and indicators may be needed, as mentioned above. A plurality of IPs are satisfied with SPSP (38 percent satisfied, 9 percent unsatisfied, 15 percent neutral and the remainder “Don’t Know” or “N/A”). When asked about effectiveness of SPSP, more respondents rated the program effective than not (46 percent “Yes” versus 17 percent “No,” and 36 percent “Don't Know”). Among respondents aware of SPSP, the F2F website is its the most noted service. While respondents almost unanimously value learning and knowledge activities, as was confirmed in KIIs, the “SPSP” as an acronym has poor name recognition, with “Don’t Know” representing more than one third of responses across relevant questions. IPs do not make the connection between Learning/ Knowledge Management and SPSP, the service provider. Since most survey respondents find the SPSP program effective, it appears to be more of a name recognition branding issue, or possible lack of validation among IPs served. SPSP lacks name recognition among IP staff. As noted above, the F2F website is an underused website, especially among field-based staff. This should be further promoted. Additional branding work may be done at the same time to increase awareness of SPSP. Thirty three percent of IPs say that SPSP has “generated new ideas and innovations,” while 32 percent report that it has “supported knowledge management between and within the organization” and 9 percent report that it has “facilitated special studies.” Twenty nine percent SPSP has provided opportunities for testing new ideas and as a source for knowledge management. However, a significant number of IPs cite that SPSP has not had any significant impacts. In an effort to enhance sharing and effectiveness, USAID should consider improving ways for SPSP grantees to engage in the learning and community of practice of the core IPs. For example, funds should be provided to all small grant and PDP implementers to attend the annual implementers meeting. viii Findings Conclusions Recommendations find that it has “no significant impacts.” Nearly half of IPs (48 percent) say they “don’t know” when asked which types of activities by SPSP have been most helpful for their work. Small grants are often excluded from larger F2F events. Being inclusive allows small-grants to learn from experienced IPs and their knowledge and insights. Further promotion of SPSP and the F2F website should be made. Develop case studies on each small grant and PDP to discuss their evolution and growth and how F2F has positively impacted their work and how effectively. Conduct additional Special Studies to reinforce the capacity of IPs and generate new ideas. “Budget constraints” (44 percent) was listed by survey respondents as the top constraint for SPSP effectiveness and impact. Logistical constraints were also commonly cited as a limit to SPSP’s effectiveness and impact. As an overall program, SPSP is under budgeted for the ambitious multiple roles the program plays. The multiple roles served by SPSP are ambitious and require more time and resources than the budget currently allows to complete what SPSP was originally designed to do (per RFA proposal). The SPSP program warrants its own evaluation. Such an evaluation would facilitate an in-depth look at the budgetary and programmatic success and challenges of Small Grant and Program Development Project awardees, and a full analysis of its communications and knowledge management services provided. The core IPs each manage an LWA, in coordination with USAID F2F. Each LWA serves as an umbrella agreement similar to an Indefinite Quantity Contract. USAID Missions can create their own Associate Awards with the IP in lieu of going through another competitive (or sole source justification) process, which allows for new programs to be designed and implemented within a few weeks of approval. LWA mechanism is easily executed with less turnaround time than a traditional solicitation, allowing USAID and its partners gain efficiencies while managing risk by allowing already competitively selected IPs to vet new ideas that once USAID approves. The team recommends that F2F keep the LWA mechanism as a way to streamline and gain efficiencies in the award solicitation and review process. The buy-in mechanism for SPSP should be replaced by the more efficient LWA mechanism. 1 1.0 EVALUATION PURPOSE AND QUESTIONS 1.1. EVALUATION PURPOSE This report describes the results of the evaluation of the United States Agency for International Development (USAID) Farmer-to-Farmer (F2F) Program, conducted between November 2016 and August 2017. The F2F Program was founded in 1985, signed into law by the United States (U.S.) Congress; it has since been re-authorized in every U.S. Farm Bill for the last 30 years, which speaks to its popularity and longevity.3 The current Farm Bill was signed into law in February 2014 and discussions have begun to design the next Farm Bill. The fact that the Program falls within the Farm Bill makes it unique and somewhat of an outlier among development programs. Its success, longevity, even existence, on the other hand, is somewhat of a mystery to development actors. This evaluation attempts to capture the essence of F2F Program and explore the factors contributing to its longevity and favorability among volunteers and recipients alike. While the core philosophy of the Program is unchanged, the environment, context, and world within which it now operates is dramatically different. The Program has adapted accordingly. This evaluation studies the current performance in the context of the Program’s evolution to identify what is working and what could be improved for greater impacts as well as what development practices and theories should be maintained, grown, or shared as models for other development programs, whether led by USAID or other U.S. Government (USG) or non￾governmental organization (NGO) entities. The evaluation has two main goals: 1) Technical Performance: To identify best practices and areas in which the Program can make mid-course corrections. This will enable implementing agencies to take stock of implementation approaches, strategies, organization, and results and make any changes that are indicated to improve performance. 2) Informing U.S. Development Policy and Strategy: To provide guidance for planning future F2F Program activities. 1.2. AUDIENCE AND INTENDED USES The primary audience for this evaluation is the USAID F2F team [Bureau for Food Security (BFS)/Country Strategies and Implementation (CSI)/Technical Division] and F2F Implementing Partners (IPs). The results of the evaluation will be used for program improvement, idea generation, and setting future strategic direction for the Program or USAID at large. A 3 2014 US Farm Bill, the Agricultural Act; Accessed from United States Senate Committee on Agriculture, Nutrition and Forestry https://www.agriculture.senate.gov/issues/farm-bill (The U.S. Farm Bill is a bundle of legislation that Congress passes that sets the U.S. agenda and policies for domestic and international agriculture, nutrition, conservation, and forestry.) 2 secondary audience is U.S. policy and decision makers and those who could benefit from the results as they consider programming USG-funded initiatives. 1.3 EVALUATION QUESTIONS All planning, data collection, analysis, and reporting activities in this evaluation center on the three evaluation questions (EQs), provided by USAID. Each EQ is accompanied by illustrative sub-questions, which the team has elaborated on, through a web-based survey and KIIs. The three main EQs are: • EQ1: How strong and effective is the F2F Program Theory of Change (ToC)? • EQ 2: How well do F2F standard indicators capture the effects of F2F activities, with particular emphasis on indicators for outreach/cultural exchange, capacity development, and environmental/natural resource management? And do they provide useful information for F2F program management? • EQ 3: How well does the F2F Program learn and integrate innovation to encourage quality improvements and good practice in agricultural volunteer programs? 2.0 PROGRAM BACKGROUND The USAID F2F Program was first authorized by the U.S. Congress in 1985, and has been re￾authorized in every subsequent Farm Bill since then. Its aim is to provide for the transfer of knowledge and expertise of U.S. agricultural producers and businesses, on a voluntary basis, to lower and middle-income countries as well as emerging democracies. The F2F Program goal is to generate rapid, sustainable, and broad-based food security and economic growth in the agricultural sector. A secondary goal is to increase the American public’s understanding of international development issues and programs, and to advance international understanding of the U.S. and of U.S. development programs. To further F2F’s secondary goal, the F2F Program designs volunteer assignments and scopes of work (SOWs) to maximize opportunities for volunteer work and interactions with rural people in the host countries where the Program is taking place. This centrally-funded Program is managed by USAID from its Washington, DC Headquarters (HQ), and works alongside external IPs to facilitate exchange of experiences between U.S. experts (producers, agribusiness entrepreneurs, academics, university professors, etc.) and counterpart host country agricultural producers, entrepreneurs, and farmer-serving institutions in developing countries. The ultimate objective of the Program is to boost economic growth and sustainability. The F2F Program has served 117 countries worldwide to date. F2F awards programs via two funding mechanisms. Cooperative Agreement Leader with Associate Awards (LWA) under which the Cooperative Agreement Leader Awards fund the F2F core program activities. Under the LWA mechanism, USAID Missions or other offices can fund additional Associate Award activities that provide agricultural volunteer services and/or provide additional assistance that complements work of volunteers. Any Associate Award activities are funded and managed separately and are not considered a part of the global F2F Program. Additionally, there is F2F 3 Special Program Support Project (SPSP), which is a relatively new program, currently in its second phase and second implementer. SPSP was designed to be a service provider to F2F IPs, in the role of a grant-maker and knowledge broker, and also to provide capacity support to the grants it funds. The awards and budgets are reflected in Table 1 below. Combined, the two types of F2F programs aim to take volunteer technical assistance inputs, including training and recommendations made by volunteers or co-designed with partners, and to turn those recommendations into behavior, production, and marketing changes that lead to better farm and labor productivity, increases in income, and nonfinancial and intangible assets, such as social, human, and environmental forms of capital. The F2F Program has been global in nature, although most programs have core countries within a single region per the list below of LWAs (see Table 1, below). Table 1: Leader with Associates and Funding FY 2016-FY2018 Program Core Activity Focus Implementing Agency Agreement No. Implementation Period Total LOP Award Obligation Ceilings ECCA (Europe, Caucasus, Central Asia) ACDI/VOCA OAA-L-13- 00007 9/30/13-9/29/18 $7,985,174 West Africa ACDI/VOCA OAA-L-13- 00004 9/30/13-9/29/18 $7,973,741 East Africa CRS (Catholic Relief Services) OAA-L-13- 00002 9/30/13-9/29/18 $8,035,223 Southern Africa CNFA OAA-L-13- 00003 9/30/13-9/29/18 $7,997,980 MENA (Middle East & North Africa) Land O'Lakes OAA-L-13- 00008 9/30/13-9/29/18 $7,999,450 Caribbean Partners of the Americas OAA-L-13- 00005 9/30/13-9/29/18 $8,000,000 Ag Education & Training Winrock International OAA-L-13- 00009 9/30/13-9/29/18 $7,998,178 Asia Winrock International OAA-L-13- 00006 9/30/13-9/29/18 $7,995,565 Special Program Support Project (F2F Funds)4 VEGA OAA-L-13- 00053 9/30/13-9/29/18 $9,000,000 Total $72,985,311 The USAID F2F is a unique program within the portfolio of USAID/BFS that works in synergy with other programs and initiatives, such as Feed the Future, but is not constrained by any one focus area, geography, or type of intervention. This diverse mix of host organization type, size, sector/industry, geographical location and climate is the result of the Program approach being 4 $3,000,000 of the SPSP funding is from Mission buy-in. 4 “needs-based,” where F2F co-designs solutions through knowledge and culture exchange that is facilitated by IPs. 3.0 EVALUATION METHODS AND LIMITATIONS The evaluation team began its work through an initial literature review of project documents and performance reports made available by USAID in order to become familiar with the project and its stakeholders. The team also held exploratory discussions and conference calls with USAID staff and IPs to better understand the context and technical aspects of the F2F Program. These discussions allowed the evaluation team to clarify any questions raised through the literature review or from the evaluation team’s discussions with other project stakeholders (see Table 2: Table Data Sources for EQs). Table 2: Data Source for EQs Evaluation Question (EQ) Data Source EQ 1—How strong and effective is the F2F Program TOC? Web survey related to volunteer satisfaction on recruitment, project assignment, and impact of work; host satisfaction with volunteers’ training and recommendations; IP satisfaction overall Key Informant Interviews (KIIs) with IPs and hosts EQ 2—How well do F2F standard indicators capture the effects of F2F activities and provide useful information for program management? Web survey related to IPs’ opinions on indicators’ usefulness for project management and project monitoring and evaluation; and indicators that are difficult to collect, need to be simplified, or need to be eliminated KIIs with IPs and hosts EQ 3—How well does the F2F Program learn and integrate innovation to encourage quality improvements and good practice in agricultural volunteer programs? Web survey related to IPs’ opinions on F2F’s practices related to learning, innovation, and best practice sharing; impact of SPSP activities on IPs’ work; and satisfaction with SPSP KIIs with IPs and hosts 3.1 METHODOLOGY Quantitative Data Collection Data was collected through a mixed-methods approach, using the insights from the exploratory interviews and literature review to create a quantitative web-based survey, which was disseminated among project stakeholders. These included IPs, host organizations, and active and 5 former volunteers5 . The evaluation team collaborated with the IPs to compile a list of contact names and email addresses for the survey’s dissemination, with a focus on individuals who had been engaged in the current program cycle. In total, 1,863 valid and unique contacts were shared by IPs for the web-based survey. Once these contacts were compiled, respondents were sent email communications to inform them of the survey, request their participation, and share a unique, secure link to access the survey. Communication with participants was conducted in English, Spanish, and French, the same three languages used for the survey administration. In total, 978 IP staff, host organizations, and volunteers began the web-based survey with 884 completing their respective modules. The survey respondents in the case of volunteers and IPs included both core program volunteers and SPSP volunteers. As a result, 47 percent of the sample of 1,863 respondents completed the web-based survey. This compares favorably to an industry average of approximately 25 percent for web-based surveys. An additional 71 surveys were excluded from the survey analysis since their relationship to the Program was outside the population of interest. Also excluded were six surveys who reported that their languages skills were either “Poor” or “Very poor” for the language in which the survey was administered. Qualitative Data Collection Following the administration of the web-based survey, an initial analysis was conducted to identify areas of interest for the qualitative data collection. The evaluation team crafted qualitative guides to verify the web-based survey results, document key experiences, solicit recommendations from F2F stakeholders, and uncover additional information not easily collected through a quantitative tool. The countries and projects for the fieldwork visits were selected after an extensive analysis conducted by the evaluation team in consultation with the IPs and the USAID F2F management team. The consultation process allowed for the inclusion of a diverse range of contexts and a focus on previously unevaluated projects, geographical spread of IPs, and types of implementing mechanisms while minimizing bias and costly logistical efforts that are unlikely to improve data precision and quality. Other factor considered included availability, since some field visits depended on logistical support from the IPs. However, the final country selection was the responsibility of the evaluation team. The final countries and projects selected for a fieldwork visit yielded a range of data and learning, not just on performance by sector or country, but also on development trends worth further exploration by F2F. For example, some country projects have a very strong women￾value chain component, others are piloting rural financial services, while another group of projects focuses on youth. In addition, through exploratory research, it became evident that there were differences among IPs in their relationships to USAID Missions and in the number of partnerships and the types of partnerships maintained by IPs. The final mix of country projects allowed the evaluators to explore how these differences impact IPs and their programs. In total, eight countries were visited by the evaluation team as detailed in Table 3. 5 Active volunteers are those currently engaged in an assignment. Former volunteers have completed an assignment 6 Table 3: Countries/Implementing Partners Visited for Qualitative Data Collection Region Country Implementing Partner Africa Senegal Winrock International, ADCI/VOCA-NCBA/CLUSA Malawi CNFA Kenya Catholic Relief Services (CRS) Asia and Middle East Tajikistan ACDI/VOCA Lebanon Land O’Lakes Bangladesh Winrock International, Tennessee State University6 Caribbean, Central and South America Jamaica Special Programs: Florida Association for Volunteer Action in the Caribbean and the Americas (FAVACA) Guatemala Partners of the Americas/Caribbean Per the selection process, the evaluation team was able to visit at least one country where each IP was engaged, and one SPSP country. The field visits allowed the evaluation team to visit host organizations that had lacked the capacity or resources to participate in the web-based survey, providing an important counterbalance to the representativeness of the web-based survey’s results. For the survey, the team selected host organizations where there was a contact with an email address that could be verified. For the field visits, the evaluation team worked with the IPs to ensure that contact information for seventy five percent of the in-country host organizations was available. Once this cohort of host organizations was identified, a random selection of hosts was chosen to be interviewed as key informants. An attempt was made to ensure geographical, gender, and host type diversity. Through the qualitative data collection, a combined total of 146 focus group discussions (FGDs) and KIIs were conducted, for a total of over 300 total participants across select F2F countries (see Table 4). Each qualitative interview was designed to last between 30-45 minutes. Translators were hired to assist in the KIIs and FGDs. Where translation occurred or where there were a large number of respondents, the time was significantly longer. Table 4: Qualitative Data Collection by Country and Disaggregated by Sex Country Number of FGDS (>=5 participants) Number of KIIs Male Participants Female Participants Senegal 2 17 27 18 Malawi 1 13 7 18 Kenya 0 18 14 9 Tajikistan 1 25 22 12 Lebanon 0 20 22 6 Bangladesh 5 13 54 54 Jamaica 0 19 10 10 6 TSU did implement a VEGA Small Grant Project in Bangladesh, but the hosts were not contacted because the activity has ended. 7 Country Number of FGDS (>=5 participants) Number of KIIs Male Participants Female Participants Guatemala 0 12 13 8 Total 9 137 169 135 Note: FGD = Focus Group Discussions 3.2 DATA ANALYSIS The study’s qualitative data was first recorded in field notes by the evaluation team. These were then standardized across all countries using a common template. Following this, the data was coded thematically in order to identify key themes and common experiences of IPs, volunteers, and host organizations across the countries visited. The evaluation’s web-based survey data were analyzed using Stata and Microsoft Excel. Stata￾do-files were created to clean and label the survey data following its export from the web￾based survey database. Do-files were also used to construct frequency tables, cross-tabulations, and graphs in Stata. Microsoft Excel was used to create tables and graphs and for the purpose of organizing and categorizing open-ended responses. 3.3 EVALUATION LIMITATIONS The purpose of this evaluation was to look at the F2F Program overall rather than any particular IP or country program. Data collection, analysis, and reporting are considered in the aggregate. Given this aim, there is need for the evaluation team to tread carefully with SPSP￾related questions because to evaluate SPSP is essentially to evaluate a single IP (not the F2F Program overall). Given the breadth of the F2F Program and the diversity of projects and host organizations with which it is engaged, it was not possible to design a performance evaluation that could reach all respondents. Due to a limited budget and schedule, the evaluation team decided that a web￾based survey followed by field visits for qualitative data collection would provide the greatest coverage. However, some populations remained underrepresented. As the web-based survey required access to a computer, Internet coverage, and an email address for participation, host organizations without one of these assets were more likely to be excluded. Due to project finance and time constraints, the web-based survey was also restricted to three languages – English, French, and Spanish – as previously mentioned. As a result, host organizations unfamiliar with any of the three languages used, were unable to complete the web-based survey. In addition, the representativeness of the web-based survey is contingent upon the quality of the contact lists shared by the IPs. The evaluation team does not consider this to be a significant issue, given the large number of contacts that were shared with the team. There were differences in participation based on the stage of the project cycle. For example, projects which were part of the current program cycle but had already ended were not always responsive to email requests for the contact information of host organizations and volunteers, despite efforts from some IPs to obtain these contacts. Similarly, projects which were early in 8 their lifecycles did not always share contacts for the web-based survey. Contact information for host organizations was the most common to be omitted, despite repeated communication. In order to help combat these sources of bias, especially as they related to host organizations, the evaluation team made a particular effort with the KIIs and FGDs to reach host organizations, which may not have otherwise been represented in the web-based survey. The qualitative arm of the evaluation was potentially limited by language barriers and by the availability of respondents. To the first of these, the use of interpreters for local languages may create bias as a result of his/her misinterpretation of the KII questions. To counter this, interpreters were briefed on the goals of the KIIs and the meaning of each question, provided by the interviewer, before beginning any discussions. The availability of host organizations for KIIs also presented a minor challenge to the fieldwork when a host organization was no longer available and a replacement had to be found. Across 304 participant respondents representing 146 host organizations, such unexpected scheduling conflicts occurred with only 18 host organizations, seven in Kenya, alone. Scheduling challenges were due to national and/or religious holidays, political instability, and personal emergencies, which affected the respondents’ availability. In one instance, interviews in an entire region had to be canceled and replaced, in this case Kenya’s northwestern region, due to security concerns related to a contentious local primary election. All seven host interviews were replaced with alternate hosts. In this case, the host organizations were replaced with seven host KIIs closer to the capital, Nairobi. As a result, host organizations in Kenya’s most remote regions were not able to participate in the field visits as planned. 3.5 RESPONDENT CHARACTERISTICS AND DEMOGRAPHIC OVERVIEW Of the 978 IPs, host organizations, and volunteers who completed the web-based survey, volunteers were the largest group of respondents, as expected, with 603 responding. Across all respondent groups, the majority of respondents were male with this trend being most pronounced among active volunteers (see Table 5). When active and former volunteers are combined, hosts have the largest percentage of male respondents. Table 5: Web-Based Survey Respondent Characteristics Gender Male Participants N (%) Female Participants N (%) Prefer Not to Answer/Missing N (%) Total Number of Survey Participants Implementing Partner 59 (58%) 40 (39%) 3 (3%) 102 Active Volunteer 153 (75%) 46 (23%) 4 (2%) 203 Former Volunteer 247 (62%) 149 (37%) 4 (1%) 400 Host Organization 199 (73%) 63 (23%) 11 (4%) 273 Grand Total 658 (67%) 298 (30%) 22 (2%) 978 Source: Evaluation web survey 9 Volunteer respondents, as detailed in Table 5 above, consisted of both active and former volunteers from the current program cycle. These volunteers had a variety of background with the largest group joining the F2F Program from educational institutions (35 percent) and a significant percentage joining from agribusiness (16.6 percent) (Table 6). Table 6: What Was Your Primary Employment Previous to Your Most Recent Volunteer Opportunity With F2F? Type of Employment Count Percentage Cumulative % Agribusiness 96 16.6 16.6 Educational Institution Employee 203 35.0 51.6 Retired 74 12.8 64.3 Student 53 9.1 73.4 Farmer 55 9.5 82.9 NGO 42 7.2 90.2 Government Organization 37 6.4 96.6 Farm Cooperative/Association 17 2.9 99.5 Rural Financial Institution 3 0.5 100 Total 580 100 Source: Evaluation web survey Host organizations which participated in the web-based survey were similarly diverse. The largest of these groups included NGO host organizations, and farmer cooperatives or associations, comprising over 34 percent and 23 percent, respectively, of host organization survey respondents (Table 7). Table 7: What Type of Host Organization Do You Belong To? Count Percentage Cumulative % Farmer Cooperative or Association 62 23.1 23.1 Individual Private Farmer 20 7.5 30.6 Other Private Enterprise 38 14.2 44.8 NGO 92 34.3 79.1 Educational Institution 35 13.1 92.2 Public Sector Agency 16 6.0 98.1 Rural Financial Institution 5 1.9 100 Total 268 100 Source: Evaluation web survey 10 4.0 FINDINGS, CONCLUSIONS RECOMMENDATIONS 4.1 EQ-1: EFFECTIVENESS: HOW STRONG AND EFFECTIVE IS THE F2F PROGRAM THEORY OF CHANGE (TOC)? 4.1.1: How effectively do F2F Program activities (i.e., volunteer assignments) lead to desired impacts? To better understand the effectiveness of the ToC, the team first examined the components of the ToC and the activities, systems, processes, and variables that contribute to successfully generating the desired outcomes and impacts. By design, the F2F ToC describes the Program’s inputs, outputs, outcomes, and impacts in relation to the Program’s broad-based goal of increasing food security and economic growth in country agricultural sectors. In working toward this goal, the range of stakeholders involved with F2F includes: 1. IPs: Organizations who manage individual F2F country projects - inclusive of award management, recruitment and placement of volunteers - and liaise with hosts. 2. Host Organizations: Farmers, farmer groups, or more formal organizations that are directly identified and assisted by the Program and receive technical assistance from a F2F volunteer. 3. Volunteers: Individuals recruited by the IP to provide technical assistance to the host. All volunteer Scopes of Work (SOWs) are developed by the IP with assistance from a host. 4. USAID Washington and Country Mission: While F2F Program is centrally funded and managed from Washington, the Program communicates directly or through IPs with the local USAID office (“Country Mission”) to collaborate or communicate.The current ToC, as described by USAID F2F, is graphically portrayed below (Figure 2). Figure 2: Flow Chart - The F2F Basic Theory of Change (ToC) THEORY OF CHANGE Input •Learning through training, technical assistance, coaching •Source: IP Staff & US Volunteers •Cross-cultural & peer-to-peer exchange Output •Knowledge Transferred •Recommendations provided to host •Citizen diplomacy and expanded worldview Outcome •Adoption of recommendations by host •Behavioral Change, new management and innovation •Good will between countries Impact •Increased productivity & incomes •Improved social, human, and environmental capital 11 The components of the F2F ToC, as shown in Figure 2, are described below: • • • • Input: Volunteer technical support, with some additional staff support, lead to learning through mentoring, workshops, and hands-on demonstrations, and strengthen the human capital. Output: Volunteer recommendations and training leads to improved efficiency in agricultural processes and create opportunities to be successful. Outcome: Adoption of recommendation or host change in behavior, management practice, and/or technologies (innovation) lead to increase in profits and disposable income among the stakeholders. Impact: Productivity and income increases, resulting in improved social, human, and environmental capital. Data from the evaluation reveal that a key driver in terms of the strength and effectiveness of the F2F ToC concerns the spirit of volunteerism and the overall uniqueness of the Program. F2F is demand-driven in the sense that host organizations identify their needs and IPs find specialists who volunteer their time to help meet those needs. FINDINGS Since initiation, over 17,600 volunteer assignments have been completed in 117 countries. The total value of services provided by F2F volunteers is estimated at over $105 million. Approximately 1.4 million farmer families (representing about six million people) have been direct beneficiaries of the F2F Program. Volunteers help host individuals and organizations adopt new technologies and build local institutions and linkages to resolve local problems. So far, they have provided direct hands-on training to over 288,000 individuals. During the period FY 2014-2016, 30 percent of all volunteers and about 41 percent of all individuals trained by F2F volunteers have been female. In addition, the returned volunteers engage in public awareness activities in their communities to share their experiences and promote better understanding of international development issues and objectives. The F2F Program has been around for 30 years and the basic ToC and Program model has endured, unchanged. Survey respondents are in large part pleased with the Program, with over 90 percent of host organizations, and 92 percent of IPs very satisfied or satisfied with it.7 Meanwhile, 80 percent of volunteers were satisfied or very satisfied with the Program, and 75 percent of volunteers felt that the host activity was improved by their work and led to desired impacts, such as increased sales (e.g., 76 percent increase in sales revenues over one year), jobs created (300 jobs for one food processor), new markets (e.g., grocery store contract), and access to credit (because of business or marketing plan). The evaluation team asked KII respondents what drives their satisfaction with the Program. The prevailing view among all types of stakeholders, especially field-based IPs and host organizations, is that the F2F Program is: 1) needs-based; 2) market-driven; and 3) volunteer based. Host 7 A small but relatively significant 5 percent of hosts who were “very dissatisfied.” The KIIs revealed that one of the reasons behind the dissatisfaction was a poorly developed SOW.” We address this further in Subquestion 4.1.2. 12 respondents more often agreed that the fact that F2F technical assistance is provided pro bono was an added value because, as several participants reported, “[we work] without money getting in the way” of the relationship between host and volunteer and/or IP. Other drivers of satisfaction reported include: “F2F opened our eyes to new things. We now have a [business plan, marketing plan, new loan, new partners, greater sales, more members, outgrown our facility].” According to one KI, “F2F contributes complementary, supplementary assistance to our activities. Helped us launch and ultimately spin off a new commercial business.” – Host (NGO intermediary). Among KII respondents which noted some dissatisfaction, several reported that “‘free’ money comes with too many strings attached.” These respondents felt that “strings” include donors which are not “needs-based” and inclusive: they develop a project, offer it to a community that does not really need it, and expect complete cooperation and results. Speaking of the quality of the Program and its services, several KII participants indicated that, often, the technical skills required to fulfill their needs were not locally available, “even among local professionals with PhD degrees.” F2F was able to fulfill this unmet need, and co-create a plan to resolve it in a very practical way, for example a business plan and not an academic thesis, with timeline that can be deployed as soon as the volunteer leaves. Or, someone teaching the science and purpose of food safety, then working with participants to then practice with them what this looks like in real-life: for example practicing new, more hygienic ways of canning shelf￾stable fruit jams to avoid rotting and disease. Satisfied and enthusiastic hosts more often than not also reported that they were “working closely with the IP,” “consulted by IP throughout the process,” and “felt like a peer to the IP, not below them.” They overall distinguished F2F as unique and different from other development programs. For the KII respondents who expressed dissatisfaction with the Program, they cited issues like poor communication, which resulted in poor SOW development, poorly prepared volunteers, poorly scheduled assignments, or complete misunderstandings such as what day a volunteer is arriving and whether there is a vehicle for them. There was also potential for duplication - as the evaluation team learned from the USAID Mission KIIs - when IPs were working with the same organizations as the Mission’s projects, but were both unaware of it. This led to wasted resources, time, and opportunity cost. These logistical and operational issues have affected somewhat the effectiveness of the ToC (see Question 4.1.2 and 4.1.4 for more on this). CONCLUSIONS F2F is an effective, low cost program that makes use of technical services to enhance sustainable economic development and produce opportunities for productive people-to-people level exchanges. The Program’s longevity is tied to both its core philosophy of peer-to-peer volunteer-based sharing of learning, and the ensuing citizen diplomacy and cultural exchange. These, combined with the Program’s economic impacts - increased sales, access to credit, and new markets and contracts – have contributed to sustainable change. The common thread tying together F2F’s core strategy is that it is “needs based” and “flexible,” even though the needs and the operating environment change over time. F2F is able to quickly adapt, because this is built into its strategy. 13 F2F stakeholders are, on the whole, satisfied with the ToC. The causal relationship from input to impact is often nonlinear and variable depending on context (e.g., country, IP, or available resources) and on the activities implemented to support the ToC (e.g., systems, processes, templates standardized for recruitment and placement). Other variables that can inhibit achievement of desired results include: IPs’ vision/mission, technical approach, and management/operational effectiveness; priorities defined by the host organizations; socio-geo￾political and economic conditions in focus countries; timely availability of expert volunteers; alignment of visions, missions, and needs with donor priorities; expected designs; and donor funding. RECOMMENDATIONS The basic ToC has been around since 1985, is strong and does not need to be changed. It could, however, be made even more efficient and effective (resulting in less human, financial, and opportunity costs) if intentionally viewed from a systems, networks, and processes perspective – where activities, people, and resources are interconnected and inter-dependent. The evaluation team, therefore, suggests a much broader ToC that includes the processes, support systems, and networks required to generate the desired impacts working within the parameters and budget of an IP. Visualizing this, and discussing this with IPs, is an important step toward identifying weak or missing links in the TOC that, if repaired or strengthened, could better lead to desired impacts, while reducing the costs related to inefficiencies and wasted time and resources. The evaluation team recommends that this be explored further in the next F2F Implementers Meeting, with the assistance of a facilitator who understands the Program. Together, IPs can finalize this broader ToC and begin to capture the ways different Program elements work individually and collectively toward the desired Program impact. This will also codify the sharing of learning and innovation, because IPs will be obligated to report on the processes they follow for the impacts they report. 4.1.2: How effectively are F2F interventions (country projects, hosts, host development plans, volunteer SOWs, etc.) selected and developed to lead to change? FINDINGS Although this question was difficult to assess from a survey, the evaluation team was still able to collect evidence that suggests the F2F interventions are effectively selected and developed to lead to change. Volunteers are matched to projects that fit their skills (96 percent of volunteers are very satisfied or satisfied on this point), and most of them (84 percent) complete their deliverables. Ninety percent of the hosts are very satisfied or satisfied with the volunteers’ involvement in developing the SOW, stating that SOWs took their priorities into account. Overall satisfaction with the project and the positive impact of the project on the hosts’ work, performance, sales, and markets (as indicated by hosts adopting most recommendations as detailed below) also indicate that interventions are effectively selected. During the KIIs, the team examined the steps the IPs take to select country project focus areas and hosts, as well as the steps involved with developing a plan with the host and individual 14 SOWs. Field-based IPs expressed an assortment of viewpoints on how a country project is selected, and some simply did not know how the country project came to be, because it either preceded their involvement with the Program, or they were not involved in its design. Interviews suggested that IPs in the field work with IPs at HQ who may visit the country periodically to, as a starting point, understand the USAID Country Development Cooperation Strategy (CDCS) and/or USAID Feed the Future 5-year Strategy, in an effort to find opportunities to add value or fill knowledge gaps in USAID-Mission programs. Identifying and cultivating a successful synergy between IP and Mission appears to depend on who is representing the IP to the Mission and on who the point of contact is at the Mission. For example, if the contact at the Mission is not interested in agribusiness they may be less motivated to partner with the IP. Or, if the IP representative is unable to communicate with effectiveness and consistency, they may not gain the support of the Mission. Others IPs take stock of what development actors are working in country, and in what areas of intervention, and will work in the same region if there is a relevant, identified need for F2F support among actors. Or, the IP will work in a different region if its work would be redundant or if there are unmet needs in another region. During the evaluation team’s field visits with KIs, nearly all host organizations mentioned having worked with either another USAID-funded project or with another international or local development agency. International agencies include United Nations Development Program (UNDP), International Fund for Agricultural Development (IFAD), and Germany-led GIZ.8 More than a few respondents throughout Africa KIIs reported working or having worked with Japan International Cooperation Agency (JICA).9 Local country-led agencies providing support include the regional/local Chamber of Commerce, Bureau of Food Safety and Hygiene, and Ministry of Agriculture Extension Service – or the local equivalent. Some IPs were well-networked and had positive relationships, thanks in part to being proactive, which had a positive and distinct effect on the outcomes. For example, with IP HQ support, the IP starts in Year 1 of their 5-year project by identifying all of the international development actors working in country, and consulting with the USAID Mission, which more often than not (according to KIIs) will gladly share the lists of projects and names of organizations they are working with. These IPs look for synergies and win-win solutions (addressing gaps or weaknesses in one another’s projects), and are seen as a help not a hindrance. Another example is IP country staff who update the Mission periodically on what volunteers are in the pipeline, and schedule debriefs with volunteers and the Mission. On the other hand, as the evaluation team learned from USAID Mission KIIs, some IPs are just not communicative; in some cases, the Mission does not invite communication, but in cases where they welcome it, it is up to the IP to reach out and stay in touch. In one example, the IP would email the new contact (potential partner or USAID-Mission) and wait until they responded, which could be never. One IP told the evaluation team that “the USAID Mission is not interested in our work,” while when interviewing the Mission, the comment was the IP has not been proactive about reaching out. Further illustrating this poor communication and the lost 8 Germany GIZ: https://www.giz.de/en/html/about_giz.html 9 Japan International Cooperation Agency: https://www.jica.go.jp/english/ 15 opportunities that accompany it, both the USAID Mission and the IP were working with the same host organizations without realizing it. KII respondents most often cited good host selection as the key to developing effective F2F interventions. A few participants mentioned having host selection criteria; while others described having an “intake” form that the host organization fills in and emails or mails back to them, in which they provide basic information such as company/organization type, size, crops, scale, members, needs, and sales. These processes help them eliminate host organizations that are not a good fit, or those that are not going to be proactive or embody self-help attitude. One IP summarized this by saying, “Host selection. It all starts here. We go to the field to assess the host first, to make sure they are clear on the F2F objectives. Sometimes they need money, and we have to focus on those who need knowledge.” Not all IP respondents interviewed could explain their system for host selection, while some not only described it but also mentioned the various templates and documents they have developed to systematize the process, including host selection criteria, host intake questionnaire, host development plan, and simplified forms for collecting baseline and M&E data. They mentioned that these documents are part of recordkeeping that helps keep hosts and partners accountable, makes terms clear, and facilitates data collection and M&E. The starting point for SOW development is the needs assessment. For farmers and host organizations with staff that have a higher level of education or understanding of the process, the needs assessment is either led by the host organization or done jointly with the IP. Following the needs assessment, IPs and the host organization develop the SOW together, come to an agreement, and then work with the IP’s HQ for recruitment. In some cases, neither the host organizations nor IPs have the technical capabilities to determine what the true need is and how to prioritize this. One savvy host organization hired a consultant to work with them because the IP did not have experience in the sector in question. With the exception of individual farmers with limited education who may not have seen a SOW document, all agreed that the development of the SOW was essential and that there are different methods and best practices to develop a good SOW that both meet the needs of the host organization, and also make recruitment, M&E, and other IP functions much easier. Some IPs, for example, had a standard form for the SOW, as well as data sheets to get content from the host organization to fill in the SOW. The evaluation team spoke to several host organizations that have been able to scale up and grow. It found that they had received numerous volunteers along a continuum, where a needs analysis was conducted across the host organization’s project/business/service and value chain, and identified and prioritized the most critical gaps in order for the host to advance and grow its activities, sales, etc. Host organizations in all countries mentioned the need for a sequence of “Most donor projects, they sit in an office and design the project and then come and implement it with you. Sometimes people are pushed on you and you’re not ready. For example, you need marketing, but you need certification first to get there. F2F is different. F2F sat down with me, got to know my needs; the design came from me, not them. I am the boss, not them.” – F2F Host Organization 16 interventions and SOWs to repeat activities so as not to have a standalone activity with no follow on. The evaluation team provides more on this in Subquestions 4.1.5 and 4.1.6. CONCLUSIONS Both host organizations and IPs emphasized the importance of a well-crafted SOW to address the needs of host organizations. As some host organizations told the evaluation team, “we know what our needs are,” and what is required is guidance to navigate them. Therefore, a needs “analysis” that asks thoughtful questions to better understand the situation in order to co￾design a proposed solution (SOW) is important. Selecting effective F2F interventions depends on a combination of factors, including: 1) a solid country project, developed in partnership, as peers, with hosts; and 2) IPs forging and maintaining good relationships with other international and local development actors, including the USAID Mission. By working with host organizations that have received support from other development actors, F2F is able to leverage those investments to fill specific gaps and needs to catalyze or expedite the desired change, while also leveraging the human resources and knowledge acquired by the host organization from other actors prior to F2F involvement. The most successful IPs (U.S. and overseas) are communicative and collaborative. The evaluation team did not see this behavior across all country visits; there were differences between how one in-country IP operated from another, which served as points of comparison for the evaluation team. Not all IPs perform equally, have the same resources, or are as equally strategic. Strong, technically qualified country staff and home office support staff are essential for successful assignments. Having other projects in country allows an IP to leverage other activities and funds to make F2F interventions more effective. “Being strategic” is part of the way successful programs work. Success and effectiveness start with a clear sense of the development problem (and precise need), a vision and strategic plan. In some cases, IPs identified host organizations and volunteers without any long-term plan, with relatively poor results. More often than not, KII hosts expressed opposition to “one off” assignments, where they receive one volunteer a year, the SOWs are not related or connected, and there is no follow up when the volunteer leaves. Hosts that expressed the most enthusiasm and had demonstrable impacts (e.g., sales receipts/records, and physical assets [factory, silos, trees with fruits, machinery]) also appreciated how effective the F2F interventions are when they are co￾created together with the IP with longer-term economic sustainability in mind. These more successful host organizations were coached and accompanied by the IP field staff to identify and prioritize, together with the host, a longer-term approach and sequence of interventions across the value chain to strengthen it and fill gaps, thereby unlocking sustained economic growth. RECOMMENDATIONS The evaluation team recommends that SOW development be a collaborative affair with as much input from the host organizations as possible. IP staff noted that when they approach the SOW this way, they have greater buy-in and accountability from the host organization. From the host organizations, the evaluation team heard that they feel like they are peers, co-designing 17 something new, and this can be very empowering as it builds their capacity for self-help and resilience during and after F2F intervention(s). The needs analysis should explore the business plan of the host organization and include a SOW with multiple volunteer visits and more than one volunteer. SOWs need to have not only an implementation plan but also a monitoring plan to be able to demonstrate change. Where appropriate, such efforts by volunteers could be formalized as a deliverable in their SOWs. This would encourage forward planning on the part of the volunteers as well as an additional data point for host organizations and IPs to compare against their own long-term thinking. Related to the above is who conducts this needs analysis. It is worth investing in higher level staff for this initial visit because a qualified technical staff member who knows enough about the subject/sector/value chain will ask the right questions to help the host organization identify its real needs and then to prioritize them and sequence them. Using an unqualified, low-level staff member can produce a misunderstanding of the host’s real needs resulting in poorly designed SOW and mismatched volunteers, as the evaluation team discovered during field visits. When the right staff is not available, consider hiring a consultant specialist if financially feasible. It costs less than the price paid for disappointed partners and soured relationships. Another option to ensure that the SOW responds to what is needed, exactly when and how, is for volunteers to assist with SOW development for the next volunteer. Many IPs are already doing this, and the result is a well thought-out strategy for a series of SOWs over the course of the Program that is then fulfilled by the volunteer him/herself or by new or repeat specialized volunteers. The benefits include cost savings and ground-truthing of strategies designed by the IP with the volunteer playing the role of honest broker that says, “This won’t work, but this will.” The team recommends that F2F explore this option further, including the economic impact, returns, and metrics related to sequencing volunteers across a business lifecycle and value chain versus training large numbers of individual host organizations. Based on the successes witnessed, the evaluation team believes the sequenced approach is worth reviewing. In other words, ensure that there is a plan for multiple volunteer assignments, if merited. This could be structured, for example, as a SPSP Special Study. 4.1.3: How effectively are volunteers recruited, selected, matched to SOWs, motivated, and prepared for their assignments? FINDINGS The evaluation team found that volunteers are effectively recruited, selected, matched, motivated and prepared for their assignments, as summarized in Table 8, next page. 18 Table 8: Volunteer Satisfaction Recruitment well organized (N=573) Project matched skills (N=574) Preferences taken into account when matching with a project (N=574) Strongly motivated for duration of assignment (N=574) Received all requested information (N=573) Felt well prepared before assignment (N=573) Strongly disagree 2.27% 1.74% 1.92% 1.92% 2.62% 1.75% Disagree 1.40% 0.35% 1.05% 2.44% 4.54% 6.11% Neither disagree nor agree 3.32% 1.39% 2.96% 4.18% 5.24% 5.76% Agree 33.16% 19.16% 19.86% 20.56% 32.29% 34.73% Strongly agree 56.54% 76.83% 67.25% 70.56% 54.62% 51.66% Not applicable 1.92% 0.50% 6.45% 0.17% 0.52% 0.00% Don't know 1.40% 0.00% 0.52% 0.17% 0.17% 0.00% Volunteers were particularly positive when asked if the project matched their skills, with nearly 96 percent of them agreeing or strongly agreeing that this was the case. Over two-thirds of volunteers also strongly agreed that their preferences were taken into account when matching them with a project. Motivation was similarly high among volunteers, with over 80 percent feeling strongly motivated for the duration of their assignment. Volunteer satisfaction, while still high, was relatively lower in terms of recruitment, receiving the information they requested, and their preparedness for their assignments. Over 7 percent of volunteers disagreed or strongly disagreed when asked if they received all the information they had requested before beginning the assignment. Similar to this, nearly 8 percent of volunteers did not feel well prepared before beginning their assignment. Volunteers were also relatively less positive on the organization of the recruitment process, with only 57 percent strongly agreeing that the recruitment was well organized. Through discussions with IPs, the evaluation team found that the recruitment process faces staffing and resource constraints at the IP level, and capacity constraints among hosts. As currently implemented, IPs’ headquarters perform all initial volunteer recruitment. However, field staff performs some “light” recruitment for repeat or follow-on volunteers. Most IPs have a dedicated volunteer recruiter position. For the few IPs with no dedicated volunteer recruiters, IP staff share volunteer recruitment. This can be challenging if one is also leading and managing the IP’s F2F Program. It can be an even greater constraint on recruitment efforts when the IP is a small organization with few or no other programs to leverage, is resource challenged, and has to make decisions requiring tradeoffs. 19 The process of selecting and placing volunteers follows a similar pattern, as observed with the development of the SOW with regard to host organizations. Namely, for farmers with little education and operating less formally, the IP reviews Curriculum Vitaes (CVs) and chooses the most appropriate volunteer. The individual farmer never sees or reviews any CVs and is never consulted on volunteer selection. For farmers at a higher level of education or comfort, the IP reviews CVs and then chooses a number of them who appear to be the most appropriate. IPs share these CVs with the host organization for them to scrutinize and make an informed choice of potential volunteers. Once the host organizations have voiced their desire for a specific volunteer, further discussions take place with F2F to ensure that the most appropriate volunteer is selected. For larger organizations, a similar pattern is followed. Again, once the selection of the volunteer is made, the type of host organization will determine the next steps. For farmers with little education and operating less formally, the IP communicates directly with the volunteer to establish the program, the length of assignment, and the potential schedule for the volunteer. The volunteer is provided with information concerning the host organization, the local conditions, and any other relevant information. The farmer is told what the program will entail and consulted to ensure that the scheduling is compatible with his/her workload. Otherwise, there are no further consultations or discussions with the farmer. A much more involved process occurs when dealing with farmers at a higher level of education and/or larger organizations. In this instance, the IP communicates with the volunteer, establishes a schedule and program, and provides some information to the volunteer. The farmer/organization, if language is not a barrier, is encouraged to communicate with the volunteer via Skype, if possible, or, at a minimum, via email. Even if language precludes direct communication between the host organization and the volunteer, the IP will intervene to ensure there is an open channel of communications. These discussions result in adjustments to the SOW, refining of the training program, and any other changes deemed necessary to have the highest potential for successful interventions. In some rare cases, IP field staff stated that they felt that HQ sometimes will try to “sell” a volunteer for an assignment because they were unable to find the appropriate individual. Some host organizations voiced the opinion that CVs many times looked “generic” and, therefore, it was hard to assess skills and distinguish between potential volunteers. Similarly, in some rare instances, host organizations revealed that the volunteer recruited did not meet their needs. A small but relatively significant 5 percent of hosts, were “very dissatisfied” with the IP’s effectiveness to field a qualified volunteer who aligned with their needs and goals and, hence, they were disappointed in the volunteer and the assignment overall. The KIIs revealed that one of the reasons behind the dissatisfaction was a “poorly developed SOW,” which speaks to the importance of effectiveness in developing F2F interventions, as addressed in Subquestion 4.1.2. Each host organization has a different set of needs and approaches, depending on the type of entity. There is a range of host organization types and backgrounds. The variety of respondents to the web-based survey illustrates this point with respondents including host organizations 20 from NGOs, farmer cooperatives, private enterprises, and public sector agencies among others. With a diverse group of host organizations, the assistance they received was similarly diverse. As seen in Figure 3, these included assistance related to technology transfers, business and enterprise development, organizational development, and environmental conservation among others. Figure 3: Assistance Received By Host Organizations Source: Evaluation web survey While the core program and goals remain consistent with what they were back in 1985, the finer points and objectives have adapted to changing needs and the reality of 21st century farming and globalization, which requires farmers to learn new technologies and engage in the business side of farming to survive and thrive in a modern farm economy. For example, the types of technical interventions, host organization needs, and volunteers recruited have shifted over time. Whereas, previously, volunteers were very production, on-the-farm oriented, now they represent less than half of the volunteers.10 This is most likely due to targeting more commercial farmers and host organizations over the years. For those who are ready to scale up, knowledge becomes a catalyst to advance them. These days, volunteer assignments include business service providers such as product development, finance, accounting, management information systems, small business development, and digital marketing (see Figure 3). For example, some host organizations are food processors that purchase crops in bulk and then process them (ex. tomatoes, mangoes, fruits, etc.). Thanks to volunteer support on business planning, a sequenced approach that walks host organizations through start up and growth, specific business planning, branding, and financial accounting and costing and pricing, host organizations have been able to grow and expand their operations. Sometimes the need was too specific, was misunderstood by the IP, or the volunteer was too much of a generalist. For example: one host stated that “I got a farmer volunteer but he didn’t know about my specific area (crop, method, such as organic or green housing), spent the whole week learning, and flew out by the time he was ready to help.” 10 USAID FY 2016 Annual Report Indicator Tables. 21 Sometimes the need was too specific, was misunderstood by the IP, or the volunteer was too much of a generalist. For example: one host stated that “I got a farmer volunteer but he didn’t know about my specific area (crop, method, such as organic or green housing), spent the whole week learning, and flew out by the time he was ready to help.” A number of F2F in-country implementers stated that their first consideration is experience, and second is whether the volunteers can cope or adapt in a rural setting and developing country. This is why repeat volunteers are the most desirable because they already know the lay of the land, understand the required assignment, and may be performing a follow up to a previous assignment. Host organizations are especially happy to get repeat volunteers, citing that they will already have a context for the situation and are able to provide follow on assistance that helps the host organization advance (or “graduate”) along a continuum of organizational or business growth. CONCLUSIONS Streamlining the recruitment process across all IPs is needed, especially with respect to the sharing, vetting, and recording the quality of volunteer performance after they have served in the program. Some IPs see their contacts database as proprietary information and hesitate to share that information. Other IPs do not have a dedicated volunteer recruiter, which means that the job is undertaken by multiple staff. The successful recruitment of the right volunteer also depends on having crafted a good SOW. Volunteers report high levels of satisfaction with regards to the project matching their skills, preferences, and motivation over the course of the assignment. Notwithstanding, volunteer recruitment, provision of information to volunteers, and volunteer preparation can be improved. The volunteer process is uneven across IPs and hosts and depends on the IP’s staffing, resources and capacity. Hosts with less capacity are especially likely to be marginalized during the recruitment process, while IPs with fewer resources, projects, or staff may struggle with recruitment. These factors are complicated as well by IP HQ occasionally struggling to find an appropriate volunteer. These cases, in turn, can lead to poor matches, affecting the assignment’s impact. Further, there is very little sharing of contacts or contact databases between IPs working in different countries. Assignments may be less effective for volunteers who lack information or who feel unprepared before their assignment begins. Given the varying nature of F2F assignments, having both returning and first-time volunteers thoroughly briefed before beginning an assignment, is very important for maximizing the impact of their work. The prevailing view among KII host respondents is that sufficient time is needed for the volunteer to interact successfully with the host organization in advance of arrival in-country, and that it is necessary to improve the quality and relevance of volunteer training, including materials and tools provided as part of the training. 22 RECOMMENDATIONS IPs at the HQ level are keeping their own sets of contacts and there appears to be little sharing of contact information between the different IPs. Hence, a volunteer who has performed poorly is not flagged by the IP that fielded him/her. At its core, F2F is a recruitment program, and from a business perspective, it makes sense that IPs would be hesitant to share information about volunteers with other organizations with whom they are competing for donor funding. At the very least, F2F should keep a central database of volunteers, and work with IPs to create a system of grading volunteer performance and otherwise recording volunteer profiles IPs should be encouraged to share information about volunteers to ensure that appropriate volunteers are hired by other IPs and to see where else they may have served, and with which IP. To solve the issue of confidentiality, USAID F2F should keep a central database of volunteers. F2F should then work with IPs to create a system of evaluating volunteer performance and otherwise recording volunteer profiles. The issues of time and timing should be top priorities to maximize knowledge and learning exchange opportunities in the limited time and resources available. F2F should address them in a deliberate way by including time/timing/time allocation as a topic for future IP workshops, and discuss all the ways that time needs to be considered to improve the Program’s effectiveness. F2F should review the steps, activities, and transactions that occur, where and with whom, and map the links and processes as part of the larger ToC exercise (above). To address the time constraints, gain efficiencies and to ensure that volunteers are well prepared for their assignment, IPs should include a step to connect the volunteer and host organization, to enable them to not only get to know one another but to also allow hosts and volunteers to discuss technical aspects of SOW, to better prepare for the work they are about to do together. 4.1.4: How effectively and efficiently do volunteers complete SOWs and provide desired deliverables? FINDINGS Through the KIIs, the team discovered that host organizations are generally happy with the work of volunteers and find that volunteers do effectively and efficiently complete their SOWs. Much of this is due to good planning and preparation of the volunteers for the assignment, which they credit the IP for doing. The consensus among hosts is that the volunteers come prepared; they are highly skilled, hands-on and practical, and ready to jump in and work hard to complete their SOW. Although connecting volunteers and future hosts is not a standard practice, several IPs are making these introductions, although it is usually the volunteers who initiates the discussion when they write the IP with questions about their SOW. Host respondents are convinced that this improves the quality of the assignment and saves time to enable the host and volunteer to maximize the learning and knowledge transfer during the limited assignment duration. And when volunteers have a better sense of the country/regional context and what resources and materials are locally available, they come better prepared and design training that is most relevant to the situation. 23 In cases where the SOW is not well thought out or done completely, the host may be disappointed, due to misunderstanding or an unrealistic SOW, which can damage the relationship and the reputation of the IP. The web-based survey suggests that 16 percent of volunteers did not complete their deliverables. When probed, the field interviews revealed that incomplete assignments were primarily the result of time constraints (poor time use or not enough time) or because the training provided was not relevant to audience, whether topically or through delivery of the training. This is also supported by the findings of the web survey, where one-third of volunteers cited the length of the assignment as the cause for not completing all deliverables (Figure 4). Figure 4: Inhibiting Factors to Completing Deliverables Source: Evaluation web survey (Most common “other” responses include: still working on deliverables currently (N=6), host unprepared (N=4), and deliverable not possible given project’s stage in lifecycle or stage of development of host organization (N=2)) Sometimes, the time issue was not due to the volunteer but rather because of poor planning by the IP. For the 17% of volunteer survey respondents who indicated they lacked support from the host organization, only two (of these 16 respondents) mentioned hosts in their open-ended response. Time constraints, however, prevented the evaluation team from speaking directly to these volunteers to collect more detailed information. Time challenges may also be an inherent challenge to the Program, as F2F uses highly skilled and experienced volunteers who have limited availability. In many cases, host organizations suggested that IPs do as much as they can to prepare the volunteer in advance. Where effective, it could buy a few extra days of quality work with the volunteer in country, especially when a volunteer better understands the local context, the specific issue, and has established a rapport with the host organization. It seems to be ad hoc, but not required, for a volunteer to be in touch with the host organization in advance. In all cases the evaluation team witnessed, volunteers who contacted host organizations before their arrival or sent questions by email to the IP (and then had some direct contact) were far better prepared for the assignment. One IP 24 local field staff mentioned that he spends a lot of time on questions and answers with volunteers. He translates an email with questions into French, then takes the responses, and translates them back to English, which can be very time-consuming. According to some KIIs, this is one of the reasons why some IPs do not spend enough time on preparation. According to the results of the survey, logistics were singled out by 37 percent of the IPs as the largest challenge for participation in the Program (see Figure 5). Figure 5: Challenges Experienced by IPs The evaluation team focused on transportation logistics and lodging in its KIIs and found transportation was generally not considered a challenge except when there were multiple volunteers in country at the same time. In most cases, the IP provides transportation (vehicle and driver); the evaluation team confirmed this in all but one case during field visits. Offices with fewer staff and vehicles sometimes have to hire vehicles, or use their own personal vehicles when a safe, dependable alternative is not available. When asked if the staff is reimbursed for this (ex. mileage, fuel), they said they were not, and bringing it up would be “more trouble than it’s worth.” Lodging is also usually provided by the IP, for various reasons, and in cases observed by the evaluation team, volunteers are not allowed to choose their own hotels and travel can sometimes be limited, both due to security risks. F2F prioritizes the security of its American volunteers, and has increased measures to screen and monitor volunteers as per the advice of GAO Evaluation of F2F Program 201511. The second issue with lodging is budgetary. Making the most of the IP’s country project budget is also a consideration, and by establishing agreements with hotels, the IP is able to secure a discounted rate for F2F with an hotelier it can trust. All of this makes it easier to monitor and ensure safety of volunteers. A third reason for the lodging is 11 “USAID Farmer-to-Farmer Program Volunteers Provide Technical Assistance, but Actions Needed to Improve Screening and Monitoring” (GAO-15-478) Statement of Actions Taken and Planned – F2F response to GAO Evaluation 2015 25 to ensure that volunteers have a standard level of hygiene and also access to utilities – e.g., electricity and internet access - necessary to effectively perform their work. The web-based survey indicated that language and communication with the host was the next largest challenge, with 32 percent of IPs citing them. However, the finding from the in-country IPs or host organizations was that these were not major problems. This requires further investigation. Many times, there was a need for a translator, but this was generally provided by IPs’ staff; when staff was not available, an interpreter was hired. The evaluation team did learn from host organizations and IPs that the translator plays a very important role. The translator and the volunteer form a kind of partnership, and the translator needs to interpret the language, technical, and culture to translate it in the most relevant way to the audience. Basic utilities can also impact an assignment, such as when there are frequent power outages and limited or no Internet access. Even when a volunteer has modern amenities, they may not always be functioning when the volunteer needs them to be. Time, timing, and time allocation are the most frequently cited barriers to a good assignment. When volunteers stay at a hotel in town 30 miles away, they must drive back and forth to the village. This eats up time that could be spent on training and informal exchange to enrich the experience. Instead, it ends up being seen as very transactional and impersonal, where a volunteer comes in, sets up, teaches, and leaves. Duration of the assignment was also repeatedly mentioned as an area for improvement by volunteers—as shown in Figure 4 above—and by IPs through KIIs. The evaluation team discovered some assignments were only one week in length. In some cases, it was only four days. For one IP, a 17-day average assignment length is a satisfactory average for an effective assignment with, in rare exceptions, a minimum assignment duration of one week. If there are multiple host organizations, they require assignment duration of at least 15 days, and they try for longer. “Shorter assignments would require a lot of preparation and have to be very specific,” stated one host organization. For some IPs, there is perceived pressure (from USAID) to reach a targeted number of assignments and quality is the trade-off. The pressure of reaching targets is another issue for IPs, causing some to field volunteers for very brief assignments, to be able to report on the work. When visiting host organizations, the team found length of assignment ranging from just four days to over a month. In the case of the four, five, or seven-day assignments, host organizations found the duration too rushed; it didn’t allow time for reinforcement of new learning. CONCLUSIONS How effectively and efficiently volunteers complete their SOWs depends on several factors, including timing and logistics. For F2F, logistics involves all the processes and steps involved with recruitment, management, placement, health, safety, transportation, security, and follow up with volunteers, as well as the steps in between. Difficulties related to local language and culture also pose logistical challenges and thus can also adversely affect volunteer performance, by hindering or slowing down the knowledge transfer process. Several KII respondents mentioned that translation slows down the knowledge transfer (“you need twice as much time or 26 more,” the team heard repeatedly when working with an interpreter who is translating from volunteer to host back to volunteer and also handling questions and answers.) Since host organizations vary in size, scope, and resources, some are able to contribute more than others to the volunteer assignment. Relatively cash-strapped hosts find it more difficult to cost share, even basic needs such as refreshments during volunteer training, which can adversely affect the quality of the assignment. Some host organizations budget for coffee breaks, snacks, and a facility. Few, however, can afford to offer lunch, so either everyone works through hunger, or they break for lunch with fewer attendees returning after lunch. RECOMMENDATIONS With regards to time constraints, the nature of the Program makes it a challenge to field highly qualified and experienced volunteer consultants who can take the time out of work and family life to serve in the Program; even one week away from the U.S. can be difficult for some volunteers. However, the issue of time constraints should be a top priority to maximize knowledge and learning exchange opportunities. The evaluation team encourages F2F to address this in a deliberate way by including time allocation as a topic in future IP workshops and to discuss all the ways that time needs to be considered to improve the Program’s effectiveness. One example that could be implemented immediately by IPs is to build in more time up front for host organization and volunteer preparation and communication; this can be done via email, Skype, Phone, and/or What’s App. For those who do not already do this, the IP should build in a step to directly connect the volunteer and host organization, and allow them both to prepare better for the work they are about to do together. Sometimes, a translator will be required. This extra up-front time is well worth it down the line as frees up more time in the field, thereby leading to greater development impacts, in addition to a higher quality assignment and increased rapport between stakeholders. The funding of activities should be worked out in the SOW so there is a clear appreciation of the required logistics and how they are being handled at the host organization-level. The IP should provide small funds for lunch or refreshment preparation at the host organization-level, especially in cases where trainees are walking several kilometers to the training or volunteers are driven back and forth to village at great distance (and are too poor to afford lunch). The cost of implementing recommendations was the number one barrier to host organizations adopting a volunteer’s recommendation. Tied to this is the amount of time it takes for host organizations to generate funds, even if they self-finance. To keep morale and momentum going, USAID could allow IPs to budget a small portion of each program budget to funding ideas that cost $200 or less. Another idea observed during the fieldwork is a volunteer or IP linking the host organization to loans and investors; an American IP or volunteer may provide the credibility to get the host organization in the door. Volunteers and IPs can also help the host organization to develop a business plan or investment proposal (such was the case when a Peace Corps volunteer helped prepare a World Bank proposal to fund the procurement of jars for women food processors). 27 4.1.5: To what extent do hosts change behaviour and/or perceptions as a result of F2F interventions? FINDINGS Volunteers typically provide end of trip reports with recommendations. They may also leave the host organization with a set of recommendations or a documented plan (e.g. business plan, production plan) upon departure. Most recommendations are carried out because they are simple and of little cost, and volunteers work with what is feasible. Some benefits were evident and observed first hand, such as a fruit trees with an abundance of fruits, thanks to a hybrid grafting method recommended and taught by a volunteer a few years ago. Among host respondents in the KIIs, few were critical of the recommendations provided by volunteers. From the web survey, the evaluation team found that 60 percent of hosts say they have adopted more than half of the volunteer’s recommendations (see below for examples). For the remaining half of recommendations not yet adopted, the reason was because they either require prerequisite activities from prior recommendations (listed in their trip report), the timing is not yet right for the recommendation, they have not yet generated the financial or nonfinancial resources like land to adopt the recommendation, or external factors like family emergencies have prevented progress. From the field visits and KIIs, the evaluation team was able to observe some changes first hand, after being provided with the “before and after story” by the IP and host (and baseline data). Examples of behavioral changes are plentiful, including: Natural Resource Management – soil science assignments that lead to new production practices like composting, mulching and inter￾cropping, which in turn lead to less money spent on fertilizers and pesticides, less labor, and higher yields in dry climate. Other changes include the way a host conducts business; for example, a marketing plan that a host co-designed with a volunteer led to their new ways of networking and promoting their farmer-owned village bank to a wider region, and their membership increase from 500 to 700 members. In addition, they have been able to increase the maximum value of capital loans they can provide from roughly $400 to nearly $1,000. Positive changes in food handling and hygiene enable many F2F hosts who are food processors to become formalized and acquire food safety certifications to do business at scale, in larger markets. Figure 6, next page, presents a breakdown of factors that influence adoption of recommendations, as expressed by host respondents in our survey. Among the hosts surveyed, nearly two-thirds cited lack of financial resources as a factor limiting their ability to fully implement all of a volunteer’s recommendations, while 30 percent cited lack of local access to goods/services as the cause. Only 4 percent of hosts stated the recommendations were not practical or helpful. The KIIs confirmed these findings. The KIIs also revealed that overall, host organizations were satisfied and, more often than not, enthusiastic about the recommendations made by volunteers, whether they required additional funds or not. Several respondents “We take our volunteer recommendations seriously, and when we had challenges implementing them, the volunteer helped us identify potential funder/lenders for the recommendations.” – Host Organization 28 expressed that “most of the volunteer’s recommendations were low cost recommendations,” and therefore “no problem to implement.” Figure 6: Factors Influencing the Ability to Implement Recommendations by Host Organizations Examples of low-cost recommendations included recommended changes to a process or practice [“host organization should begin tracking its transactions (e.g., inventory, accounts payable), with suppliers as well as clients in a logbook, on a computer,”] using a template developed by the volunteer during assignment. Or, “host organization should handle and store crops/ingredients/products in such a way as to avoid cross-contamination.” The lack of financial resources appeared to be the most acutely constraint for those organizations which were ready to scale; i.e. they had outgrown their equipment and methods, and needed additional space, equipment, or labor to implement. In several instances, the host organization was qualified for local financing, but the financing was not enough. “Our needs are too big for microfinance; $5,000 is nothing for us, but that is all they offer. We need more, but traditional banks will not lend to us,” said one such host. Another host took a year to gather the funds required, which impacted timelines and productivity. In another case, a host organization had the funds, they were able to acquire a low-interest loan with a two-month repayment grace period. However, the equipment they ordered using the loan, could only be procured from China. After ordering and paying, it took more than three months for it to arrive, which resulted in additional interest fees and a near default on the loan. For host organizations that are formalized and beginning to scale (e.g., small scale food processor to factory hiring employees), the recommendations become more costly when equipment or certifications are required. Several host respondents have either asked or been offered by the volunteers support to identify local and international partners, lenders, or donors, with some success. For others, the volunteer did the ground work, identifying and connecting the host to local and international development agents who can help them move forward with recommendations. 29 CONCLUSIONS F2F depends on two key steps: 1) identifying and structuring good assignments; and 2) identifying volunteers capable of responding to the needs of the host organizations and transferring their knowledge in the most relevant way. This, combined with selecting the right host (e.g., proactive, motivated, self-help attitude) and understanding the host’s capabilities (e.g., financial and other assets), impact the extent to which a host adopts recommendations that lead to the behavioral changes and desired impacts reflected in the Program ToC. F2F is changing the behavior and perceptions of hosts, with regards to their work. The biggest challenge appears to be less about motivation and more about practical aspects of adopting a recommendation, like money and time. Volunteers and IPs can be of particular support here in terms of coaching, described further below (see Recommendations). In this way, the host is receiving the capacity development it needs to enable it to address barriers and issues as they arise, and empower it to become more self-sufficient in problem solving, and identification of new partners and support outside of F2F. Most host organizations are “small businesses” or “small farms” with staff sizes of less than 50 people, and farm size no more than an acre. Often, such organizations do not have the human or financial resources to implement the recommendations. Even if a host organization sees an economic benefit from a recommendation, it can be difficult or time consuming to raise the required funds. Sometimes, moreover, materials or expertise are not available locally. Or, the resources available locally may not be sufficient (e.g., micro lenders whose loans fall short of the money needed for the recommendations, or the materials available locally are poorly made or lack quality). In other cases, volunteer recommendations are simply not appropriate or are not adapted to the local reality. Yet, other recommendations are long-term strategies with difficult-to-achieve interim requirements (e.g., food safety certification as a condition for accessing a specific market or buyer). In rare cases, host organizations are just not convinced or are skeptical of the outcome, which could be a reflection on the quality of the match of volunteer or assignment. RECOMMENDATIONS The volunteer and, more importantly, local IP staff must check back with the host organization after a volunteer departs, to discuss next steps and debrief the quality of the intervention (some IPs do this, but not all). A timeline and plan may be necessary to help the host stay on track, and to not lose motivation or momentum, which can happen when there is no communication from IP or volunteer, and the host is unable to implement their recommendations on its own. Although IPs generally say they need more resources, and many host respondents identified the lack of finances as a limiting factor preventing them from implementing recommendations, the paradox is that this is a volunteer/pro bono program, and the gift of knowledge transfer “without money getting in the way,” as stated previously, is the core value of the F2F Program and part of what makes the ToC effective. Knowledge is F2F’s currency – no money transacted, no money expected. The evaluation team, therefore, recommends that F2F not provide additional 30 financial support to host organizations, but instead explore with IPs creative ways to address the challenges of adopting recommendations. When financial resources are the critical piece to a host organization’s growth, then the IP and/or volunteer can play a role as bridge-builder to external sources of funding, and to identifying appropriate lenders for the size and needs of the host organization (e.g., microfinance loans may not be sufficient for a host organization that is scaling up). The IP and/or volunteer can also become an advocate for the host organization to help it negotiate with a lender. This topic could be a session in an upcoming IP workshop, with participation by field staff leadership so as to ensure that field-level perspectives are incorporated in the deliberations. The evaluation team also recommends that if IPs strategically design a sequence of SOW assignments, there be at least one volunteer who can provide technical assistance and training on networking and new partnership development. It is also recommended that the volunteer seek to identify other sources of development support near the host organization, while also possibly taking the lead to initiate introductions, depending on the volunteer’s status or prominence. As for increasing the rate of adoption of volunteer recommendations, there are several steps that could be taken. These include: 1) IP staff should review volunteer recommendations and discuss with the volunteer during the debrief that usually takes place at the end of each assignment (in the capital city, before the volunteer returns to the U.S.). This allows the IP a chance to understand the context of the recommendation and to flag any recommendations that may not be relevant or could be problematic to adopt; 2) Volunteer should sit with the host to discuss the recommendations, prioritize them, determine the sequence, and decide on a plan and a timeline for implementing them; and 3) IP and/or volunteer should follow up with the host periodically (every three months, for example) to discuss the host’s progress on recommended actions and any questions or issues they have experienced or anticipate. This can be done cost effectively by Skype or What’s App in a three-way call with the host organization. 4.1.6: To what extent are persons trained/directly assisted drivers of change in relevant host organizations? FINDINGS As discussed previously, the KII data show that F2F interventions are effective and lead to desired change when the IP strategically and systematically selects country projects, hosts, volunteers, and SOWs. Further, when volunteers are open and engaging, practical and hands-on in their training and coaching approach, hosts are reported to being more satisfied with the assignment. Among those who have been directly trained by volunteers, approximately 84 percent are very satisfied or satisfied with the formal training received from the volunteer and 89 percent are very satisfied or satisfied with the on-the-job training received from the volunteer. Several KII host organization participants reported that they were now a model to others in the host organization and/or community. Some have become production and education farms; for example, a greenhouse operation that has now partnered with the local agriculture college is now charging money for tours because “we are so busy.” Others take what they have learned, 31 such as low technology food processing, and go out to other communities to teach them. The team found this in several cases with women-owned/managed host organizations. Some hosts and IPs reported that there are many beneficiaries they will never be able to quantify, because they do not know when a host organization is sharing his/her knowledge acquired from F2F with someone else. Based on KII findings, more often than not, the host organizations that drive change were supported by the IP and volunteer after the assignment and are in active and regular contact with the IP and/or volunteer. Respondents expressed that this allows them to discuss new ideas and areas of challenges with the IP and/or volunteer who serves as a sounding board and coach. For example, in the KIIs, several respondents described their ongoing communications with volunteers, via email, What’s App, Skype or by phone. In some cases, they describe talking through issues. For example, in one case, the volunteer and host shared photos of new packaging and labeling ideas for feedback. Interviews also suggest that there is much informal communication and knowledge sharing among host members and other host organizations. The team observed this first hand when discovering that in one instance the leadership of one host organization referred a farm association to F2F Program because they were struggling. In another instance, a F2F Program brochure was being passed around from farm to farm. CONCLUSIONS Success depends on good host selection. IPs are doing a good job selecting hosts, because they, more often than not, have recommendations and a clear vision of what agricultural economic development looks like in their country. In the words of the host organizations respondents, just to name a few, “we couldn’t walk; F2F showed us the way,” and “the volunteer opened my eyes to what is possible.” It is as if those who have been directly assisted/trained have been validated and empowered to think bigger and outside the box. Host organizations that have directly received effective interventions, have achieved or will achieve desired changes, and that have a communicative and supportive IP in country, are more likely to be drivers of change. When a host organization is motivated, the IP and Program can lose traction and momentum when there is little to no follow up. When an enthusiastic host hears from IP and volunteers, it can provide validation and inspiration and, in some instances, helps hosts get through challenges so they stay motivated. These hosts are more likely to become drivers of change. Word-of-mouth promotion and “show and tell” by a host organization is a very strong message that can inspire others to change and drive change. Hosts who have been trained and are satisfied with their training tend to share the knowledge with others, whether formally (as in Agriculture Education Training (AET) or extension service) or informally with a walk-through on their farm, as they explain their new practices and what changes they made. RECOMMENDATIONS IPs and/or volunteers should provide follow up and follow-on (coaching) support, or at least attempt to communicate periodically with the host. This can provide the extra motivation and moral support a host may require during challenging moments. It can also build the capacity and 32 sustainability of these drivers of change, and develop drivers of change who may not have otherwise been drivers. At the very least, it may avoid host disappointment when they are inspired after a volunteer assignment and receive no follow up. IPs should be proactive about staying in touch with host organizations, especially those which are enthusiastic and connectors to other hosts, beneficiaries, or partners. An enthusiastic and satisfied host will be more likely to become an active champion of the F2F Program and will gladly and freely promote the Program to his/her connections: other farmers, local government, international NGOs and others. IPs should be noting how many “referrals” are generated by hosts. Strengthening and leveraging these networks can enable these drivers of change to generate multiplier effects that benefit many stakeholders. Follow-on support is needed, regular communication is recommended, but at the least there should be a debrief of the assignment, and periodic checking in, even if it is a quick phone call or What’s App message from IP. Volunteers or IPs can undertake the follow-up but, in either case, consistent follow-up and communication are key to achieving sustainable impacts. 4.2 EQ-2: INDICATORS: HOW WELL DO F2F STANDARD INDICATORS CAPTURE THE EFFECTS OF F2F ACTIVITIES? AND DO THEY PROVIDE USEFUL INFORMATION FOR F2F PROGRAM MANAGEMENT? Table 9, below, describes the standard performance and impact indicators used for the current phase of F2F Program (FY 2013-FY 2018). Table 9: Standard Performance and Impact Indicators in Use in the F2F Program 2013-18 Inputs Outputs Outcomes Impacts Number of vol. assignments: • Male • Female • Total Volunteer demographics: • Sex/gender • State of residence • Job/occupation category • Race/ethnicity • Prior F2F service Number of volunteer days completed Value (U.S.$) of volunteer professional time Value of host contributions (U.S.$) Type of volunteer assistance: • Technology transfer • Organizational development • Business/Enterprise development Hosts assisted: • Cooperatives and associations • Individual private farmers • Other private enterprises • Non-profit, public interest NGOs • Public and private education institutions • Rural financial institutions • Public sector technical agencies Host demographics • Country • Country Project • Gender Persons receiving formal training: Volunteer recommendations adopted Value of resources mobilized by host Value of resources leveraged by grantee/ volunteers in the U.S. Economic: • Annual net income (U.S.$) • Value of gross annual sales (U.S.$) • Area under improved production technology (ha) Organizational capacity: • Number of new or improved products and/or services • Organizational development index Financial services: • Value of rural/ agricultural loans (U.S.$) 33 • • • Financial services Environmental conservation Administrative Type of value chain activity: • • • • Information and input (pre-production) support services On farm production Processing (product transformation, storage, transportation) Marketing (branding, advertising, promotion, distribution, sales) • Male • Female • Total Persons assisted: • Male • Female • Total Number of volunteer recommendations made Number of public outreach activities • Number of rural/ agricultural loans Environmental: • Area under improved NRM (ha) Number of beneficiaries • Members/owner s • Employees • Clients & suppliers • Family members 4.2.1: How well do the indicators for F2F serve program management and the monitoring and evaluation process? FINDINGS IPs use indicators in their management plans to allow them to concentrate on areas of maximum impact. Of the IPs that responded to the survey, over 75 percent use the information provided by F2F indicators to make program management decisions; while 85 percent of them indicated that the indicators provide information that is important for the M&E of their program’s work. Through discussions with IPs, the consensus is that indicators are most useful for tracking progress of their 5-year country work plan and, when reviewed against budget expenditures, can help IPs ensure that funds are directed to maximize effectiveness of the program to reach desired impacts. IPs also identified several indicators as particularly valuable for making management decisions related to their program. The “number of volunteer assignments” was selected by over one￾third of IP respondents, while the “number of persons trained” and the “number of volunteer recommendations adopted” were selected by 30 percent and 24 percent of respondents, respectively (Table 10). Table 10: Indicator Difficulty, Simplicity, Value, and Need to Eliminate Indicator Difficult (N=98) Simplified (N=96) Valuable (N=98) Eliminated (N=96) 1. Number of volunteer assignments 3.1% 7.3% 36.7% 1.0% 2. Number of volunteer scopes of work 3.1% 11.5% 10.2% 11.5% 3. Number of days of volunteer service 5.1% 6.3% 14.3% 1.0% 4. Type of volunteer assistance 3.1% 4.2% 12.2% 6.3% 5. Type of value chain activity 4.1% 4.2% 11.2% 3.1% 6. Value of host contributions 8.2% 6.3% 4.1% 3.1% 34 Indicator Difficult (N=98) Simplified (N=96) Valuable (N=98) Eliminated (N=96) 7. Number of persons trained 2.0% 3.1% 30.6% 3.1% 8. Number of persons direct assisted 7.1% 6.3% 13.3% 7.3% 9. Number of volunteer recommendations 4.1% 6.3% 8.2% 7.3% 10. Number of host organizations assisted 0.0% 3.1% 13.3% 3.1% 11. Area of potential production influence 18.4% 17.7% 4.1% 11.5% 12. Area of potential environmental/ natural resource influence 31.6% 24.0% 3.1% 13.5% 13. Number of potential beneficiaries 9.2% 9.4% 4.1% 9.4% 14. Value of resources mobilized by host 12.2% 2.1% 4.1% 5.2% 15. Number of volunteer recommendations adopted 10.2% 3.1% 24.5% 1.0% 16. Number of beneficiaries 4.1% 7.3% 12.2% 2.1% 17. Value of annual gross sales 21.4% 10.4% 11.2% 7.3% 18. Annual net income 30.6% 6.3% 10.2% 6.3% 19. Organizational development index 16.3% 28.1% 12.2% 9.4% 20. Value of rural/agricultural lending 7.1% 3.1% 1.0% 4.2% 21. Number of rural/agricultural loans 9.2% 1.0% 2.0% 8.3% 22. Area under improved production technology 18.4% 15.6% 7.1% 5.2% 23. Area under improved environmental/natural resource management 23.5% 22.9% 3.1% 10.4% 24. Number of new or improved products and/or services 2.0% 2.1% 10.2% 1.0% 25. Number of press releases 3.1% 2.1% 0.0% 11.5% 26. Number of media events by implementers, volunteers, hosts and partners 3.1% 5.2% 0.0% 6.3% 27. Number of group presentations by implementers, volunteers, hosts and partners 1.0% 4.2% 3.1% 3.1% 28. Value of resources leveraged by volunteers and programs in the US 6.1% 2.1% *See note below 5.2% 98. Don't know 5.1% 10.4% 4.1% 12.5% 999. None 1.0% 2.1% 3.1% 17.7% *The indicator “Value of resources leveraged by volunteers and programs in the US” was not included for the question “Which indicators provide valuable information for making management decisions related to your program?” due to a programming mistake. As such, no data was collected on this indicator for this question. Through the KIIs, IPs did, however, voice the opinion that not all indicators are useful and some are collected only to satisfy reporting requirements for USAID-Washington. Generally, IPs both in the U.S. and in-country, felt the number of indicators is too high and that many are cumbersome to measure. They also thought that the full impact of F2F activities is not captured 35 by the existing indicators. Further details on these latter points are included in the relevant sub￾questions below. The evaluation team discovered during field visits that some IPs had dedicated M&E staff for collecting these indicators, while others with smaller staff size and fewer funded projects, shared the M&E task among several staff, with the local F2F Country Director supervision. M&E staff respondents were not only well versed in M&E and F2F indicators, but also placed high value on indicators, with one describing the prevailing view by stating, “outcome indicators are the most important; they help us see what activities took place and where we are as compared to our targets.” Those with M&E staff, typically had systems and records in place to ensure quality data collection, and even provided training and outreach to host organizations and customized data collection forms for their use, to make the process easier and more relevant to hosts. On the other hand, during the field visits, the evaluation team found that many host organizations are unfamiliar with the F2F indicators. Hosts with experience on other funded projects understand indicators in general, though they may not have knowledge about the F2F indicators in particular. Those with some knowledge of the F2F indicators have gained this understanding due to the efforts of IPs to speak with them specifically about these points. IPs carry out a baseline study prior to a volunteer’s assignment in order to track the impact of the work. Many F2F implementers were concerned about the quality of these baseline studies, however, noting an absence of written or other records (on the part of the host) to consult, as the host may lack the literacy or skills to keep effective records. While IPs did note that hosts can provide answers when asked, without records to reference, these data are subject to recall bias on the part of the host. To help address this, IPs reported that they are providing volunteers to teach record keeping, which they maintain is improving data quality and reducing the time required of IPs to collect data. Six months and one year after an assignment, F2F gathers data to compare with the baseline. In certain cases, this is a straightforward operation such as measuring the increase in milk production due to improved feeding methods introduced by volunteers. In other instances, such as improved fruit tree grafting, the improvements will not be evident for years, and therefore cannot be measured in a relatively short span of time. As for USAID Mission indicators, almost all in-country USAID Missions report on F2F activities in their annual review. Some actually use one or two of the F2F indicators though many USAID in-country Missions question why there is a separate set of F2F indicators from those of Feed the Future. However, in the view of the evaluation team, having separate indicators makes sense since many F2F countries are not Feed the Future countries. CONCLUSIONS The indicators are used mostly by USAID and the F2F IPs. Only a few host organizations track indicators. Overall, the standard indicators do serve program management and the M&E process, giving IPs and all stakeholders a common language and common ground from which to start, as well as a 36 common focus. There are some exceptions, however, and they are detailed in the following sub-questions related to indicators which are difficult to collect and those which should be eliminated. Different stakeholders place different value on different indicators, so the capacity and incentive to report one indicator over another may vary. This is because, while diversity is viewed favorably by stakeholders, the implementation of the ToC at the F2F Program (HQ) level draws on many disciplines. The IPs and the host organizations bring their fundamental sectors of interest, expertise, and ability to introduce innovation. Agriculturists focus the analysis in agricultural development with higher yields; the economists push on economic improvement with lower production cost; others put their energy in gender, youth, and employment. Most IPs work in more than one sector while few others prefer to emphasize one of the relevant sectors at the expense of others. Empowerment and resilience evaluation looks into local capacity to recognize and coherently measure growth towards their local goals. Indicators are essential in the ToC and need to be discussed in order to be well utilized, from IP HQ staff with IP Country Staff, and from IP Country Staff with Host Organizations. There are challenges in collecting accurate data from hosts, due to capacity issues and lack of knowledge around indicators in general, and F2F indicators, in particular. As a result, IPs rely on recall data from hosts or find other creative solutions for collecting the required data. Each of these approaches, however, may introduce bias into the data, which can affect its usefulness in making programmatic decisions. RECOMMENDATIONS Based on the evaluation team’s review of indicators and fieldwork interviews with IPs and host organizations, the indicators, while extremely useful, in and of themselves, do not tell the complete story, and USAID should be reviewing data in context. The evaluation team recommends that every set of Indicator Tables has a brief narrative, similar to budget notes, and a short one to two-page summary that provides the reader with an overview of the indicators and why they are being collected, numbers, and any outliers or specific areas of concern. The evaluation team also suggests that IPs continue providing, whenever possible, volunteers that teach record keeping, in order to improve the quality of data available. As part of this activity, IPs will also need to foster buy-in among hosts by demonstrating the value of the indicators and how these may be used to support the Program and the host themselves. As not all indicators are considered valuable by IPs, as detailed below, these indicators should be revisited by USAID to make them more valuable to program management and M&E, or to remove them from the list of indicators completely. Although some Mission staff question the need to have duplicate F2F and Feed the Future indicators, some duplication is necessary since not all F2F countries are part of the Feed the Future Program. The elimination of F2F measures may result in a failure to capture progress along program goals if all duplicate measures are jettisoned. 37 4.2.2: Do some indicators present particular difficulties to hosts and IPs for collection and management interpretation? FINDINGS “IPs need to understand our portfolio. Otherwise it can be a waste of money, focusing on different sectors, with one off assignments.” –USAID Mission Officer when discussing synergy between USAID Centrally-Funded mechanisms working with Country Missions. Almost all host organizations are unfamiliar with the F2F indicators. The only ones who fully understand them are other USAID-funded projects. Most host organizations that are individual farmers keep very basic records, which makes it hard to feed data into indicators. IPs need to be creative with some indicators such as financial data because host organizations are, in many instances, not willing to divulge such information. Many times, they need to use proxies to measure gross sales and annual income. However, these indicators require analysis of the underlying cost. The evaluation team found that among all indicators, the “area of potential environmental/natural resource influence” was most commonly cited as being difficult to collect, with nearly one-third of IP respondents selecting this indicator. Second to this, “annual net income” was selected by 32 percent of respondents. The “area under improved environmental/natural resource management” is difficult to collect according to 24 percent of respondents, while the “value of annual gross sales” is difficult to collect according to 21 percent of respondents. All other indicators were selected by less than one-fifth of respondents (Figure 7). The evaluation team’s field visits, likewise, found that collecting data related to hosts’ financial information can be difficult, as some host organizations are reluctant to divulge these details. Hosts cited the proprietary nature of these data as well as fears about tax collection if these data are shared. As a result, many times, IPs are forced to use surrogate measures to capture gross sales and annual income. These measures may miss key details, such as information about underlying costs and debt that may have triggered an increase in sales, thus excluding details that are important to the data’s accuracy and interpretation. IPs may also take different approaches when a surrogate is required, undermining the comparability of the data across programs. IPs also face, on the whole, data quality issues, despite many IPs having a dedicated M&E person. Much of the data, such as income or number of (indirect) beneficiaries, is derived by proxy at best and by guesswork at worst. M&E staff needs to be creative in many instances to gather data that is not readily available, for example sitting with host respondents as a group, to work through the math at the individual farmer level to then aggregate and report as a group/host organization (e.g. Cooperative). Further, when a volunteer’s activity involves a non-economic or productivity-based activity, for instance improved hygiene, home canning, or family health, there is no measurable basis for a before-and-after improvement. 38 Figure 7: Indicators Reported as Difficult to Collect Source: Evaluation web survey In extreme cases, difficulties collecting data can affect program selection, a problem underscored by IP commentary. During the evaluation team’s field visits, several IPs lamented that they had to drop hosts who were highly motivated simply because the data would have been ultimately too difficult to collect and report. As summarized by one IP country staff, “Some farmers are just not organized. There’s no structure. It’s hard to collect data because everyone is doing something different.” In these cases, the need to report on indicators drives host selection, potentially to the detriment of the F2F project. CONCLUSIONS IPs use indicators to report on management of activities, survey findings show. However, even with that, there is no standardization within the IPs of how to measure some events. For example, IPs considered a volunteer assignment in different ways. IPs indicated it was very difficult to track the value of sales because hosts are often reluctant to provide that information, fearing questions about taxation. In addition, there is often the absence of written or other records to consult to establish a baseline and, therefore an accurate measure of economic gain. In the view of F2F IPs, indicators related to the “area of potential environmental/natural resource influence” and “annual net income” are the most difficult to collect. These F2F indicators are also commonly cited among those that need simplification, have low value, or need to be eliminated. This underscores that while these indicators are difficult to collect, IPs also find that there is little value in collecting data for these indicators with regards to their own project management decisions. One of the most serious methodological problems with F2F indicator data collection relates to attribution. Given that many F2F assignments are working with larger organizations that have a myriad of other associated activities, it is nearly impossible to precisely attribute F2F activities 39 to a change in given indicators. F2F has a hard time matching training with identifiable impact, because the change does not happen immediately. Additionally, it is hard to attribute change to one particular activity, as there are often a number of other activities taking place in tandem. RECOMMENDATIONS Although it is difficult to track sales, it is important to provide a context for revenues, and develop measurement indicators that demonstrate profitability. Sales could be changed or another indicator could be added to put the sales in context. Debt and costs are important when factoring success. Alternatively, an additional indicator—such as debts and costs—could be included to put the sales in context. In developing such a metric/indicator, the challenge of transparent financial disclosure remains. Participants can be advised that their information is private and for program reporting purposes only. This would assure hosts that their personal information will remain private and possibly create a friendlier reporting environment. USAID should determine which measures can be standardized and which must remain country￾specific. Once that process has occurred, measures should be conceptualized (defined) and operationalized (how it will be measured in the field). IPs should be trained regularly on the standardized measures and receive training on data collection for complex measures if necessary. Recommendations related to the indicators for environmental/natural resources are included in the sub-questions below. 4.2.3: Are there indicators that might be simplified or eliminated and are any others needed? FINDINGS According to the survey, 18 percent of the IPs said there is no need to eliminate F2F indicators. However, as mentioned earlier, KIIs with IPs found that they tend to believe that not all indicators are useful, there are too many of them, and that the full impact of F2F activities is not captured by the existing indicators. The Organizational Development Index (ODI) was identified by 28 percent of IP respondents— more than any other indicator—as needing simplification through the evaluation team’s web survey. However, there was a significant difference of opinions from respondents concerning this indicator, as found during the evaluation’s field visits. One F2F IP stated, “The Organizational Development Index is pointless.” However, another interviewee lauded the ODI as a good tool, especially for AET. Most IPs, however, felt this indicator was too subjective and cumbersome. While the greatest percent of IPs felt no indicators needed to be eliminated, as noted above, 14 percent—the second highest percentage—felt that the indicator for the “area of potential environmental/natural resource influence” should be eliminated. This latter result is consistent with the low value IPs placed on this indicator, as only 3.1 percent of IPs felt this indicator 40 provided valuable information for making program decisions. Similarly, 24 percent of respondents marked this indicator as needing to be simplified. During the field visits, interview findings from hosts were mixed regarding Natural Resource Management (NRM). The prevailing view among some hosts was that NRM is not an important aspect of their work. Their first priority is productivity, the second is price/profit, and their last priority is the environment. Given the many challenges at the community level due to the levels of poverty, environmental conservation was not a high priority for these hosts, with individual farmers stating that they would not pay extra for better NRM. Further, few host organizations and IPs believed that improvements in NRM could be quantified. It should be noted however that interview findings were mixed on these points, with some hosts strongly committed to NRM. The evaluation team also investigated if new indicators should be added, with 23 percent of IPs responding that this was necessary. The most popular of these suggestions was an indicator related to impact with respondents suggesting an indicator on the impact on nutrition, organization health—outside of the ODI, and simply impact in general. While this was the most common suggestion from respondents, it should be noted that only nine individuals suggested such an indicator. CONCLUSIONS IPs generally felt that the ODI indicator needed to be simplified, with some IPs suggesting that there is little value in this indicator. As noted in previous sub-questions, IPs have found the indicators related to environmental/NRM are difficult to collect, and they are commonly cited as those which need to be simplified or eliminated. Hosts’ opinions of the importance of these indicators, however, are mixed. While IPs suggested new indicators related to the impact of projects, such measurements would be biased without a rigorous evaluation. Such an evaluation would be beyond the scope and resources of F2F programs. RECOMMENDATIONS The team suggests that the ODI be considered for elimination, given the absence of clear, objective rubrics for measurements that can be applied across the board to all IPs, and thus contribute to reliable data collection. Adding a new indicator to measure impact—as suggested by several respondents—will be difficult to implement in a rigorous way. As such, the resulting data will be biased, and so there will be little value in including new indicators specifically for this purpose. The evaluation team suggests that the definition for the “number of volunteer assignments” should be changed from “A ‘Volunteer Assignment’ is equivalent to a volunteer ‘trip’ from his/her point of origin to country or countries in which s/he provides services to a host(s)….” to “…is equivalent to each country the volunteer visits in which s/he provides services…” The 41 logic behind this change is that each country has its own standalone program and will be counting the volunteer’s assignment regardless of the other countries visited. 4.2.4: Is the F2F Program’s performance and impact on outreach/cultural exchange, capacity development, and environmental/natural resource management being captured effectively? FINDINGS Among the indicators for outreach/cultural exchange, capacity development, and environmental/NRM, IPs felt that indicators for cultural exchange and environmental/NRM were particularly weak. To the first of these, 47 percent of respondents felt that the indicators for cultural exchange did not effectively capture the program’s performance. Similarly, 38 percent of respondents felt the indicators for environmental/NRM were also not capturing the program’s performance. IPs were more positive with regard to the indicators for outreach and capacity development; however, they stated that these could still be improved. One-third of respondents felt the Program’s indicators for outreach did not capture the Program’s performance and impact. With regards to capacity development, 18 percent of respondents responded negatively. For each of these indicators, the evaluation team investigated what changes should be made to improve the indicators’ performance. Among those respondents who felt the current indicators were underperforming, three-quarters suggested additional indicators related to cultural exchange and over 60 percent suggested new indicators for environmental/NRM. Responses were more evenly split with regards to outreach and capacity development, with 53 percent suggesting new indicators for the former, and 56 percent suggesting fewer indicators for the latter. It should be noted that for each of the results described in this paragraph, the sample size was relatively small—given that questions were only asked to respondents who expressed dissatisfaction with the current indicators. From these limited samples, the most common responses for new indicators to include were: • • • • On outreach: Indicators (e.g. number of posts) related to social media On cultural exchange: Number of cultural activities volunteer has participated in On capacity development: Indicators related to changes in host capacity On environment/natural resources: Indicators related to the number of new environmental initiatives or individuals adopting climate-smart practices CONCLUSIONS IPs generally feel that improvements can be made to the indicators related to outreach/cultural exchange, capacity development, and environmental/NRM, with some suggesting adding additional indicators. A large number of IPs suggested new indicators for environmental/NRM. RECOMMENDATIONS 42 The evaluation team suggests creating a new indicator called “Number of Outreach Activities.” This indicator would eliminate several existing indicators, as they do not significantly measure anything different. The indicators that would be eliminated by this new indicator are: Number of Press Releases; Number of Media Events by Implementers, Volunteers, Hosts and Partners; and Number of Group Presentations by Implementers, Volunteers, Hosts and Partners. The team suggests adding to reporting requirements, “How did you first hear about F2F?” as awareness indicators. Knowing how someone heard about the F2F Program is a useful indicator for what is working and not in the area of marketing and promotional outreach. Having this information from both volunteers and host organizations can provide IPs with a way to see what publicity methods are working better than others, and then focus the funds and time on what works best to identify host organizations. The evaluation team does not suggest including a new indicator related to cultural exchange, contrary to the results of the web survey detailed above. While cultural exchange is an important outcome of the F2F Program, measuring this—even through cultural events—would face definitional issues that would be difficult to standardize across IPs and various country contexts. In addition, as noted in previous sections, there is a sense among many IPs that there are too many indicators. Thus, indicators which assist in program management and M&E should be prioritized as additions. Some suggestions included: job creation (one host organization created 300 jobs); multiplier effect (identifying pre-and-post financial and nonfinancial assets); secondary impacts (Intervention + host organization action = new and different impact); how people (host organization, volunteer) first heard about F2F; and health outcomes from F2F interventions (more fruits and vegetables consumed). The team recommends that this should be studied to assess the feasibility and burden of collecting this information. NRM and environment activities can be monetized and quantified. The team recommends exploration of NRM and environmental indicators, given the possibilities of measurement. The team also suggests for consideration, indirect NRM and environmental impacts, such as increased volume of product harvested due to new practice, or increased consumption of healthy foods like produce (food security). 4.3 EQ-3: LEARNING & INNOVATION: HOW WELL DOES THE F2F PROGRAM LEARN AND INTEGRATE INNOVATION TO ENCOURAGE QUALITY IMPROVEMENTS AND GOOD PRACTICE IN AGRICULTURAL VOLUNTEER PROGRAMS? 4.3.1 How well do F2F IPs engage in a community of practice to improve the Program? FINDINGS With respect to learning and innovation broadly, data from the web survey indicate that 60 percent of IPs are very satisfied or satisfied with the level of collaboration and information sharing among F2F implementers. However, 15 percent of IPs reported dissatisfaction with the current level of collaboration and information sharing. In addition, often, KII respondents 43 described that the Program does well on a surface level, but more could be done to improve in capturing and sharing learning, and uncovering innovation to be shared across IPs. The F2F core IPs, as LWA Cooperative Agreement recipients, represent a community of practice (CoP) given their regular communication with one another and with USAID. As an overall CoP, IPs strive to make improvements through regular, virtual, and in-person meetings. These meetings provide the opportunity for: 1) peer-to-peer exchange with fellow IPs working in other regions of the world; 2) cross-training through discussion and analysis of innovations and sectors; and 3) learning about interventions by fellow IPs that may be new to another IP. All of these give IPs the venue and a supportive environment for focused discussion and analysis of issues and challenges. All participants are free to contribute feedback or ideas to resolve issues unique to the IP or related to the F2F Program overall. Among the core group of IP HQ staff management, the IPs engage well, but when it comes to IP field staff, KIIs reveal that many, beyond perhaps the Country Director, are disconnected from the learning that occurs at the IP HQ level. Some IP staff, such as field staff or U.S.-based staff in lower ranking positions who may have insights and unique perspectives, may not be at the table as the HQ-level Program Directors to coordinate and communicate. The team did not get the sense that the IPs are trying to be exclusive; rather, it appears more a matter of lack of time and resources, and needing someone to continue backstopping while CoP communications and meetings take place. Formal and informal working groups that gather under the F2F umbrella function as CoPs. Among the formal working groups are the Outreach and M&E Committees. The Outreach Committee is an official part of the Program and is apolitical. This group focuses on getting the word out about F2F, providing general public outreach, communications, and educational materials. Members of the Committee come from various IPs. For the Outreach Committee, while members meet regularly, attendance is inconsistent, with different staff attending each meeting, and there is a lack of accountability or mandate for the Committee members. There are also no dedicated resources for follow through on the group’s recommendations generated in meetings, presenting F2F at conferences, strategically placed advertisement and/or articles in trade publications. All of these, limit the Committee’s effectiveness as a communications and dissemination vehicle for the Program, though IPs do provide their own individual outreach and promotion of the F2F Program. The M&E Committee gathers virtually and in person to discuss matters pertaining to indicators, targets, and results reporting. In recent months, the Committee has been studying the effectiveness of capacity building assessment tools, such as an ODI, discussing its advantages and disadvantages, as well as what could be improved or replaced. While USAID coordinates the formal CoP, SPSP facilitates more informal working groups and learning opportunities within the F2F community. By organizing seminars and panel discussions on topics of interest to the community, SPSP provides opportunities for informal collaboration and learning. In addition, SPSP manages the resource database on Farmer-to-Farmer.org, which provides space for implementers to share tools and resources, such as tips on using Excel for reviewing M&E data or lessons learned on specific topics. SPSP is both an IP (internal) and a third party grant-maker and knowledge broker (external) facilitator. It is more challenging to 44 assess the role of SPSP given that SPSP was designed to be a third party objective outlier to support IPs, but is itself an IP as it manages grant-making and capacity assistance to small and medium-sized grants CONCLUSIONS Whether an IP engages effectively or not with the CoP or Committees varies by IP and IP staff type. KIIs revealed lack of time and resources as most common barrier to more effective and regular engagement. Respondents in KI interviews suggested that CoPs are most useful if they are relevant. The direction and utility of CoP discussions depends on who is assigned to sit on what CoP Committee and who decides to attend meetings and conference calls. RECOMMENDATIONS F2F should prioritize, discuss and decide which learning groups would be of most value to all IPs. Our IP interviews indicate that the idea of a recruiters’ CoP has been floated in group discussions, but needs some support to get organized. Also, under consideration but not yet confirmed, is a CoP for country. The evaluation team’s recommendation would be to first examine and agree on the goals and objectives of each learning group, whether CoP, committee, or something else, then ensure that the appropriate people with the right skill set are in that group to provide the best value, or to ensure that there are numerous viewpoints represented, not just those of IP HQ management. The evaluation team also recommends that F2F be more inclusive and engaging of whom it selects to be part of CoP, and to not undervalue the knowledge, experience, and insights that non-management staff and field staff may have. 4.3.2 In what ways could the Program improve learning and innovation to become more efficient and effective? FINDINGS F2F shares innovations and lessons learned through a variety of media. In some cases, the methods are internal to the individual IP and their own networks, while in other cases, the methods are Program-wide and largely managed by SPSP and/or USAID F2F. The web survey reveals that a majority of IPs (63 percent) are satisfied or very satisfied with their ability to find and use innovative practices for their work on the F2F Program. However, 11 percent of IPs are dissatisfied or very dissatisfied. Nearly three-quarters (77 percent) of IPs felt that budget constraints were a factor in their ability to implement relevant, innovative practices, which they learn about. Further, IP respondents expressed lack of time and resources to dedicate to the learning and sharing, even though they desire it. IPs realize that hearing from their stakeholders is important, but not always possible due to time and budget limitations. Among IPs, 63 percent feel that “listening to stakeholders’ perceptions and ideas” is important to the Program’s learning to become more efficient and effective. Almost 61 percent feel that it is important to “facilitate share of practices between organizations.” Some IPs with the resources, organize regional learning events for IP staff working in a given region. In-country field staff described these regional meetings as helpful, and 45 a useful way to see what is working in other F2F countries where the IP is working. One IP organized farm tours in the U.S. for its IP field staff. The tours were sponsored with funds outside of F2F, as a follow on to a large implementers’ meeting. IP field staff are eager for more of these types of meetings. Among U.S.-based IP staff, 78 percent consider the F2F Program Implementers’ Meeting to be an effective way to encourage learning among F2F stakeholders. Further, the survey data indicate that 76 percent of IPs feel that F2F should facilitate additional country, regional and global partner meetings to exchange innovations. IPs are also thinking about how to maximize returns on investment of time and training, by working with intermediaries, and training of trainers (ToT) and those who can perpetuate the work of F2F (“multiplier effect”). AET is a formal example of this practice; however, there are informal networks of shared learning, including peer-to-peer farmer word-of-mouth. “AET maybe the way to go to make the broader impact F2F seeks. They are able, as intermediaries, to reach many beneficiaries, and institutionalize what F2F does (sustainability) – fewer hosts, deeper impacts. A relationship is built,” and this network creates a conduit for communications and learning. CONCLUSIONS KIIs with IPs working with intermediaries who connect them to hosts, or those who are training trainers or are explicitly focused on AET, revealed that because these host organizations or partners are “hubs” who connect many current and potential F2F stakeholders, they can be particularly effective as a mechanism for shared learning and unearthing new ideas. AET networks, such as a government-led national vocational or agricultural school, can lead to tremendous impacts as learning is spread throughout the network, and trainees are training other trainers. Several host organizations engaged in AET invite other trainers to the volunteer TOT sessions. All trainees go on to teach more people, and they are also educating the next generation of youth and enabling youth to be prepared to join the workforce with a practical education, which addresses the wide gap between jobs available and qualified candidates due to poor training or educational system, or outdated curriculum. While field staff indicated an eagerness for more ways to connect with each other, they also expressed that meetings dominated by F2F IP HQ staff have limited usefulness. Virtual meetings such as webinar also can be challenging largely because of potential technology issues such as internet disruption, and logistics like working across time zones. Through the KIIs, IP field staff members expressed interest in peer-to-peer learning among fellow IP staff within their own organization or regionally among various IPs, or even among peers (other field-based staff) in the U.S., combined with field visits in U.S. to farms and food processors. The evaluation team concludes that there are three major obstacles that can prevent efficient and effective learning. First, an obstacle to open sharing is the reality that IPs compete against each other every five years when F2F is competitively bid. During this period of competition, IPs are expected to continue to collaborate. A second obstacle to learning concerns challenges in coordinating learning across members of the F2F community with different levels of experience and engagement; specifically, USAID HQ, SPSP grantees, and core IPs. A final obstacle to learning is that effectiveness of the formal CoPs is limited by lack of dedicated resources, both financial and personnel, to follow-up on and implement recommendations. 46 RECOMMENDATIONS Encourage peer-to-peer learning. It may have cost implications but it can also bring longer term benefits in terms of the knowledge and innovation, reducing redundancies and avoiding pitfalls that another IP has already been through. Another way to reduce costs would be to leverage other projects within the IP portfolio, or to tack on additional objectives to a regional meeting, such as HQ IP working with SPSP and field staff to develop learning products through the course of the time they are together to be completed by the time they part. The team suggests a rapid cost/benefit analysis be conducted; it does not have to be formal, and could be a topic of discussion where the group attempts to quantify tangible and intangible costs and benefits to determine what is best. F2F should review and address three major obstacles that prevent effective and efficient learning and innovation. There is no “silver bullet” solution to eliminate the tension between IPs being competitors and collaborators. The team recommends that USAID prioritize and discuss this internally and with IPs individually to determine how to strengthen the CoP without threatening any IP’s success or competitive advantage as compared to other IPs in the Program. To promote more effective and consistent engagement in the CoP, we recommend that USAID and SPSP work together and then with the group to clarify “rules of engagement” of the CoP, Committee, and/or meeting (who, what, when, why of the CoP), so expectations are clear. Finally, where possible, USAID and SPSP should follow up after CoP meetings to debrief and to discuss what resources are needed to implement recommendations that were discussed in CoP meetings to confirm what is reasonable and what may require additional resources. 4.3.3 What learning and innovation activities or community of practice support are particularly effective or needed and what may be of lesser effectiveness or use? FINDINGS Learning and innovation activities that are particularly effective, as informed by IP interviews, include the below. Several respondents qualified their response, so qualifiers are included in parentheses: • • • • F2F Annual IP Meeting (but would be great if country-level IPs could attend). Seminars and webinars (but we need more input from the field). Study tour (but this is costly). Website: (it is both an internal IP tool and public-facing outreach tool, but does not include call to action or any volunteer openings; volunteer candidates can fill in a request for information, which is then passed on by SPSP to all IPs, and may never get a response from one IP). Activities or CoP support that are less effective, according to the consensus among IP respondents include: • Activities that are exclusive and not representative of the population of F2F stakeholders or attendees are poor match for the goal of an activity (e.g. communications person attending a highly technical F2F Program strategy meeting). 47 • • • Activities that are poorly timed, especially virtual events (webinars scheduled while field staff are sleeping). Any activity where participants are not empowered to follow through or are not accountable to the goals and objectives of a CoP, event, workshop. Any activity with no “call to action,” such as a volunteer candidate who is interested not being able to apply or browse assignments on the main F2F website. IP in-country field staff expressed interest in regional meetings, with staff from the IP’s other countries, or with other IP staff, preferring this to a big meeting of all IPs coming together in one place. They suggested that regional clusters for learning are more relevant because IP participants will be able to relate to one another, coming from similar context and geo-political experience. For example, meetings and workshops could be organized by common experience, such as Former Soviet Bloc countries, or by geographic similarity, such as the West Africa region. Another approach is to organize by sector or focus area (for example, poultry, high value horticulture, or food processing). Similarly, those IPs based in Washington, and/or M&E and Outreach Committees, can participate in affinity and membership organizations such as Society for International Development12 or Devex International Development13 for learning, recruitment, and outreach. CONCLUSIONS To be effective takes time, persistence, and interest. In the fast-paced world many (mostly Washington-based) work in, it can be challenging to take time out to engage in a CoP, even when one appreciates sharing and learning. Respondents appear to be overwhelmingly in favor of in-person gatherings among peers, but paradoxically the prevailing view among IPs is that they also do not have time for stepping away from the office. Some host organizations have been able to travel to another country to observe the work of their counterparts managed by the same IP. According to the evaluation’s findings, the IP pays for this outside of the F2F budget. Host organizations emphasized the importance and need to connect motivated and like-minded colleagues/peers with one another in country or in a region. The F2F website, while quite robust and well maintained, is also an underutilized resource for learning and could expand its current reach with a few improvements. It contains a searchable database of resources that is little-known to field-based staff. Selection of people that attend learning activities is important in order to trigger a more dynamic discussion that could lead to discovery of new innovations. Based on the team’s review of press releases, congressional testimonies, and other news pieces the team located, the team concludes that there may be unexplored opportunities to leverage the tremendous interest in the F2F Program by people outside of program IPs, and also opportunities to promote the Program in ways that could yield multiple benefits: new recruits, public interest, or technical interest by potential partner. RECOMMENDATIONS 12 Society for International Development: https://sidw.org/ 13 Devex is “the media platform for the global development community.” https://www.devex.com/ 48 Since host organizations consider staying connected (through meetings, study tours, etc.) with other organizations in the country or the region very important, the team recommends more of these exchanges take place. The team recommends that F2F explore how it might better engage and maintain relationships with volunteers who have served, and among those who are actively interested in continuing the discussion once stateside, or in facilitating topical or other discussions as a learning group. Peace Corps has a Returned Peace Corps Volunteer network. F2F could develop a similar type of network. In interviews with IPs and volunteers, the evaluation team learned that for many volunteers they may be the only person in their community who has ever traveled abroad and the community eagerly invites the volunteer to speak at church and community events. Having an online network, chat room, or social media group, could be welcome, and provide a chance for IPs to gently nudge volunteers to share their experiences and potentially talk to the elected officials in their state or district. There are also other models of volunteer programs that might help F2F generate ideas, such as Rotary International and others. It is not unusual for volunteers who have served in these programs to continue to serve the U.S.-based regional or national headquarters to provide what support they can. To be more effective, representative, and to hear new voices, opinions and ideas, F2F should discuss ways to be more intentionally inclusive. F2F could include this in its next IP meeting, or SPSP could lead a virtual discussion that invites under-represented participants, such as support staff and field staff. The F2F website could be even more effective if stakeholders were to objectively review the capabilities and scope of the website. A questionnaire could be developed to identify what is useful and not useful on the website and what recommended upgrades could be added. SPSP could also test out the website or contact new candidates to ask them about their user experience on the site. With additional investment and access, the website could reach a much wider audience. An implementers-only section could also be added to the site, where discussion groups are hosted and facilitated. At the very least, a person who is interested in volunteering with F2F should be able to peruse what volunteer assignments are open without having to go to each individual IP website, and to receiving a follow up call from an IP. 4.3.4 Are there unaddressed needs for support to core F2F Program quality and good practice? What are key issues for future success of the F2F Program? FINDINGS When asked what needs to be prioritized to improve the effectiveness of the F2F Program, 73 percent of IPs responded “extension and advisory services,” while 68 percent stated “agribusiness/value chains.” In addition, the most popular answer among IPs, when asked about the key issues for the future success of F2F, was improvements to M&E systems/indicators (42 percent). Second to this, 34 percent of IPs felt recruitment of volunteers with more specialized experience was a key issue for the Program’s future success. All F2F IPs bid for funds every five years through the release of a Request for Application (RFA). Generally, the competitive bidding process occurs near the end of an active IP’s 5-year Life of 49 Project (LOP). F2F IPs must therefore compete against each other and against new bidders to win new F2F awards. USAID encourages IPs to collaborate and share learning, but the stark truth is that IPs see themselves as competitors as well as collaborators. Therefore, information, networks, and contacts become valuable resources to be guarded to stay competitive in this open market. In field visits, the team found situations where there was one F2F IP, but there were other organizations working in country that had a F2F award in another country. The F2F IP with the current award may perceive the other organization(s) as a threat, even if they are not a current F2F IP in the same country. Our field visit KIIs revealed that the IP field staff sees this most acutely as they are on the ground. In addition, if there is another F2F IP working in the same country, there can be competition or a complete disconnect, where the IPs do not, or will not, work together, or they have identified their own areas to focus on. In one instance, an IP field staff told the team that another NGO called them to ask questions about how they manage F2F and wanted contacts; the staff member that answered the call was reluctant to share information that would “put us individually at a disadvantage,” and impact competitiveness. There are some IPs who have been with the Program since its inception. Newcomers are rare, and when an implementer is new it injects diversity and fresh perspectives. However, it also puts the newcomers and smaller implementers at a disadvantage as they spend the early years of their award getting systems in place, while the old-timers are already placing volunteers and fulfilling targets. While this would seem to make the process fairer, it poses a challenge when an IP in one country changes because the new IP may or may not have as many resources (human, financial, experience), or may have an entirely different management style or capabilities. In field visits, the team learned from host organizations that when an IP changes, the F2F quality changes, too. In one instance, the evaluation team heard that the new IP “is so much better; better communicators, not doing things at the last minute.” In another, the team heard, “F2F used to be more communicative and organized.” This could positively or negatively impact the F2F brand overall. CONCLUSIONS There are key issues that prevent F2F from collaborating, innovating and sharing as well as it could across IPs, with the top issue being the tension of IPs being both competitors and collaborators. Competition can breed innovation and enhance performance, which supports an effective ToC (“it keeps it fresh,” we were told by USAID), but makes a collaborative CoP challenging to cultivate. This appears to be more of an issue for field-based IP staff than HQ based staff, as many U.S.-based HQ staff have known each other for years. If this is not addressed, at best, there could be missed opportunities for synergy among IPs in the field and, at worst, duplication and wasted time and money doing similar work. RECOMMENDATIONS There is no perfect fix to the competitor/collaborator challenge, especially with a program that is already in place, where there are already IPs in the field executing awards. Forcing IPs to work together could be problematic and too late to implement, as the current award comes to an end in FY 2018. Thus, in the near term, one way to potentially address the issue (before the 50 end of this 5-year award phase), would be to organize regional working groups that come together in person, with rotating location/country, and are represented by all IPs in the region. These groups provide a more neutral setting for IPs to share learning and innovation more effectively, and to see each other. In the mid- to long-term, F2F may want to consider revising its policy so that, at a minimum, there is no more than one core IP (Cooperative Agreement with LWA) implementing F2F in a country at the same time. As for the scenario where there is a F2F IP working in the same country as an NGO that has a F2F award in another country, this would require discussion at the F2F leadership level to determine the cost/benefits of making any changes to policy. F2F is like an unsung hero – too busy doing the good work that it has no time to stop and tell anyone about it. However, this can be to the detriment of the Program, because once people hear about F2F they are generally quite impressed and are happy to hear the USG is sponsoring programs like this. It must start from within, at USAID. The Program should start with identifying who and where the allies and advocates of the Program are across industries and sectors, and then examine how to amplify the message working with these allies. This could be a brainstorming exercise among IPs in an Outreach Committee meeting, or a list/directory started by USAID F2F for IP input. When it comes time to do outreach, IPs and SPSP especially, need to be given the recognition they are due, but someone in authority, or who is granted authority, and resources are necessary to make this happen. 4.3.5 How effectively have SPSP projects (including sub-awards) supported F2F learning, innovation, and community of practice; and how has that support been perceived by the F2F IPs? FINDINGS To keep the team’s commitment to anonymity, the evaluation team is only able to report what is non-attributable to the single SPSP prime implementer or any individual SPSP field staff members. F2F SPSP is a relatively new program, currently in its second phase and on to its second implementer. SPSP was designed to be a service provider to F2F IPs, in the role of a grant￾maker and knowledge broker, and also to provide capacity support to the grants it funds. SPSP has developed a grant-making program tailored to small and historically excluded organizations, such as small NGOs and Minority Serving Institutions, and new entrants to international development bringing fresh, innovative, non-traditional models to development. Besides grants, as a knowledge broker, SPSP facilitates learning across audiences and conducts special research studies (“Special Studies”) for USAID and F2F IPs. IPs appear to perceive the support they receive from SPSP positively. In the survey, questions about SPSP and its effectiveness were posed to U.S. and field-based IPs. The data reveal that 38 percent of IPs are satisfied with IPs versus 9 percent unsatisfied (the remainder “Don’t Know” or “N/A”). When asked about effectiveness of SPSP, more respondents rated the program effective than not (46 percent “Yes” versus 17 percent “No,” and 36 percent “Don't Know”). Among respondents aware of SPSP, the F2F website is the most noted service. When asked what is likely to limit the impact of SPSP, budget constraints ranked number one, as was 51 confirmed by the KIIs. While respondents almost unanimously value learning and knowledge activities, as was confirmed in KIIs, the “SPSP” as an acronym has poor name recognition, with “Don’t Know” representing more than one third of responses across relevant questions. As an overall program, SPSP is under budgeted for the ambitious multiple roles the program plays. Data from the survey reveal that 33 percent of IPs feel SPSP has “generated new ideas and innovations,” while 32 percent report that SPSP has “supported knowledge management between and within the organization.” Further, 29 percent of IPs report that SPSP has had “no significant impacts.” Only 1 percent of IPs report that SPSP has had a negative impact on their F2F Program or activities. Nearly half of IPs (48 percent) say they “don’t know” when asked which types of activities by SPSP have been most helpful for their work. SPSP was designed to be a third party objective outlier to support IPs, but is itself an IP as it manages grant-making and capacity assistance to small and medium-sized grants. This makes SPSP difficult to report on in the aggregate. That said, interviews suggest that IPs place great value on the SPSP small grants and Program Development Projects (PDPs), as well as smaller and newer sub-awards, because they bring new ideas and perspectives, and are often piloting new ideas and innovations that could be shared. SPSP leads the development and management of digital marketing media such as a website, and social media, such as Twitter. The F2F website includes a searchable database of promotional and technical resources pertaining to the F2F Program, for F2F and external audiences, http://farmer-to-farmer.org/. The evaluation team reviewed the site regularly over the last six months, and saw that it is consistently updated, fresh, and has new resources. There is also an effective search engine, where one can find everything from M&E best practices and tools, to volunteer trip reports. CONCLUSIONS Knowledge of and awareness about SPSP varies. From the KIIs, it appears that field IPs, in particular, do not make the connection between Learning/ Knowledge Management and SPSP, the service provider. However, a plurality of survey respondents finds SPSP to be effective, indicating that lack of connection may be more of a name recognition branding issue, or possible lack of validation among IPs served. Given the design and aims of SPSP, the program grapples with several management challenges. First, SPSP has a dual role in that the program is both a third party service provider as well as an internal IP representing a portfolio of small projects. Whoever implements SPSP will experience the challenge of being both an IP (internal) and third party service provider (external). This ambitious agenda requires time and resources sufficient to the tasks involved with being a both knowledge broker and grant maker or manager, including multiple skill sets and more time and resources than the budget currently allows. Second, knowledge management is often underfunded in any organization, and thus not prioritized as an IP focuses on achieving funded indicators. As an overall program, SPSP is under budgeted for the ambitious multiple roles the program plays. 52 RECOMMENDATIONS USAID can and should play the role of validating the importance of SPSP and openly communicating the level of authority SPSP should have to lead F2F’s cross-program knowledge and learning component. There are certain aspects of F2F that would benefit greatly and could advance the ToC through collaboration, such as those aimed at increasing the American public’s understanding of U.S. development efforts, require specialized skills and dedicated staff, such as website design and management, social media strategy, desktop publishing, etc. Others, such as attending conferences, have communal value as promoting the overall F2F Program. Such activities can be implemented more efficiently and effectively if they are centrally coordinated. In that way, F2F can invest only once in skilled labor and resources that support the entire community and the goals of F2F. The SPSP program warrants its own evaluation. Such an evaluation would facilitate an in-depth look at the budgetary and programmatic success and challenges of Small Grant and Program Development Project (PDP) awardees. SPSP Special Studies potentially reinforce the capacity of IPs and generate new ideas for USAID, and should be continued. Topics could include those identified by SPSP from IP surveys and other means, as well as ideas from this evaluation. Ideas for future Special Studies mentioned in the KIIs include: 1) F2F and gender; 2) Youth agriculture, food marketing and entrepreneurship for job creation and resilience; 3) Financial and supportive non-financial services for rural people and farmers; 4) Information communication technology tools to improve value chain, e.g. mobile money, ordering and inventory management by phone, pricing, and other information learning tips and tools can also be disseminated; 5) Leveraging and partnership for mutually benefits and resource mobilization; and 6) Multiplier effects: the role and power of AET and intermediaries to multiply and sustain volunteer interventions and capture the impact. 4.3.6 What impacts have SPSP sub-award projects had on core F2F activities? FINDINGS In adherence with reporting requirements, the evaluation team attempts to address this question while maintaining the anonymity of the stakeholders involved. Therefore, the team is unable to speak to the performance of SPSP and the impact of individual sub-awards. As mentioned in other areas of this report, the SPSP mechanism warrants its own evaluation apart from core IPs, to study more profoundly this and other sub-questions. SPSP is a supporting entity for the overall F2F Program. The project aims to develop new F2F implementers, reach new volunteers, and advance learning, innovation, and growth within the Program. SPSP meets these objectives, in part, through the development and management of a sub-awards program under SPSP, which includes two funding mechanisms, Small Grants and PDPs. 53 Small Grant projects aim to test innovative approaches for using U.S.-based volunteers overseas in the food and agriculture sector, draw from non-traditional sources of U.S. volunteers, and allow new U.S.-based organizations to participate in the F2F Program. These grants, seen as an entry point into the F2F community, reduce the barriers to entry by simplifying the process of receiving and implementing an F2F award, and providing enhanced guidance and award oversight to ensure that all programs meet F2F best practices. PDPs also incorporate innovative ideas into F2F programs and engage new organizations— those not currently implementing a core F2F Program—in the F2F community. In contrast to the Small Grants, the PDP program aims to encourage organizations with some experience and capacity in volunteer-sending to take on a larger role within F2F. Whereas Small Grantees’ program requirements are scaled back, PDPs are expected to follow the same guidelines and requirements as core F2F programs. This model provides PDPs with a supportive environment to successfully complete the requirements of the core implementer programs. As mentioned, small grants and PDPs bring in new implementers, fresh ideas, and are a low(er) risk/cost method for pilot testing new ideas and innovations, such as engineering for agriculture, minority serving organization (MSO) engagement. Document review as well as interviews with Small Grant awardees in the U.S. and the field indicate that small grants include emphasis on underserved award recipients, whether they are new entrants to international development, minority serving organizations, or bring a new innovation that does not exist or is under-represented in the development field. Our research uncovered that these grantees are new to international development and need significant capacity assistance from SPSP, from design to planning to implementation and close out. They may not be able to yet engage with the larger F2F community because they are pre-occupied with startup activities to get their projects performing. When these smaller grantees do become established and are ready to share with the larger F2F community, the opportunities may not be there due to resource constraints; it costs money to bring in small grants, albeit with innovative ideas, to a large gathering away from their home country. CONCLUSIONS It is difficult to evaluate the effectiveness of SPSP projects as a whole, because so many of their subgrants are new, having come on board in FY2016 or FY2017. However, the team has drawn a few conclusions detailed below. It is difficult for organizations that have never implemented an F2F program to feel like they can compete with organizations that have years of experience. This can be perceived as threatening and hamper collaboration. This is a role that SPSP should be playing, to build the bridge between small and pilot projects and core IPs. Due to the newness of some of these grantees, there has not yet been an opportunity to do this, and there are limited resources to what can be done (e.g., bringing small grant recipients to Washington for orientation and meet-and-greet, an idea that came from the KIIs). Related to the above, it may take a new small-grant award one year to get started up and running and have enough learning and experience to be able to share back with SPSP and IPs. At 54 this time, some small grants are new to the world of international development and grant funding, and the learning generated may take time to process and share. It appears that small grant participants are often excluded from larger F2F events (which by default comprise the core IPs). Perhaps this is due to the nature of the event or to the fact that a small grant may be in too early of a stage to share. However, being inclusive allows small￾grant participants and those in the fields to learn from experienced IPs and vice versa, and their knowledge and insights are valuable no matter how long they have been in F2F Program. RECOMMENDATIONS Small grants could be provided greater start up technical assistance and capacity building beyond the support of SPSP by broadening the learning to include core IPs. While SPSP is tasked with providing capacity building of small grants, hearing from and sharing ideas and challenges with IPs who have been through it could be a mutually beneficial activity, in that the core IP gets a chance to hear a new perspective from an organization it may not have otherwise worked closely with. Getting country level staff and those working in the field together in person or virtually could be a very inspiring experience generating ideas and innovation to advance the ToC and further advance the quality and value of F2F programming. Beyond special studies, it would be useful for SPSP to develop case studies on each of its small grants and PDPs to discuss their evolution and growth and how F2F has positively impacted their work and how effectively. 4.3.7 What factors might limit or facilitate the two SPSP projects’ effectiveness and impact? FINDINGS SPSP has a mandate to implement five main activities: 1) developing and managing a sub-awards program to test innovative approaches for implementing F2F; 2) providing technical services to support core F2F programs; 3) defining, documenting, and disseminating lessons in best practices; 4) organizing workshops for F2F program staff and stakeholders; and 5) completing evaluations and special studies related to F2F program implementation. When respondents of the web survey were asked what they believed to be the top constraints for SPSP effectiveness and impact, the most common response was budget constraints (44 percent) with logistical constraints the second most common response (28 percent). Among country directors, in-country staff, and recruiters, budget constraints was the most common response (43 percent), with logistical constraints the second most common response (30 percent). Document review, as well as interviews with small grant awardees, in the U.S. and the field similarly indicate that budget constraints pose a challenge for SPSP projects. In particular, it appears that the scope and expectations for SPSP projects, specifically PDPs, is comparable to F2F core projects when the budget is not comparable. SPSP’s project effectiveness is however reflected in the ways smaller NGOs bring innovative approaches and the capacity to meaningfully add to the already rich F2F Program. 55 CONCLUSIONS As an overall program, the evaluation team believes SPSP is under-budgeted for the ambitious multiple roles the program plays. As mentioned previously, the real unspoken challenge is that SPSP is both a third-party service provider and an implementer. SPSP has an ambitious agenda, but the roles and tasks involved with knowledge broker and grant maker/manager require multiple skill sets by the SPSP contractor/implementer, or simply more time and resources than the budget currently allows. RECOMMENDATIONS USAID should empower SPSP or anyone facilitating knowledge and innovation sharing to help validate their role and to help ensure IPs take their engagement in these activities seriously. It should also set aside sufficient resources for SPSP to achieve full support learning within F2F and work with implementers that have little or no experience with USAID or F2F. While there is no perfect mechanism and configuration, the evaluation team recommends that USAID maintain SPSP as a third-party mechanism to be most effective. USAID should review the way the SPSP program is structured; the current model may in fact be the best, but this needs to be discussed further and a cost/benefit analysis should be completed to explore how a project can be most supportive and how to balance the objectivity and neutrality of being a third-party facilitator with the value of knowing how volunteer programs function, having those networks, and actually implementing volunteer programs first hand. 4.3.8 How effective has the Leaders with Associates (LWA) mechanism been in promoting F2F objectives? FINDINGS F2F Program operates the core program using global LWA cooperative agreements to achieve its goals and expand its reach, while creating efficiencies by streamlining the award process for its Cooperative Agreement partners. The core IPs each manage an LWA, in coordination with USAID F2F. Each F2F LWA is an umbrella agreement. The LWAs come with core funding, which implementers use to support several country programs designed by the IP and approved by USAID Washington. Each LWA also has additional ceiling, which allows USAID Missions to easily design and fund additional F2F projects through Associate Awards. These projects achieve objectives related to agriculture, food security, and climate change adaptation. The F2F LWAs are global and enable IPs to provide volunteer technical experts or undertake Associate Awards in any F2F Program-eligible country for activities relating to any agriculture program elements. This enables F2F to have an even greater reach, and to be able to respond to more opportunities with fewer costs and turnaround time. Because each IP has already completed a rigorous and open process to secure their cooperative agreement, each IP is approved by USAID to implement agricultural activities. USAID Missions can create their own Associate Awards with the IP in lieu of going through another competitive (or sole source justification) process, which allows for new programs to be designed and implemented within a few weeks of approval. USAID Missions have several options for using the LWA mechanism, depending on activity objective, resource availability, and desired level of Mission control. The general process is for the Mission to coordinate with USAID F2F Washington to confirm that the Mission’s proposed activity fits within F2F’s 56 mandate. Missions need to provide an outline of their planned results, including objectives and areas of work, an estimate for required funds, and an approximate time period. After receiving confirmation from USAID F2F, the Mission and the IP then work together to design the final activity, budget, and any specific requirements the IP may have. In addition to the LWAs held by core implementers, SPSP has a buy-in mechanism, which is designed to meet many of the same goals. However, there are several critical differences between the LWAs held by core implementers and the buy-in mechanism under SPSP. With the buy-in mechanism, USAID Missions cannot give money directly to the implementer to fund the project. Instead, funds must be transferred to USAID Washington and obligated to SPSP, which creates an additional administrative burden. As a result of this system, the AOR for the project is the USAID Washington AOR instead of a Mission appointed AOR, which can lead to complexities in project direction and approvals. However, the current buy-in mechanism also offers something that the other LWAs do not: an open competition for the project, which provides Missions with greater access to potential implementers and proposals. CONCLUSIONS Because they are easily executed with less turnaround time than a traditional solicitation, and because they are awarded on a non-noncompetitive basis, USAID and its partners gain efficiencies while managing risk by allowing already competitively selected IPs to vet new ideas that once USAID approves, the IP is at liberty to develop further and implement with the USAID Mission or project. The mechanism is flexible, allowing F2F to maintain its core value of being “needs-based” and “flexible,” in where F2F works and on what specific food/farm projects. And though the project must include substantial technical assistance from volunteers, it can also fund other program inputs such as grants, credit, consultants, and other assistance. The SPSP Cooperative Agreement is fundamentally different because it is not an LWA. Instead, it contains a buy-in mechanism which awards projects on a competitive basis. While it is an advantage to have a mechanism available to Missions that incorporates competition, this could more effectively be done with an LWA under SPSP rather than a separate and lesser-known buy-in mechanism. RECOMMENDATIONS The team recommends that F2F keep the LWA mechanism as a way to streamline and gain efficiencies in the award solicitation and review process. The buy-in mechanism for SPSP should be replaced by the more efficient buy-in mechanism. 57 ANNEXES 58 ANNEX A: EXPRESSION OF INTEREST PEEL TASK ORDER EXPRESSION OF INTEREST – PERFORMANCE EVALUATION I. BACKGROUND INFORMATION A) Identifying Information 1. Project/Activity Title: John Ogonowski and Doug Bereuter Farmer-to-Farmer Program 2. Award Number: OAA-L-13-00002, 3, 4, 5, 6, 7, 8, 9; OAA-A-13-00053 3. Award Dates: FY14-18 4. Project/Activity Funding: 8 LWAs, ~$8M; 1 SPSP Cooperative Agreement, ~$5.6M 5. Implementing Organization(s): LWAs: ACDI/VOCA (2), CNFA, CRS, Land O’Lakes, Partners of the Americas, Winrock (2); SPSP: VEGA 6. Project/Activity COR/AOR: Gary Alex B) Development Context 1. Problem or Opportunity Addressed by the Project/Activity Being Evaluated The F2F Program goal is to generate rapid, sustainable and broad-based food security and economic growth in the agricultural sector. A secondary goal is to increase the American public’s understanding of international development issues and programs and international understanding of the U.S. and U.S. development programs. 2. Target Areas and Groups F2F Programs are global in nature, although most programs have core countries within a single region: • OAA-L-13-00002 – East Africa: Uganda, Kenya, Tanzania, Ethiopia • OAA-L-13-00003 – Southern Africa: Malawi, Mozambique, Angola • OAA-L-13-00004 – West Africa: DRC, Senegal, Liberia, Ghana • OAA-L-13-00005 – Caribbean: Dominican Republic, Nicaragua, Haiti, Guatemala • OAA-L-13-00006 – Asia: Bangladesh, Burma, Nepal • OAA-L-13-00007 – Europe, Caucasus, Central Asia: Armenia, Georgia, Tajikistan, Kyrgyzstan • OAA-L-13-00008 – Middle East, North Africa: Egypt, Lebanon • OAA-L-13-00009 – Agricultural Education and Training: Senegal, Nigeria, Guinea • OAA-L-13-00053 – Uganda, Ethiopia; Colombia; Jamaica; and others In furthering F2F’s secondary goal of increasing public understanding and involvement in international development efforts and improved international understanding, F2F Programs design volunteer assignments and scopes of work to maximize opportunities for volunteer work and interactions with rural people in the host countries. C) Intended Results of the Project/Activity Being Evaluated See Attachment A: F2F Description and Theory of Change 59 D) Approach and Implementation See Attachment A: F2F Description and Theory of Change II. EVALUATION RATIONALE A) Evaluation Purpose This Evaluation of the F2F Program has two purposes. First and foremost, it will provide guidance for planning future F2F Program activities. The last Farm Bill was delayed from 2013 to 2014. Current budget deficit talks make legislative processes unpredictable. But, it is expected that a 2018 Farm Bill will fund the F2F Program for the period FY2018-2022 (implementation period FY2019 – 2023). This Evaluation will review the goals and implementation of the Program and ensure that the program operations are efficient and effective. Information from the evaluation will be used in shaping future F2F programs. Past evaluations and subsequent analytical and design work have provided a basis for adjustments to the F2F Special Program Support Project (SPSP) and to simplify F2F standard indicators. The 2005 F2F Program Manual was revised in FY15 and FY16 to reflect updates in current implementation practices. This Evaluation will assess implementation of F2F in light of changes to the program and the operating environment. The second purpose is to identify best practices and areas in which the program can make mid-course corrections. Evaluation recommendations will be shared with current implementing organizations and adjustments in implementation made to the extent possible. Documentation on the recommendations will be provided to applicants for implementation of future F2F programs. This will enable implementing agencies to take stock of implementation approaches, strategies, organization, and results and make any changes that are indicated to improve performance. The Evaluation is being carried out following an extensive GAO Assessment of the F2F Program as completed in 2015 (see: http://www.gao.gov/products/GAO-15-478). This GAO review thoroughly examined (1) how USAID administers the program, (2) how partners implement volunteer assignments and screen volunteers, and (3) the extent to which USAID uses monitoring and evaluation to manage the program. The Evaluation is not expected to duplicate the GAO review of the entire Program implementation, but to address selected issues in greater depth. The F2F Program has had positive reviews in the past and has a strong positive reputation. However, the operating environment for the program is continuously evolving, with a changing pool of potential volunteers, different needs and priorities for hosts, and challenges in promoting broad change based on time-limited volunteer assignments. There may be value in reassessing and identifying good practice to strengthen the rationale underlying the Program theory of change. Past evaluations have noted the heavy management burdens inherent in reporting on F2F Standard Performance and Impact Indicators. The 2012 evaluation recommended simplifying the reporting indicator set. The current indicator set has eliminated a few indicators, but as a result of consultations with implementing partners, some new indicators have been included. Indicators for impacts on institutional capacity and environmental objectives are particularly challenging. There may be value in reassessing and identifying good practice to further streamline the standard indicators for the Program and identifying different indicators for organizational capacity and environmental impacts that capture Program results more effectively. The SPSP is now in a second phase of implementation. The project is intended to strengthen the overall F2F Program by involving new implementing partners, testing new approaches to use of volunteers, supporting knowledge management in implementing agricultural volunteer programs, and facilitating special studies and 60 activities on behalf of the F2F Program. There may be value in assessing and identifying good practice to improve support to the F2F Program for relevant knowledge management, quality improvement, and outreach. The Evaluation is expected to provide input related to the abovementioned purposes in a cost-efficient manner. The team will assess the implementers’ progress toward meeting their objectives and make recommendations on the future design of the program. The program, in general, appears to be operating well, as reflected by annual and semi-annual reports and site visits. Implementation issues identified during the 2012 Evaluation have been addressed and are not expected to require extensive assessment by the team. This, however, is to be confirmed by the Evaluation team’s own assessment. While this SOW may focus the Evaluation team on specific issues, the team is encouraged to provide comments and recommendations on any issues identified during the course of the Evaluation. B) Audience and Intended Uses The USAID F2F team (BFS/CSI/Technical Division) and the F2F implementing partners will use the evaluation to identify potential for changes to program strategies, implementation arrangements and approaches as appropriate to achieve F2F objectives. USAID also will use the evaluation to inform design of future F2F programs. C) Evaluation Questions The Evaluation Team will evaluate progress with the implementation of the current F2F Cooperative Agreements against the stated objectives of the F2F Program. The Evaluation team will need to be creative in using existing data and reports to assess expected effectiveness and efficiency of the program. In this the Evaluation Team will need to draw on documented experience from earlier F2F programs, especially the FY09-13 F2F Program, and from preliminary results achieved to date. Using that information, the report will assess F2F Program experience and strategies and provide empirical evidence to support conclusions and recommendations, with focus on the following questions. 1. How strong and effective is the F2F Program Theory of Change? How effectively do Program activities (i.e., volunteer assignments) lead to desired impacts? How effectively are F2F interventions (country projects, hosts, host development plans, volunteer SOWs) selected and developed to lead to change? How effectively are volunteers recruited, selected, matched to SOWs, motivated, and prepared for their assignments? How well do volunteers complete SOWs and provide desired deliverables? To what extent do hosts change behavior and/or perceptions as a result of F2F interventions? To what extent are persons trained/directly assisted drivers of change in relevant host organizations? 2. How well do F2F standard indicators capture the effects of F2F activities (particular emphasis on indicators for outreach/cultural exchange, capacity development, environmental/natural resource management) and provide useful information for Program management? How well do the indicators for F2F serve purposes for program management, monitoring and evaluation? Are there indicators that present particular difficulties for collection and management interpretation? Are there indicators that might be simplified or eliminated or are any others needed? Are the Program’s performance and impact on outreach/cultural exchange, capacity development, environmental/natural resource management being captured effectively? 3. How well does the F2F Program learn and integrate innovation to encourage quality improvements and good practice in agricultural volunteer programs? 61 How well do F2F implementers engage in a community of practice to improve the program? In what ways could the program improve learning innovation to become more efficient and effective? What learning and innovation activities or community of practice support are particularly effective or needed and what may be of lesser effectiveness or utility? Are there unaddressed needs for support to core F2F Program quality and good practice? What are key issues for future success of the F2F program? How effectively have SPSP projects (including sub-awards) supported F2F learning, innovation and community of practice and how is that support perceived by the F2F implementers? What impacts have SPSP sub-award projects had on the F2F program and its activities? What factors might limit or facilitate the two SPSP projects’ effectiveness and impact? How effective has the LWA mechanism been in promoting F2F objectives? For each of the three above questions, the evaluation team should attempt to identify ways in which current processes can be improved. The team should discuss good practices for agricultural volunteer program planning and implementation, discuss any limitations of current practices, and suggest methods to address any limitations. The evaluation team is not required, but is encouraged, to comment on any other aspects of the F2F Program planning and implementation that they encounter that may be relevant to improving the Program’s efficiency, effectiveness, and benefit sustainability. III. TIMEFRAME & TRAVEL A) Timeframe The Evaluation should start on/about October 2016 and shall be completed no later than June 30, 2017. Work of the Evaluation Team members does not have to be continuous and may be phased as appropriate to the most efficient, effective and convenient completion of the required Evaluation. B) Travel TBD – The team shall visit selected country programs to obtain insights to program operating procedures, reporting and activities, for a total of 3 weeks. Selection should attempt a balance among different implementers. Since country conditions must be considered and USG country clearance will be required, the team shall select proposed sites to be visited and the composition of the teams for each country and obtain USAID approval for the required travel. It is expected that the Evaluation Team will split up to cover as many countries as possible within the limited timeframe and limited resources. IV. DELIVERABLES & DESIGN A) Deliverables All deliverables listed in section F. 7 of the PEEL-Task Order contract. B) OPTIONAL: Evaluation Design The Evaluation will have to be carried out within a limited resource envelope. The Program operates in about 30 countries under multiple implementers and project types. The Evaluation team will be able to visit a few selected F2F program countries and will have to draw from multiple sources for data and insights in carrying out the Evaluation. The evaluation team will finalize and elaborate the evaluation approach in the evaluation plan. However, a mixed methods or process evaluation approach to collecting and analyzing qualitative and quantitative data is recommended to arrive at robust findings and conclusions and fully address all the evaluation questions. 62 The evaluation should review the experience and impacts of the FY2009-13 F2F Program (core programs and SPSP) as reflected in final reports, as well as the experience to-date in the current FY14-18 program. The programs are largely similar, such that the FY09-13 Program may suggest results that can be expected from the current Program. Assessment of potential impact from any changes between the programs, if any, would be of interest. V. TEAM COMPOSITION USAID envisions a team of four Key Persons including an Evaluation Specialist/Team Lead, a Research Analyst, an Agricultural Development Specialist, and a Technical Agriculture Specialist. Taken together, the team must have familiarity/experience in six areas: a) agricultural development, technology transfer, organizational and capacity development; b) gender analysis; c) quantitative and qualitative evaluation, including formulating questionnaires or surveys, and knowledge of cultural exchange measures; d) USAID-agricultural projects; e) the US agricultural sector; and f) agricultural volunteer program management, implementation or service work. All team members must also have the ability to analyze issues and formulate concrete recommendations orally and in writing. A mix of resources is proposed below but the team composition and individual skill/experience mix may vary, as determined by the Team Leader and in collaboration with USAID. Evaluation Team Leader – 65 work days A senior-level evaluator with a minimum of 10 years of experience designing, managing and/or evaluating multifaceted international development teams, involving farmers, private sector, and public sector involvement. S/he will provide leadership in the conceptual design of the project and must have strong project management skills. The Team Leader must have experience with agricultural development and capacity building activities and experience with monitoring and evaluation and project/activity design experience. S/he should understand how to use the private sector as the engine of economic growth, have experience in agribusiness or agricultural economics, understand the dynamics of institution building, human capacity building, the value chain, and/or have experience working with volunteers and consultants, monitoring and evaluation experience, project/activity design experience, and an appreciation of the need for and complexities of applied agricultural research. The candidate will also have: a) a demonstrated capacity to conduct independent program evaluation; b) project/activity design experience; and c) an understanding of USAID’s foreign assistance goals, and its particular objectives related to agricultural development and food security. Research Analyst – 50 work days Must be an experienced analyst (5 years or more) with research and/or socio-economic analysis in international development related to agriculture. S/he will take the lead in data collection and analysis as well as conducting surveys. This team member will have an excellent understanding of the tools and research methodologies for monitoring and evaluating agricultural development programs. S/he should have experience assessing the impacts of technology transfer and capacity development on various types of beneficiary organizations, as well as in assessing the adequacy of monitoring and evaluation indicators in reflecting the accomplishments and impacts of development and capacity building activities. The qualified candidate must also demonstrate experience recommending practical and pragmatic development indicators. S/he must have some familiarity with the US agricultural community, NGO programs, and volunteer programs. Agricultural Development Specialist – 50 work days Must be experienced expert (5 years or more) in international development related to agriculture. Team member must have experience developing and implementing technical approaches to agricultural development and capacity building activities, particularly related to technology transfer, capacity development, and value chain development. S/he should understand production systems, producer and community organizations, the dynamics of institution building, human capacity building, and agricultural research/extension/education institutions. Team member will also have demonstrated experience in effectively conducting outreach and dissemination to policy makers, development practitioners and/or the private sector. S/he should have experience with production technology and 63 production projects oriented toward market demands. Experience with agribusiness, agricultural value chains, and working with volunteers and consultants is desirable. Technical Agriculture Specialist – 40 work days Must be experienced agricultural professional (5 years or more) with knowledge and contacts in the US agricultural sector. Experience with international agriculture is desirable. S/he should have experience in agribusiness, value chains, and farm production. S/he should understand institutional and human capacity needs for an effective agricultural sector. S/he should understand the science and technology behind current agricultural systems, and have knowledge of international agricultural standards and market entry requirements. The Level of Effort (LOE) and professional qualifications of the specialists may be inter-changed. The LOE also may be allocated differently, with for example a fifth Specialist included with the same total LOE (205 work days). Specialists should have university degrees in fields relevant to agriculture and rural development. No specific degree qualification is indicated as this requirement will be subsumed by the requirement for experience in international development. Attachment A: F2F Description and Theory of Change General Description of Farmer-to-Farmer Program The F2F Program goal is to generate rapid, sustainable and broad-based food security and economic growth in the agricultural sector. A secondary goal is to increase the American public’s understanding of international development issues and programs and international understanding of the U.S. and U.S. development programs. In furthering the secondary goal of increasing public understanding and involvement in international development efforts and improved international understanding, F2F Programs design volunteer assignments and scopes of work to maximize opportunities for volunteer work and interactions with rural people in the host countries. Farmer-to-Farmer Theory of Change 64 Farmer-to-Farmer Program Approach The F2F Program facilitates delivery of a broad range of U.S. agricultural expertise, using American volunteers who work with farmers, agricultural support systems and agribusinesses in developing countries. F2F has been an effective, low cost program that blends two goals—providing technical services to enhance sustainable economic development and opportunities for good people-to-people level exchanges. The F2F volunteer program by its nature involves a number of fundamental trade-offs that must be balanced in implementation. These generally do not require either/or decisions, but present challenges to planning and implementing the F2F programs so as to maximize impact and respond to the multiple demands on the volunteer program. These trade-offs include: a) Development impact vs. the volunteer experience: The key objective in the program is developmental impact. At the same time, an important subsidiary objective is to provide the volunteer with meaningful cross-cultural experience and understanding of development and foreign policy issues. The balance in this affects such decisions as: how often to use repeat volunteers, as opposed to maximizing numbers of discrete individuals placed as volunteers; how much emphasis to put on outreach activities by returned volunteers; how much to spend on support costs to make volunteer services more effective; and what types of volunteer assignments to develop. b) Poverty reduction vs. economic growth: These two objectives can be quite compatible. Volunteers and implementing partners often prefer to work directly with the poor rather than with larger firms and farmers, and USAID strategy supports broad-based economic growth and poverty reduction. Still, larger farmers and businesses may be better able to use volunteer services and might ultimately have broader, but indirect, impacts on the poor. Often the need is to enhance spread of innovations and benefits from volunteer assignments to prevent “capture” by elite hosts. c) Poorer vs. wealthier countries: Similar to the issue with type of hosts, countries with more dynamic economies and better regulatory and governance systems are best able to use services and benefit from volunteers. However, needs may be greater in countries with less developed private 65 sectors and market economies. While a bad policy environment may limit ability of volunteers to influence change, volunteer assistance can benefit the immediate host organizations and can help establish a base for future policy reform that improves a country’s prospects for development. d) Immediate vs. long term impacts: As development program, F2F must recognize that many activities may have broad impacts, but only in the long term. It may be more effective to work on regulatory or extension programs that ultimately will affect many farmers than to work with a few individual agribusiness firms. Still, volunteers and funding agencies want to see and be able to report on immediate impacts that come from activities with short-term objectives. e) Demand vs. supply driven: Programs typically should seek volunteers to meet needs of hosts rather than seeking host assignment opportunities for pre-identified potential volunteers. Still, volunteer supply considerations are important to providing volunteer opportunities to qualified people and to being able to provide volunteers to fill specific host requests. f) Focused activities vs. targets of opportunity: Volunteers can work with host institutions throughout the agricultural sector, and good assignments often come up in diverse sub-sectors. Experience has shown limiting work to one sub-sector allows for better planning, synergistic assignments and greater impact. The concept of a ‘demand-driven’ program that responds only to needs of hosts argues for providing volunteers exclusively in response to host requests. Such an approach, however, need not imply a commitment to responding to all sub-sectors, as responsiveness to hosts can be limited to pre-identified sub-sectors. Even with the most extreme demand-driven approach, requests must be for activities that are legal, viable and in the public interest. Program parameters can impose appropriate restrictions on host eligibility and program promotion to focus work on industries or sub-sectors expected to produce high impact, while still responding to host demand within the defined sub-sector. g) Maximizing number of volunteers vs. ensuring adequate support: Programs must seek to hold costs down and make volunteer services cost effective. Adequate logistical and planning support is essential to volunteer safety, morale and effectiveness. Qualified program staff are needed to plan, support, follow up and report on volunteer assignments. h) Longer vs. shorter assignments: Longer assignments appear more cost efficient in that they lower cost per volunteer-day of services. Many potential volunteers are unwilling or unable to serve for longer that two to three weeks and some hosts do not want to spend more time with volunteers. The two to three week assignment has been the standard for F2F programs and has proven successful. Ultimately, the scope of work and length of assignment must be defined by the requirements of the hosts, country sector needs, and ability to recruit qualified volunteers. Alternative assignment options may provide volunteer opportunities to other populations of potential volunteers and offer new services to hosts. This might also allow for use of intern￾volunteers or volunteer consultants to support implementation of the volunteer program itself. i) Supporting partner programs vs. achieving independent results: Volunteer programs generally work best when volunteers work with an on-going program that provides a framework for their assistance and follow up as necessary. F2F program effectiveness is enhanced when it draws on other resources to support its work. USAID and other donor projects are often good partners, but use of volunteers to substitute for or duplicate work of consultants available from other programs is questionable. F2F volunteer assignments should add value, not simply replace other donor or private sector resources. j) Volunteer services vs. complementary support: Complementary support (training, additional technical assistance, travel by hosts, demonstration grants and pilot programs, etc.) can often facilitate adoption of volunteer recommendations and expand impact from volunteer assignments. F2F Program funding is limited and funds only volunteer services. Other USAID funding, if available, 66 does not have the same restrictions and can fund needed complementary support, either through Associate Awards or through parallel-financed projects. Farmer-to-Farmer Program Structure F2F Programs vary depending on country and sector needs and conditions, but all follow generally similar approaches for planning and fielding volunteers under Country F2F Projects and as flexible volunteer assignments. Definitions used in the F2F Program are defined below: • • • • • • • • • • Core Country: a country with one or more Country F2F Projects Country F2F Program: the F2F activities in a specific country Country F2F Project: the F2F activities in a country focused on a specific sub-sector objective Country Partner: an institution, project or program with which F2F works in country to plan and implement Country F2F Projects Farmer-to-Farmer Program: the over-all F2F program administered by USAID Flexible assignment: a volunteer assignment undertaken outside of a Country F2F Project Host project: may be used to refer to all F2F activities with one host Host: a local institution or person receiving assistance from one or more F2F volunteers Implementing Partner: the Cooperative Agreement organization implementing an F2F Program F2F Program: the program administered by an individual with funding from this RFA Country F2F Project Descriptions are used to summarize the problem to be addressed, important implementation issues, how volunteer services and other resources will be used, and results expected from work on the defined country subsector. The following is an outline for a proposed Country F2F Project. 67 Outline for Country F2F Project Descriptions 1. Objective (approx. one sentence) 2. General description of sub-sector being targeted for assistance and why it is targeted 3. Description of the key problems/constraints to be addressed or opportunities to be exploited 4. Proposed F2F activities a. How will volunteers be used? (Numbers and types of volunteer assignments planned) b. What are the key country partners (if any) and their roles c. What are likely target hosts (Number/type of expected hosts; criteria for identifying them) d. How can volunteer recommendations be shared broadly to maximize impacts 5. Any key feasibility issues (economic/financial, institutional, environmental, social, technical), including some economic assessment relating costs to benefits. 6. Other resources available/needed to accomplish desired objectives (partner projects, loans, etc.) 7. Logframe or similar results framework summarizing project design with quantified targets for numbers of volunteers, hosts, beneficiaries and impacts. Note: the Country F2F Project Description should provide evidence of understanding the sub￾sector for the Project and planning for expected activities along with clear targets for performance and impact indicators. It is understood that experience during the course of implementation may lead to substantial changes in activities, hosts, targets and other aspects of the Project. Flexible assignments are any F2F volunteer assignments programmed outside of Country F2F Projects. Flexible assignments: respond to targets of opportunity for substantial developmental impact, carry out exploratory activities in new areas or sectors, and respond to unforeseen needs and opportunities of USAID Mission programs and partners. Up to 15% of volunteer assignments may be used for flexible assignments in eligible countries and sectors outside of Country F2F Projects. They generally require little additional management or staff costs and are approved in Annual Work Plans. Flexible assignments may be in any country, within or outside the geographic region for the core country F2F programs. Flexible assignments are generally not undertaken in countries in which another implementer has an on￾going Country F2F Program. F2F Programs retain flexibility to adjust activities based on experience and changing conditions. Most Country F2F Projects are five-year activities, but some may be shorter or shortened if experience so indicates. New Country F2F Projects can be initiated during the course of the five-year program. Geographic Focus: Countries and Regions The Leader Award F2F Program will be global in scope, but with core country F2F programs in a limited number of core countries typically within a given geographic region. Country F2F Programs are limited to eligible countries in the geographic regions defined in the Table below. 68 Table: F2F RFA Eligible Countries and Geographical Program Regions F2F Focus Region Feed the Future Core Countries Feed the Future Aligned Countries Other Countries Caribbean & African Regions: Caribbean Basin: Guatemala, Haiti, Honduras Dominican Republic Belize, Costa Rica, Dominica, El Salvador, Grenada, Guyana, Jamaica, Nicaragua, Panama, St. Lucia, St. Vincent and the Grenadines East Africa: Ethiopia, Kenya, Rwanda, Tanzania, Uganda Southern Sudan Congo, Comoros, Djibouti, Seychelles, Sudan Southern Africa: Malawi, Mozambique, Zambia Zimbabwe Angola, Botswana, Lesotho, Madagascar, Mauritius, Namibia, South Africa, Swaziland West Africa: Ghana, Liberia, Mali, Senegal Democratic Republic of the Congo, Nigeria Guinea, Benin, Burkina Faso, Cameroon, Burundi, Cape Verde, Central African Republic, Chad, Republic of the Congo, Cote d’Ivoire, Equatorial Guinea, Gabon, Gambia, Guinea-Bissau, Mauritania, Niger, Sao Tome and Principe, Sierra Leone, Togo Non- Caribbean/African Regions: Asia: Bangladesh, Cambodia, Nepal East Timor, Indonesia Afghanistan, Burma, India, Mongolia, Pakistan, Philippines, Sri Lanka, Vietnam Middle East & North Africa: Egypt, Lebanon, West Bank & Gaza, Yemen Jordan, Morocco Europe, Caucasus & Central Asia Tajikistan Georgia, Kosovo, Kyrgyz Republic, Serbia, Ukraine Albania, Armenia, Azerbaijan, Belarus, Bosnia and Herzegovina, Kazakhstan, Moldova, Turkmenistan, Uzbekistan Latin America: Bolivia, Brazil, Colombia, Ecuador, Mexico, Paraguay, Peru There are various organizations implementing F2F program activities. Programs generally do not overlap geographically but will engage in prior consultations before any overlap of country F2F activities. Management Information System (MIS) The F2F Program requires a management information system (MIS) adequate to inform program management decisions and report routinely on inputs, activities, outputs and outcomes. The MIS will report against: a) Standardized F2F Program Indicators, b) Country F2F Project-specific performance and impact indicators. The Standard F2F Performance and Impact indicators will be reviewed and finalized in consultation with USAID and other F2F Program Implementing Partners. Since USAID is required to report on performance and impact of the overall F2F Program covering the programs and activities of various implementing partners, common definitions are required for the Standard F2F Indicators. 69 Standardized F2F Program Indicators should be generally relevant to Country F2F Projects, but it is likely that other indicators, specific to the activities and objectives of the specific Country F2F Project, will be needed to assess performance and impact. USAID country Missions may provide input to such country-specific indicators to assist with their needs for reporting on development impacts in the country. Such indicators should be identified in the Country F2F Project Descriptions to track and report on implementation progress and program impacts. F2F Program Implementation Requirements The will establish necessary institutional capacity and management systems to effectively and efficiently implement F2F volunteer assignments in support of the objectives of the F2F Program. While other functions and activities may also be required, the will establish systems required to carry out the functions, as described below. a) Sector Analysis and Planning: Earlier F2F programs assigned volunteers wherever good hosts were found, but evaluations indicated that greater impact was achieved when activities were focused on targeted commodity and service sub-sectors. Past F2F Program Assessment noted the need for a balanced assessment of the agricultural sector and potential sub-sectors in selecting and designing country projects. Analyses will include, as appropriate, economic, social, environmental, and institutional issues, with emphasis on development of private sector market chains for agricultural products. Various methodologies can be used in analyzing agricultural sector needs and opportunities. Some of these include: sub-sector analysis, supply chain analysis, value chain analysis, industrial organization, food systems framework, or SWOT (Strength￾Weakness-Opportunity-Threat) analysis. Because of limitations on funding for F2F programs, the will make maximum use of sector analytical work available from other sources, such as USAID Mission projects, other donors, host country governments, private institutions, and others. b) Selecting Country Partners: Country Partners may assist in development and implementation of the country F2F program and project strategies and plans. Such partners can be public or private agencies and can include trade associations, NGOs, commercial firms, projects, government departments and programs (e.g., Peace Corps) or educational institutions. Partners are commonly involved in providing goods and/or services to agricultural producers or agribusinesses. The clients of such partners are often the public and private sector organizations that are potential hosts for volunteers. Roles of partners vary, but commonly include serving as: a) local representative for the F2F program implementer; b) retailer of volunteer services; c) link to clients (hosts); or d) technical backstop. In some cases, volunteers may work with partners to strengthen the partner’s own capacity to provide goods and services to their clients (in which case the partner is also the “host” organization). Generally, good partners will help to see that the specific knowledge or skills that volunteers bring to a host organization are spread more widely to benefit the sector. Working with and possibly embedding volunteer technical assistance and Country F2F Project implementation within a local Country Partner organization may serve to advance potential for sustainability of development activities in line with the objectives of the USAID Forward policies. c) Selecting hosts: Hosts are organizations or individuals that request volunteer services and work directly with the volunteers. Host organizations commonly include private businesses, community groups, trade or business associations, local government offices, cooperatives, or 70 public or private service providers (inputs, credit, information, marketing, processing, storage, etc.). Hosts may be the ultimate beneficiaries (typically small farmers) that the F2F Program is designed to assist, but may also be intermediaries (service providers, agribusinesses, etc.) whose improved operations benefit the target populations. The ability of host organizations to absorb, utilize, and disseminate information provided by volunteers is a critical consideration in determining the impact of a volunteer’s assignment. Host organizations should be involved early on in developing the F2F program strategies and plans. They should also, where possible, contribute to the costs of fielding volunteers and assist in assessment of the impact of the volunteer’s efforts. The impacts of a volunteer program are primarily a function of what host organizations do as a result of the information and training provided by volunteers. Recruiting from active farmers is encouraged. d) Development of assignments: The will establish systems to develop scopes of work (SOW) that describe services required by host organizations and identify the skills and experience volunteers will need for the assignment. SOWs may also include background on the sub-sector and host project, relating the specific volunteer assignment to the Country F2F Project strategy and plan. The SOW identifies the results anticipated from the assignment. The SOW may also identify complementary inputs that are expected to be forthcoming from the host and other partners. SOWs translate program and project plans into specific tasks for volunteers. Ideally, there should be a high degree of participation by host organizations in the preparation of SOWs. The SOWs document the understandings reached among the parties involved, namely the host organization, any partners, and the F2F program implementer. The clearer the SOW can be in terms of specifying who, what, where, when, why, and how, the easier it will be to recruit for the volunteer assignment, guide the volunteer once on the job, and finally assess the success of the volunteer assignment. The process of approving a SOW also clarifies the roles of the implementing agency, the volunteer(s), any partners, and the hosts in making the assignment a success. Ideally, individual volunteers work with farmers and target beneficiaries to develop personal relationships and mutual understanding. e) Volunteer recruitment: The F2F Project requires systems for identifying and recruiting volunteers to provide services as per defined SOWs, including consider options for recruiting from new or non-traditional sources of volunteers to maximize the pool of volunteer resources available to the F2F program and to maximize the potential for public education and outreach. The F2F Program will support services of U.S. volunteers (U.S. citizens and permanent residents); non-U.S. volunteers will not be used, unless prior approval is obtained from the USAID AOR for any unique situation requiring such use of non-U.S. volunteers. f) Volunteer orientation: The F2F Program requires systems for appropriate volunteer orientation, including information about travel, health and security issues, other logistics and support, the country, and the organizations and individuals with whom the volunteer will work, as well as the specific tasks of the assignment. The orientation process begins as part of volunteer recruitment and extends through the initial days of their arrival in the host country. The more informed volunteers are about what they are getting themselves into, the better equipped they will be to make their assignment successful both for the host as well as for themselves. g) Volunteer logistics and support: The F2F Program requires systems to ensure volunteers are provided with adequate logistical and technical support to provide efficient and effective 71 services to hosts. The logistics and support services greatly impact the effectiveness of volunteers. Logistical support includes transport, housing, translators, secretarial assistance, compensation arrangements, and other services. Technical support includes information, links to local technology programs, teaching aids, and equipment. Volunteers that are appropriately supported will have more time to focus on their assignments and will develop a better understanding of the host’s operating environment and constraints. Technical support can help volunteers adapt their technical knowledge and skills to local conditions and develop appropriate linkages to local technical support institutions. Such support is particularly important given the relatively short duration of many volunteer assignments. They will provide adequate medical and evacuation insurance, vaccination and health briefings, in-country security and other support to ensure safety and wellbeing of volunteers. h) Complementary support to hosts: The F2F Program must consider needs and make appropriate arrangements, as far as is possible, to ensure that complementary services (e.g., financial services, equipment and infrastructure, input supply, marketing, and processing arrangements, etc.) are available to enable hosts to make use of volunteers’ recommendations. Complementary support services represent the range of resources available to enable host organizations to operate effectively, including such things as financial services (especially credit), equipment and infrastructure, input supply, database services (e.g. marketing and pricing information), marketing, and processing arrangements. These services are generally not within the scope of an individual volunteer assignment, but volunteers’ recommendations often implicitly or explicitly require a host to draw on such resources to implement recommendation. Thus, while a volunteer may complete his/her assignment satisfactorily without such services, service availability varies according to the environment in which the host operates and they are often critical determinants of the level of impact from a volunteer assignment. F2F programs with limited budgets are not able to provide complementary support to hosts, but may be able to facilitate access to such services from other programs. i) Project staffing: The F2F Program requires adequate staff and staff support resources for efficient and effective implementation of the volunteer program, including the planning, implementation, and monitoring and evaluation of the F2F program. j) Volunteer outreach activities: The F2F Program requires systems to encourage and maximize the amount and impact of outreach activities carried out by returned volunteers. Outreach encompasses the full set of activities designed to inform both the general public and particular interested parties of the character and accomplishments of the F2F program. Outreach has three principle benefits: 1) educating the public on global development issues and market opportunities; 2) involving the public in international development; and 3) recruiting new volunteers. Past F2F program assessments noted the need to strengthen outreach activity by returned volunteers, perhaps requiring an increase in funding support for such activities. k) Associate Awards: The F2F Program requires capability to respond, as appropriate, to requests from USAID Missions or offices with proposals for Associate Awards under the Leader Cooperative Agreement. Associate Awards may provide for: a) additional volunteer services in a core F2F country or other country; b) complementary support for Country F2F Project sub￾sectors (additional technical assistance, grants, training, equipment and facilities, and/or other inputs) that can improve the F2F Program outreach and impact; or c) volunteer services and complementary support for agricultural projects addressing a specific development objective in 72 an F2F core country or other country. Associate awards may expand F2F program activities into additional countries within or outside of the given region. Other Critical Implementation Requirements The F2F Program requires attention to planned and unplanned results and impacts, including especially as may relate to USAID policy issues of importance to the F2F programs, as discussed below. a) Gender: The F2F Program requires provide systematic consideration of gender issues and impacts in the F2F program and will seek to promote gender equity. Each Country F2F Project description will assess gender issues relevant to the sub-sector and measures the project may take to enhance positive gender impacts. Although gender analysis considers impacts on both men and women, in practice, the concern is nearly always ensuring equitable participation and access to benefits by women. Biases against women are pervasive in much of the world, as are differentiated roles between the sexes. Programs can have unintended negative impacts on women. For example, economic development has occasionally increased the gap between the status of women and men by introducing technologies and crops that require more labor (e.g. weeding, watering) from women while men control the profits from the increased yields. Benefit distribution is a key consideration for ensuring gender equity in development. Furthermore, it is not uncommon for development programs to fall short of their goals (e.g., improvements in health and education) specifically because women have not been permitted to participate fully in program design and implementation. The will assess gender impacts and gender equity issues in all Country F2F Programs. b) Environment: The F2F Program requires ensure systematic consideration of the implications of the F2F Program activities on environmental and natural resources and should seek to mitigate any adverse impacts. Economic activities impact on environmental and natural resource resources. In the case of agriculture, development may be completely dependent on natural resources and potentially deplete them or degrade environmental resources. A healthy environment and the sustainable use of natural resources are essential to long-term economic growth. All U.S. government financed programs, including the F2F program, are obligated to comply with U.S. environmental regulations stipulating that the environment is to be safeguarded from adverse consequences in the implementation of all program activities. USAID's environmental guidelines require programs that it funds to: • Ensure that the environmental consequences of USAID-financed activities are identified and considered by USAID and the host country prior to a final decision to proceed and that appropriate environmental safeguards are adopted; • Assist developing countries to strengthen their capabilities to appreciate and effectively evaluate the potential environmental effects of proposed development strategies and projects, and to select, implement, and manage effective environmental programs; • Identify impacts resulting from USAID's actions upon the environment, including those aspects of the biosphere which are the common and cultural heritage of all mankind; and • Define environmental limiting factors that constrain development and identify and carry out activities that assist in restoring the renewable resource base on which sustained development depends. 73 The Initial Environmental Evaluation (IEE) completed by USAID for the F2F Program made a Recommended Threshold Determination of a Negative Determination with Conditions. The majority of F2F Program activities qualify for Categorical Exclusion under 22CFR216.2(c): • • • • per 22 CFR 216.2(c)(2)(i) for activities limited to education, technical assistance or training programs except to the extent such programs include activities directly affecting the environment (such as construction of facilities, etc.) per 22 CFR 216.2(c)(2)(iii) for activities limited to the performance of analyses, studies, academic or research workshops and meetings. per 22 CFR 216.2(c)(2)(v) for activities limited to document and information transfers. per 22 CFR 216.2(c)(2)(xiv) for activities limited to studies, projects or programs intended to develop the capability of countries to engage in development planning, except to the extent designed to result in activities directly affecting the environment However, even though the F2F program will be essentially training and advisory activities with no direct effect on the environment, the subject of the training or advice may involve use of pesticides. While there will be no procurement or physical use of pesticides, the training in how to decide when to use them and how to use them safely can be seen as constituting a “use.” This requires that some environmental safeguards be put in place on any involvement with pesticide use. The same holds true for any involvement with genetically modified organisms (GMOs), which are politically sensitive in some countries and which present some environmental dangers. The F2F Program must ensure that no pesticides are provided, used, or sold under the program. Any pesticide-related training will be in an IPM context and emphasize training on safe use of pesticides. Training and recommendations for use will be in an IPM context where pesticide use is the last resort. The focus of the training will be on IPM and the environmentally responsible and safe use of pesticides when they are needed. They will ensure this caveat is included as a standard element in briefing materials provided to all volunteers prior to their traveling. No genetically modified organisms will be provided, used or sold. Only the benefits and disadvantages of GMOs will be discussed during any training. If research, provision, use or recommendation for use of a GMO is subsequently planned, it will first be submitted to USAID for clearance by the USAID bio-safety committee and subsequently the information flowing from the USAID bio-safety committee will be used as the basis for an amendment to this IEE to make a final determination whether such GMO may be researched, provided, used, or recommended for use. This IEE amendment must be approved in writing by the BFS Bureau Environmental Officer prior to a final decision or expenditure of funds to research, provide, use or recommend the use of a GMO. Given the sensitivity of both the use of pesticides and the use of GMOs, they will ensure that the following specific conditions are applied in its F2F Program: • • The syllabus for each training event will be reviewed and approved by the Mission environmental officer and/or the USAID AOR. A representative from USAID (preferably the Mission) should attend the training sessions to the extent possible. 74 • • • • If any specific pesticides are to be used, procured or recommended for use, the USAID Pesticide Procedures (§216.3(b) must be followed. An amended IEE will be prepared prior to the training and must include a separate section evaluating the risks and benefits of the proposed use. The 12 factors outlined in Section 216.3(b) will be analyzed in that evaluation. This evaluation is known as a Pesticide Evaluation Report and Safe Use Action Plan (PERSUAP). If the training includes recommendation for use of Genetically Modified Organisms (GMOs), a Biosafety Analysis will be required and an amendment to this IEE will be prepared. The extent of such an analysis will depend on the type of activity proposed and the current status of acceptance of the GMO. If the activity will have a significant adverse effect on the environment, based on the analysis in an amended IEE, a full Environmental Assessment will be required. They will familiarize all staff with the summary “F2F Program Environmental and Natural Resources Management Guidelines for Implementing Partners” (Annex E) and with F2F guidelines for volunteers. The F2F Program must ensure that volunteers consider potential environmental consequences of their work and promote active environmental stewardship. All local laws regarding pesticide or GMO regulation will be respected. Any variation to these principles must be preceded by USAID approval of an amended IEE. If, during implementation, activities are considered other than those described in this RFA as the basis for the categorical exclusions, they will inform the USAID AOR and request approval for an amended IEE before proceeding with such activities. They may use funds from the Cooperative Agreement to pay for the cost of undertaking any required IEE amendments. The training of applicators, vendors, decision makers, and farmers will improve IPM decision making and reduce pesticide misuse, thus benefiting the agricultural areas. The eventual effect on the environment is expected to be highly beneficial as trained host institutions and individuals are likely to make better decisions regarding pest management technologies and, when pesticides are used, they will be used in a safe and environmentally responsible manner. Given the wide use and importance of pesticides in agricultural production systems, they should plan to collaborate in preparation of a programmatic PERSUAP for the F2F Program within the first six months of award. c) Restricted sub-sectors and activities: The F2F Program does not support F2F Program activities that are restricted by USG policies and restrictions. USAID policies and regulations restrict support for certain activities involving: tobacco production and processing; African oil palm production or processing; degradation or conversion of tropical forests; agricultural products in surplus on world markets; assistance to activities or projects of Communist countries; assistance to military, police or prisons; luxury goods; alcoholic beverage production or marketing; export of U.S. jobs, relocation of U.S. businesses or restrictions on internationally recognized workers’ rights; or production of agricultural products that compete with U.S. products in international markets. Activities involving support for any host projects that might involve support to such sub-sectors, products, or activities will be discussed with USAID before initiating or making any commitments to work with hosts on such activities. They will obtain 75 prior approval from the USAID AOR in Annual Work Plans or activity-specific approval prior to undertaking work on any activities involving these products and sub-sectors. USAID will not support the growth of tobacco as a cash crop, nor will it support agribusiness activities contributing to tobacco production, promotion, and use. In countries in which tobacco is a major cash crop and an important source of income for low-income farmers, USAID may work with local agricultural interests to identify crops that are economic alternatives to tobacco and may support the introduction or expansion of these alternative crops. Attachment B: Key Reference Materials for Farmer-to-Farmer Program Evaluation • • • • • • • • • • • • • • • • • Farmer-to-Farmer Program Annual Report FY16 Farmer-to-Farmer Program Annual Report FY15 Farmer-to-Farmer Program Annual Report FY14 Gilbert, Alex, Navin, Benschoter, Cullen, and Flannery, “Managing International Volunteer Programs, A Farmer-to-Farmer- Program Manual,” for USAID/EGAT, PN-ADC-356, March 2005. Updated 2016. GAO review of Farmer-to-Farmer Program (2015) Cooperative Agreements, Work Plans, Semi-Annual and Annual Reports, country project descriptions for FY14-18 F2F Programs Final Reports for F2F Programs FY09-13 Lessons Learned from F2F Programs FY09-13 USAID F2F annual reports, special studies, on-going analysis, and official letters to the implementing partners concerning their reports and plans US Farm Bill PL 480 legislation governing the F2F program. 2013 RFAs for John Ogonowski and Doug Bereuter Farmer-to-Farmer Program Cooperative Agreements USAID Agricultural Strategy USG Feed the Future Strategy for Global Hunger and Food Security Initiative Cooperative Agreement, Work Plans, VEGA SPSP documentation on PDP and small grant projects; and Progress and Special Reports for the F2F SPSP, including sub-project agreements, work plans and reports for the sub-projects. Task Order, Work Plans, Weidemann SPSP documentation on PDP and Niche projects; and Progress and Special Reports for the F2F SPSP, including sub-project agreements, work plans and reports for the special projects Reports on F2F Implementers Meetings—Istanbul (2009), Washington (2010), and Volunteer Appreciation Event (2010), Morocco (2014), 30th Anniversary Event (2015) John Ogonowski Farmer-to-Farmer Program Evaluation (2012) 76 ANNEX B: EVALUATION PROTOCOL 77 EVALUATION PROTOCOL: The USAID Farmer-to-Farmer Theory of Change and Impact Learning Evaluation 2017 Submitted: March 1, 2017 78 This document was prepared by ME&A,14 for the United States Agency for International Development under USAID Task Order AID-OAA-I-15-00024/AID￾OAA-TO-16-00008. The views expressed in this document and its annexes do not necessarily reflect the views of USAID or the US Government. USAID/Washington Contact: Janina Mera jmera@usaid.gov Mendez England & Associates Contact: David Hughes Address: 1020 19th Street NW #875 Washington DC 20036 Phone number: (240) 762-6296 Email: dhughes@engl.com 14 ME&A: Mendez England and Associates. A Bethesda company providing innovative solutions for economic development and social progress throughout the world. 79 Table of Contents Contents .................................................................................................................iv Executive Summary.................................................................................................i Evaluation Purpose......................................................................................................i Program Background ..................................................................................................i Evaluation Questions (EQs).......................................................................................ii Evaluation Methods ....................................................................................................ii Evaluation Limitations ...............................................................................................ii Main Findings, Conclusions, and Recommendations...............................................ii 1.0 EVALUATION PURPOSE AND QUESTIONS...........................................1 1.1. Evaluation Purpose ............................................................................................1 1.2. Audience and Intended Uses.............................................................................1 1.3 Evaluation Questions .........................................................................................2 2.0 PROGRAM BACKGROUND.........................................................................2 3.0 EVALUATION METHODS AND LIMITATIONS ......................................4 3.1 Methodology .......................................................................................................4 3.2 Data Analysis ......................................................................................................7 3.3 Evaluation Limitations .......................................................................................7 3.5 Respondent Characteristics and Demographic Overview .............................8 4.0 FINDINGS, CONCLUSIONS RECOMMENDATIONS............................10 4.1 EQ-1: EFFECTIVENESS: How strong and effective is the F2F Program Theory of Change (ToC)? .............................................................................. 10 4.1.1: How effectively do F2F Program activities (i.e., volunteer assignments) lead to desired impacts?.................................................................................................................................10 4.1.2: How effectively are F2F interventions (country projects, hosts, host development plans, volunteer SOWs, etc.) selected and developed to lead to change? ..............................13 4.1.3: How effectively are volunteers recruited, selected, matched to SOWs, motivated, and prepared for their assignments? ..............................................................................................17 4.1.4: How effectively and efficiently do volunteers complete SOWs and provide desired deliverables?........................................................................................................................................22 4.1.5: To what extent do hosts change behaviour and/or perceptions as a result of F2F interventions?.....................................................................................................................................27 4.1.6: To what extent are persons trained/directly assisted drivers of change in relevant host organizations?............................................................................................................................30 4.2 EQ-2: INDICATORS: How well do F2F standard indicators capture the effects of F2F activities? And do they provide useful information for F2F program management?.................................................................................. 32 4.2.1: How well do the indicators for F2F serve program management and the monitoring and evaluation process?....................................................................................................................33 4.2.2: Do some indicators present particular difficulties to hosts and IPs for collection and management interpretation?............................................................................................................37 4.2.3: Are there indicators that might be simplified or eliminated and are any others needed?................................................................................................................................................39 4.2.4: Is the F2F Program’s performance and impact on outreach/cultural exchange, capacity development, and environmental/natural resource management being captured effectively?...........................................................................................................................................41 4.3 EQ-3: LEARNING & INNOVATION: How well does the F2F Program learn and integrate innovation to encourage quality improvements and good practice in agricultural volunteer programs? ..................................... 42 80 4.3.1 How well do F2F IPs engage in a community of practice to improve the Program? ...42 4.3.2 In what ways could the Program improve learning and innovation to become more efficient and effective?.......................................................................................................................44 4.3.3 What learning and innovation activities or community of practice support are particularly effective or needed and what may be of lesser effectiveness or use?.................46 4.3.4 Are there unaddressed needs for support to core F2F Program quality and good practice? What are key issues for future success of the F2F Program?...................................48 4.3.5 How effectively have SPSP projects (including sub-awards) supported F2F learning, innovation, and community of practice; and how has that support been perceived by the F2F IPs?................................................................................................................................................50 4.3.6 What impacts have SPSP sub-award projects had on core F2F activities?.....................52 4.3.7 What factors might limit or facilitate the two SPSP projects’ effectiveness and impact? ..............................................................................................................................................................54 4.3.8 How effective has the Leaders with Associates (LWA) mechanism been in promoting F2F objectives? ...............................................................................................................55 Annexes..................................................................................................................57 Annex A: Expression of interest............................................................................. 58 Annex B: Evaluation protocol................................................................................. 76 Annex C: Web-based survey questions ............................................................... 129 Annex D: Key informant interview (KII) guide and questions .......................... 173 Annex E: Tally sheets by country......................................................................... 186 81 ACRONYMS ADS (Chapter 201) Automated Directive System/Program Cycle Operational Policy BFS Bureau for Food Security CA Cooperative Agreement CLA Collaborating, Learning, and Adapting COR Contracting Officer's Representative CP Concept Paper DQA Data Quality Analysis F2F Farmer-to-Farmer FY Fiscal year GAO Government Accountability Office HO Local Host Organization IP Implementing Partner KII Key Informant Interview LWA Leader with Associates LOE Level of effort M&E Monitoring and Evaluation MERL Monitoring, Evaluation, Research, and Learning NGO Non-governmental organization NORC NORC at the University of Chicago PDP Program Development Projects PEEL Program Evaluation for Effectiveness and Learning - USAID task order contract PERSUAP Pesticide Evaluation Report and Safer Use Action Plan RFA Request for Applications SFTP SSH File Transfer Protocol SI Standard Indicators SNG Small Niche Grants SOW Scope of Work SPSP Special Program Support Project STIP Science, Technology, Innovation, and Partnerships TOC Theory of Change USAID United States Agency for International Development. WinSCP Windows Secure File Transfer Protocol 82 1. Introduction This planning document, the “Evaluation Protocol,” was prepared for the F2F 2017 Theory of Change & Impact Learning Evaluation, as requested by U.S. Agency for International Development (USAID) Bureau of Food Security under USAID Task Order AID-OAA-I-15-00024/AID-OAA-TO-16-00008.15 Following USAID approval of the concept paper on December 28, 2016 that outlined the team’s approach for evaluating the performance of the Farmer-to-Farmer (F2F) Program. This evaluation protocol describes in more detail the plans for carrying out the performance evaluation,16 being conducted between November 2016 and June 2017. It was developed following a review of existing literature (prior evaluations, studies, and reports), meetings with Implementing Partners (IPs), consultations with USAID, and additional desk research. Team members involved in its design include: Herman Sanchez, Eric Johnson, Michelle F. Muldoon, and Ron Wendt, (technical team) contracted by the Feed the Future Global Program Evaluation for Effectiveness and Learning (PEEL). This final version of the Evaluation Protocol also includes and/or addresses comments and feedback provided by USAID staff Gary Alex, AOR, Erin Baize, Alternate AOR, and Janina Mera, Activity Manager. The document will serve as a reference, roadmap, and strategic work plan, recognizing that variables can often change quickly in development work. Therefore this evaluation protocol is also a flexible and adaptable plan that could be modified as needed, with agreement from USAID. 2. Project Information A. Project Description About the Evaluation The F2F 2017 Theory of Change & Impact Learning Evaluation will focus on three themes. a. The F2F Program Theory of Change (TOC) b. The F2F Program standard indicators17 c. The F2F Program learning and integration of innovation These three themes are closely interdependent and will be examined to assess how they affect the performance and impact of the overall F2F Program and to what extent they interrelate. This evaluation is not about the ability of individual IPs to carry out their respective F2F country projects. Rather, it looks at the effectiveness of the F2F Program itself: what makes it successful and what opportunities for improvement (OFIs) can be identified. It is hoped that this evaluation will be used to streamline and better design future F2F activities to ensure maximum impact and greater financial and nonfinancial returns on every American taxpayer dollar invested. The evaluation will also identify best practices, which could be incorporated into current and future F2F activities and 15 Annex 1: USAID Task Order AID-OAA-I-15-00024/AID-OAA-TO-16-00008. 16 USAID Evaluation type is required, impact, or performance evaluation. It is a Mandatory Reference found at the ADS Chapter 201. 17 F2F Indicators as December 2016 are in Annex 9. 83 shared among IPs and external partners, as appropriate. This evaluation will include: desk reviews; review and analysis of the F2F indicators and Theory of Change; a survey of volunteers, host agencies, implementing partners, and government officials; field visits to a number of F2F focus countries; Key Informant Interviews (KIIs); and discussions with IPs. The main source of new unbiased information will be a standardized online web survey applied to IPs, volunteers, host organizations, and government officials relevant to the country projects. This information will be analyzed in conjunction with the standard indicators and in the learning and innovation components of the evaluation. Whenever possible, correlations will be identified among indicator data and survey results. About the Program (USAID F2F) The USAID F2F program was first authorized by the U.S. Congress in 1985, and in 2015 formally celebrated its 30th Anniversary with a series of knowledge and learning events that honored volunteers, IPs, and program advocates including Congressional Representatives. The F2F Program is managed by the USAID Bureau for Food Security,18 and works alongside external implementing partners to facilitate exchange of experiences between U.S. experts (producers, agribusiness entrepreneurs, academics, university professors, etc.) and counterpart host country agricultural producers, entrepreneurs, and farmer-serving institutions in middle income and developing countries. Volunteers’ work helps to improve the quality of the agricultural sector workforce through transfer of knowledge, expertise, technology, training, and advisory services which are provided to farmers, private agribusinesses, cooperatives, services providers, and local government leaders to generate rapid, sustainable, and broad-based food security and economic growth in the agricultural sector. It continues to earn respect from stakeholders, including the U.S. Congress, for the high-quality technical services it provides to developing country host institutions, and is lauded for its cultural exchange aspect, which has fostered goodwill in many countries. The 2014 Farm Bill19 designated the F2F Program as the “John Ogonowski and Doug Bereuter F2F Program” in honor of pilot Ogonowski, killed September 11, 2001, and former Nebraskan Congressman Bereuter, who initially sponsored the F2F Program. A secondary goal is to increase the American public’s understanding of international development issues and programs and international understanding of the U.S. and U.S. development programs. In furthering the secondary goal of increasing public understanding and involvement in international development efforts and improved international understanding, F2F Programs design volunteer assignments and scopes of work to maximize opportunities for volunteer work and interactions with rural people in the host countries. In FY16, the eight core F2F programs were implemented under the Leader with Associate (LWA) Cooperative Agreements (CA) and one CA for Special Program Support Project (SPSP). Three new country projects were approved in 2016 for livestock (Ethiopia), legumes (Angola), 18 www.feedthefuture.gov 19 The United States Farm Bill Agricultural Act of 2014. https://www.usda.gov/wps/portal/usda/usdahome?navid=farmbill 84 and legumes (Ghana) bringing the total of approved country programs to 64. Annex 8 shows the current countries and project focus areas for each of the IPs. Since 1985, over 16,000 volunteer assignments have been completed in 103 countries. The total value of services provided by F2F volunteers is estimated at over $103 million. Approximately 1.2 million farmer families (representing about six million people) have been direct beneficiaries of the F2F Program. Volunteers help host individuals and organizations adopt new technologies and build local institutions and linkages to resolve local problems and have provided direct hands-on training to over 288,000 individuals. Approximately 24 percent of all volunteers and about 40 percent of all individuals trained by F2F volunteers have been women. In addition, the returned volunteers engage in public awareness activities in their communities to share their experiences and promote better understanding of international development issues and objectives. Table 1: Volunteer Assignment Breakdown FY2016 FY16 Volunteer Assignments Program Volunteer Assignments Completed Volunteer Days Average Days/ Assignment “Standard Assignment Equivalents” (= 21 days) Hosts Assisted Male Female Total ACDI/VOCA / ECCA 69 16 85 1484 17.5 71 249 ACDI/VOCA / West Africa 86 28 114 1949 17.1 93 193 CNFA/ Southern Africa 53 22 75 1338 17.8 64 118 CRS/East Africa 93 46 139 2723 19.6 130 134 Land O’Lakes/ MENA 51 25 76 1177 15.5 56 91 Partners/ Caribbean 89 37 126 2141 17.0 102 139 Winrock/AET 58 20 78 1469 18.8 70 82 Winrock/Asia 60 13 73 1596 21.9 76 183 VEGA/SPSP - FAVACA PDP 13 6 19 230 12.1 11 51 - Purdue University PDP 9 6 15 207 13.8 10 16 - VWB PDP 14 39 53 1890 35.7 90 65 - IESC PDP 1 1 2 25 12.5 1 0 - Small Grants Projects 20 10 30 415 13.8 20 50 616 269 885 16644 18.8 793 1371 The success of the F2F Program depends on two key steps: first, identifying and structuring good assignments, and second, identifying the right volunteers to respond to the needs of the hosts, customers, and clients. Strong technically qualified country staff and home office support staff are essential for successful assignments. Among USAID and other development actors, what makes the F2F Program unique is that it strives to be purely demand-driven, whereby local markets and stakeholders dictate the needs and US IPs find specialists that meet those needs. B. Theory of Change In a Theory of Change (TOC) development approach, expected behavioral changes result from a set of interlinked factors and, quite often, nonlinear actions. While there are various forms and approaches to a Theory of Change, the core elements of an effective TOC include a focus on continuous and critical learning and reflection, and lay bare and challenge the assumptions held by development actors, which may or may not be accurate. To understand the F2F TOC, it is necessary to appreciate that while the 85 basic hypothesis guiding F2F is relatively unchanged since 1985, USAID has in recent years begun to critically deconstruct and refine its approach to development impact, and is building critical and continuous learning into its work. The resulting TOC was co￾designed with IPs through discussion and technical deliberations, and has since been institutionalized across all F2F projects, and is a common unifying vision for the overall Program. While F2F appears to have an effective Theory of Change, the underlying components, interventions, and support processes require further in-depth study. This is especially important given the complexity of the Program and anecdotal evidence on the value of the intangible inputs and impacts, such as cross-cultural exchange and learning feedback loops. Because impacts are often nonlinear and highly dependent on a set context, having a well-thought-through TOC is important in F2F Program, where so many external and sometimes rapidly changing variables challenge success. These variables can include: IPs’ vision/mission approaches; priorities defined by the host organizations; the socio-geo-political and economic prevalent conditions in focus countries; the timely availability of expert volunteers; aligning visions, missions, and needs with donor priorities; expected designs; and donor funding. C. F2F Program Standard Indicators “Performance indicators”20 are the measures USAID uses to detect progress towards the results included in a Results Framework. When it pairs results with appropriate and precise measures, USAID eliminates a good deal of the ambiguity that is inherent in many results statements. USAID has identified some performance indicators as the best choices for characterizing progress in each of the sectors and technical areas in which it works. This set of indicators is called the “standard indicators” (Annex 9). Depending on the F2F country project area, IPs may also develop and track custom indicators of interest to a particular USAID mission. 20 USAID ADS 201.3.5.7 Program Cycle Operational Policy, Chapter 201. F2F Basic Theory of Change Input: Volunteer technical support with some additional staff support lead to learning through mentoring, workshops, and hands-on demonstrations. Strengthen the human capital. Output: Volunteer recommendations and training leading to improved efficiency in agricultural processes. Agribusiness with opportunities to be successful. Outcome: Adoption of recommendation or host change in behavior, management practice, and/or technologies (innovation) leading to increase in profits and disposable income among the stakeholders. Impact: Productivity and income increases. Improved social, human, and environmental capital. 86 A good assessment practice of indicators requires a baseline study. The information available indicates that IPs are collecting host baseline data on all host organizations before or at the time of the initial volunteer assignment with the host. During FY16 (Program Year 3), the IPs (except for Small Grant projects) collected data on outcome and impact indicators for FY16 annual reports; during its first two years, a program is not required or expected to collect data. This fact will limit analysis of the indicators to one year. However, some one-year indicators could reflect potential for benefits from assistance to the host, while others could serve as baselines to measure change during the succeeding years of implementation. In cases in which no baseline is available, the analysis of the indicators will be done by estimating the relative potential impact with the percentage of volunteer recommendation adopted by the host organization. The adoption process for any innovation generally involves a gradual period of testing and phased adoption. There are challenges associated with measuring and attributing impacts from short-term volunteer assignments21 especially when the F2F volunteer programs are associated with development objectives such as environmental conservation, natural resource management social/gender equity, poverty reduction, expanding financial services, building civil society, reforming policies and legal frameworks, strengthening institutions, and improving a range of quality of life indicators. The analysis of the F2F Program indicators will therefore identify trends and possible issues. When assessing and comparing indicators, a standardized checklist is helpful for guiding the analysis process. Unfortunately, the large number of F2F indicators limits that opportunity. However, where possible the evaluation team will try to identify correlations among two or more indicators. The evaluators, through the KIIs, will seek to validate whether IPs have a standardized, verification processes and whether there is follow-up for data accuracy, completeness, and timeliness. The evaluation of these indicators will not be used as a performance measure of the IPs or the host, but of the F2F Program. The evaluation team will not conduct a data quality assessment (DQA22). In each country to be visited, a full day will be dedicated to randomly verifying data. D. Program Learning and Integration of Innovations The stakeholders are expected to make a better use of monitoring, evaluation, and research leading to a team’s ability to rapidly learn and incorporate findings into program design. This creates sustainable improvements in the F2F Program, creating lasting positive impacts on programs’ target audience. Furthermore, through these innovations: activities, programs, and strategies are informed by development theory and shaped by the local context; stakeholder engagement is guided by culturally-sensitive and ethical practices; resources are appropriately matched with program and strategic goals; valid, reliable, and objective results are found; and decision-making is evidence￾driven and reflective of a broad range of stakeholders’ voices. 21 Managing international volunteer programs (2016) A Farmer-to-Farmer F2F Program Manual of Good Practices. 22 ADS 203.3.5.2 Data being reported is measured against five data quality standards: validity, integrity, precision, reliability, and timeliness. 87 The program designed for USAID and its partners allows users to source, co-create, and co-design development solutions that innovate on traditional approaches to monitoring, evaluation, research, and learning. It allows USAID to work with partners to collaboratively identify, design, and test cutting-edge solutions to more effectively understand and measure the impacts of development programs. This approach is useful when specific outputs and outcomes are not as easily identifiable up front, and where change might happen in a non-linear manner. The F2F evaluation will adopt the USAID concept of Monitoring, Evaluation, Research, and Learning (MERL), to analyze and make recommendations to projects working under complex country contexts. The evaluation team will address the challenges of how to: • • • Foster increased development impact through improved and innovative tools for MERL. Determine what cutting edge approaches are being used to measure impact, understand complex development challenges and solutions, and use evidence to drive smart decision-making and policy. Test and provide evidence to demonstrate the added value of this new Science, Technology, Innovation, and Partnerships (STIP) model of development, given that the impacts of many such programs are only discernible over the long term. 3. Evaluation Design A. Evaluation Purpose This evaluation of the F2F Program has two purposes. First and foremost, it will provide guidance for planning future F2F Program activities. F2F Program funding cycles, or phases, are typically five years in length. At the time of this report, the Program is in Year 4 of its FY2014-2018 phase. This evaluation will review the goals and implementation of the F2F Program and ensure that the Program operations are efficient and effective. Documentation on the recommendations will be provided to USAID/F2F for implementation of future F2F Programs. The second purpose is to identify best practices and areas in which the Program can make mid-course corrections. Evaluation recommendations will be designed to be shared with current IPs and adjustments in implementation made to the extent approved by USAID. This will enable implementing agencies to take stock of implementation approaches, strategies, organization, and results and make feasible changes that are indicated to improve performance. Over the years, the F2F Program has been evaluated internally (by USAID and partners) and by third party evaluators. These program evaluations have been conducted regularly (typically every five years) either with a mid-term or with end-of-program evaluation. These program evaluations have consistently found that the F2F Program provides high quality technical assistance services from volunteers. This 2017 evaluation builds on these past evaluations. Furthermore, this evaluation is being carried out following an 88 extensive 2015 GAO Assessment of the F2F Program.23 This GAO review thoroughly examined: 1) How USAID administers the Program; 2) How partners implement volunteer assignments and screen volunteers; and 3) The extent to which USAID uses monitoring and evaluation to manage the Program. This 2017 evaluation is not to duplicate the GAO review, but to address specific questions raised by the F2F USAID Program in the Task Order and through subsequent ongoing discussions. Past evaluations have noted the heavy management burdens inherent in reporting on F2F Standard Performance and Impact Indicators. The 2012 evaluation recommended simplifying the reporting indicator set. The current indicator set eliminated a number of those indicators, but, as a result of consultations with IPs, some new indicators have been developed. The 2005 F2F Program Manual24 was revised in FY15 and FY16 to reflect updates in current implementation practices. This evaluation will assess implementation of F2F in light of changes to the Program and the operating environment. Past evaluations have not addressed the F2F Special Program Support Project (SPSP) as this is a relatively recent development. The F2F SPSP was designed to test new approaches to the use of volunteers and innovations, supporting knowledge management in implementing agricultural volunteer programs, and facilitating special studies and activities on behalf of the F2F Program.25 . The evaluation will reassess and identify good practices to further streamline the standard indicators for the F2F Program and identify different indicators for organizational capacity and environmental impacts that capture F2F results more effectively. Suggested improvements in the indicators used in F2F have been included in the survey being administered to IPs, hosts, and volunteers. In the last two program cycles there have been some changes that affect the design, priorities and focus of this evaluation. For example, programming changes include: 1) Shifting focus from country focus to sector focus. (Look for USAID statement in a final report); 2) The addition of Farmer SPSP Program Development Projects (PDPs) and Small Niche Grants (SNGs); and 3) The inclusion of IPs who are new to F2F. The evaluation is expected to provide input related to these purposes in a cost-efficient manner. The evaluation team will assess the IPs’ progress toward meeting their 23 Annex 4: Volunteers Provide Technical Assistance, but Actions Needed to Improve Screening and Monitoring GAO-15-478: Published and released: 4/30/2015. http://www.gao.gov/products/GAO-15-478 24 Managing International Volunteer Programs (2016): A Farmer-to-Farmer Program Manual of Good Practices. http://farmer-to-farmer.org/tags/best-practices 25 Managing international volunteer programs (2016); A Farmer-to-Farmer Program Manual of Good Practices. 89 objectives and make recommendations on the future design of the Program. Implementation issues identified during the 2012 evaluation are thought to have been addressed and are not expected to require extensive assessment by the team. This, however, will be confirmed by the evaluation team. B. Overall Approach to this Evaluation • All the evaluation processes are designed to respond the key questions proposed by USAID (see section 3 c.). • The evaluation will follow eight distinctive phases. 1) Pre-evaluation phase. Synchronize concepts among the evaluators and the USAID/F2F Officers, and gain an in-depth knowledge of the ultimate purposes of this evaluation. 2) Literature review. Become acquainted with the F2F Program through understanding the documentation available from this Program and retrieval of indicators and other relevant data for analysis. 3) Conduct an information need assessment to define the most useful information for USAID and stakeholders. 4) Design a concept paper and evaluation protocol. In these documents the evaluation team will: identify data collection tools; prepare a survey tool; identify countries for data collection; and outline the overall design of the evaluation. 5) Design, program, translate, and administer an online survey to answer the key questions raised by USAID. Questions, analysis, and representation of results will be presented in tables and graphs. KII questions will be developed based on the findings of the survey. 6) Conduct formative information and exploratory research gathering. Meetings with USAID/F2F officers and IPs before, during, and after the annual meeting held on December 6, 2016 using a standardized format26 with a fixed number of questions. 7) Primary in-country data validation with IPs, current/active volunteers, host entities, and national government agents whenever possible. 8) Data analysis and reporting. Team discussions, end of data analysis, preparation of final draft report, and presentation of the final report to USAID. The evaluation team will use a holistic approach to the evaluation. Data and opinions from all the stakeholders will be considered and all facts and available figures will be analyzed, including reports, special reports, and new data from meetings, KIIs, and surveys. This will include information provided by USAID (donor), the IPs represented both in the USA as well as in the field, current and former volunteers, host organizations, and, whenever possible, host government officials. The team will also work closely with IPs to identify what could and should be tracked in the evaluation. The field activity expected for the evaluation team will be coordinated through the IPs countries’ offices and with IPs HQs in the U.S. As much as possible, the access to host 26 Annex 5: Guidelines - Exploratory Interviews with F2F Stakeholders USAID Farmer-to-Farmer Evaluation 2016-2017. 90 organizations will be channeled through the IPs; the access to government officials will be channeled through the host organizations. The evaluation team will keep the IPs and host organizations informed. The second phase, the literature reviews assessed an estimated 4,500 pages of information provided by USAID, as well as supplementary desk research. The purpose of the literature review was to gain a better understanding of the F2F Program; identify what is known about F2F as established in reports and special publications; provide opportunities to identify additional questions not posed by USAID that should be included in the survey and potentially the KIIs; and identify how the F2F IPs responded to the prevailing indicators. The third phase of the evaluation was the development and approval of the Concept Paper (CP) describing briefly what the evaluation is, why it is important, and how it will be carried out. The CP was used to help the evaluators develop this evaluation protocol which outlines the strategy for this evaluation. The fourth phase of the evaluation will begin with the development, programming, translation, and implementation of a universal online survey with a number of questions designed to capture the perceptions of volunteers (approximately 53 questions), officers at the IPs (64 questions), host organizations (31 questions), and host government agencies (14 questions). The survey, which will be conducted electronically in English, French, and Spanish, will be implemented in all regions and all countries where F2F has projects. The survey is expected to last between 15-30 minutes, depending on the stakeholder group of the respondent. Individuals who do not complete the survey will receive two follow-up emails as further encouragement to participate. The survey questions are focused on the issues raised by USAID, and on issues raised during the exploratory meetings. The survey is programmed by NORC.27 It largely uses yes/no responses so that the analysis of the survey will use a set of pre-defined arithmetic and statistical analyses such as frequency tables and graphs which will be defined and ready before the survey is implemented. The sixth phase is formative information and exploratory research gathering. Meetings with IPs occurred before, during, and after the formal kick-off meeting, were held during the IP annual meeting on December 6, 2016. The team conduct an introductory interview with each IP to explain the evaluation. Additional follow up discussions will take place to clarify themes or questions that arise during the entire evaluation process; and will be a means to get clarification on information gathered in the survey and KIIs. The seventh phase is primary in-country data validation through KIIs with IPs, current and former volunteers, host country organizations, and, where applicable, relevant government officials. The survey will be used as a guidance tool for the development of a standardized Key Informant Interview guide (KII guide). The KII guide will be used by the evaluation team to verify data collected in the survey. The KII questions and the guide will be reviewed and approved by USAID. During this phase, extensive informal, 27 Annex 4: NORC at the University of Chicago is an independent research institution that delivers reliable data and rigorous analysis to guide critical programmatic, business, and policy decisions. http://www.norc.org 91 ad hoc discussions will be held to better interpret results. A total of eight countries will be visited by the evaluation team. These countries have been selected to represent an even cross section of geographical regions (Caribbean, Central and South America, Africa and Asia, and the Middle East) and to ensure that all IPs have at least one of their programs evaluated in-depth. The eighth phase is data analysis and reporting. It will include team discussions, data analysis, preparation of the final draft report, and presentation of the final report to USAID. The following descriptive statistics will dominate the mathematical techniques to analyze the indicators’ data and the data from the web-survey: • Numerical count or frequencies – tables and graphics. • Percentages – tables. • Measures of central tendency – mean, mode, median. Whenever possible, the data analysis will include estimations for ranges (max/min) and error. Additionally, correlations analysis will be attempted between the indicators’ data and the results from the web-survey results. Otherwise, a separate analysis will be made with interdependent conclusions. Once the evaluation is completed and with the approval of USAID, the relevant data will be made publically available through the USAID Development Data Library. As mentioned above, this evaluation is not the typical evaluation in that it focuses less on project and program performance, and more on program design, effectiveness, logic, and unique value. The goal is to avoid duplication of previous evaluations and studies, the most recent of which was conducted by GAO in 2015. However, the evaluation will also add value to the landscape of what we know about agriculture, market systems, and food security development, with results that could benefit not only USAID F2F, but other USAID programs as well. With one-third of its focus on the F2F Theory of Change, this evaluation goes beyond a conventional evaluation by studying not only F2F performance indicators, but also the context, internal and external systems, and processes that are necessary to the success of the Program. 92 Figure 1: F2F Evaluation Work Flow & Research Approach 93 C. Evaluation Research Questions (“F2F Three Themes for Evaluation 2017”) There are three major research questions provided by USAID for this evaluation, which we refer to as USAID F2F Three Themes for Evaluation 2017. These key questions serve as overarching themes or pillars, for the evaluation. Under each question is a set of illustrative subset of questions that were developed at the start of the evaluation process. The subset of questions will be further refined as necessary as the team begins to uncover priority issues, bottlenecks, gaps, insights, and ideas. To respond to these overarching questions, the team will confirm key topics that warrant deeper analysis and which may not be immediately or adequately found in secondary research and resources (e.g. other evaluations, annual reports, IP reports, and meetings). 1. How strong and effective is the F2F Program Theory of Change? • • • • • • How effectively do F2F Program activities (i.e., volunteer assignments) lead to desired impacts?28? How effectively are F2F interventions (country projects, hosts, host development plans, volunteer SOWs, etc.) selected and developed to lead to change? How effectively are volunteers recruited, selected, matched to SOWs, motivated, and prepared for their assignments? How effectively and efficiently do volunteers complete SOWs and provide desired deliverables? To what extent do hosts change behavior and/or perceptions as a result of F2F interventions? To what extent are persons trained or directly assisted as drivers of change in relevant host organizations? 2. How well do F2F standard indicators capture the effects of F2F activities, with particular emphasis on indicators for outreach/cultural exchange, capacity development, and environmental/natural resource management? And do they provide useful information for F2F program management? • • • • How well do the indicators for F2F serve program management and the monitoring and evaluation process? Do some indicators present particular difficulties to hosts and IPs for collection and management interpretation? Are there indicators that might be simplified or eliminated and are any others needed? Is the F2F Program’s performance and impact on outreach/cultural exchange, capacity development, and environmental/natural resource management being captured effectively? 3. How well does the F2F Program learn and integrate innovation to encourage quality improvements and good practice in agricultural volunteer programs? 28 The “desired impacts” are defined as the positive outputs, outcomes, and impacts of the Farmer-to-Farmer Program as detailed in the Program’s Theory of Change. 94 • • • • • • • • How well do F2F IPs engage in a community of practice to improve the Program? In what ways could the Program improve learning and innovation to become more efficient and effective? What learning and innovation activities or community of practice support are particularly effective or needed and what may be of lesser effectiveness or use? Are there unaddressed needs for support to core F2F Program quality and good practice? What are key issues for future success of the F2F Program? How effectively have SPSP projects (including sub-awards) supported F2F learning, innovation, and community of practice; and how has that support been perceived by the F2F IPs? What impacts have SPSP sub-award projects had on core F2F activities? What factors might limit or facilitate the two SPSP projects’ effectiveness and impact? How effective has the Leaders with Associates (LWA) mechanism been in promoting F2F objectives? D. Methodology for Quantitative and Qualitative Data Collection The evaluation will collect primary data through a mixed-methods approach and will make use of secondary data through project documents and performance reports— made available by USAID—as well as other publicly available project documents. The evaluation team will begin their work through an initial literature review of existing documents to become familiar with the project and its stakeholders. The interaction of these various facets of the evaluation are represented in Figure 1. The evaluation team will use the insights from its exploratory interviews and literature review to create a web survey, which will be implemented among project stakeholders. The purpose of the survey is to acquire new unbiased information from the universe of F2F stakeholders and IPs. The output of the survey will provide evidence on the strengths and weaknesses perceived by the IP and users of the F2F program. The full population made of IPs, host organizations, and volunteers engaged in the F2F Program during the current Program cycle will be invited to participate in this web survey.29 The targeted groups and expected sample sizes for each stakeholder group are detailed in Table 1. 29 The survey will be translated into French and Spanish. It is recognized that some hosts will not be able to participate due to their lack of knowledge of English, French, or Spanish. The factual degree of non-response due to language cannot currently be estimated as the language capabilities of all host organizations are not known. 95 Table 1: Web Survey Sampling Stakeholder group Sample invited to participate in web survey Sample size (approximate) Implementing Partners All IPs engaged in the current Program cycle. IPs have been invited to submit the contact details for their staff engaged in the F2F Program including recruiters, program directors, program staff, country managers, and country staff engaged in the Program. 90 individuals Host organizations All host organizations engaged in the current Program cycle who are able to read English, French, or Spanish and whose email address is shared by IPs. 867 individuals F2F volunteers All current and past volunteers30 engaged in the current Program cycle. 865 individuals All the IPs and host organizations that provide unique email address are invited to participate in the survey, regardless of the language skills. Early in the survey, participants will have the opportunity to self-define their ability to understand the questions and to define the ability to answer the questions. Due to project finances, the web survey will be restricted to three languages—English, French, and Spanish. IPs have been informed on the language limitation. While alternative options were analyzed extensively with the IPs, it was agreed with USAID to limit the translation to these languages. Due to technical limitations and in order to secure the integrity of the data, when more than one organization uses the same email address, only one invitation will be sent. IPs and host organizations whose email addresses are unknown, will be excluded from the sample. The language limitation will impact the accuracy of the results but will not introduce a bias, per se, since the exclusion follow a general rule applied to all, and it is not a decision made solely by the evaluation team. The evaluation team will visit eight F2F program countries managed by eight IPs in three geographical regions to validate the F2F program results obtained through the web survey. The validation will be conducted through KIIs. The countries and projects for these visits were selected after an extensive analysis made by the evaluation team in consultation with the IPs and the USAID F2F management team. The consultation process allowed for the inclusion of a diverse range of contexts and a focus on previously unevaluated projects, geographical spread of IPs, and types of implementing mechanisms while minimizing bias and costly logistical efforts that are unlikely to improve statistical precision. Other factors considered included availability, since some field visits will depend on logistical support from the IPs. However, the final country selection was undertaken by the evaluation team. In summary, the countries and projects selected for a visit will yield a range of data and learning, not just on performance by sector or country, but will also yield learning on development trends worth further exploration by F2F. For example, some country projects have very strong women-value chain component, others are piloting rural financial services, and still others focus on youth. In addition, some IPs are doing very 96 well working with local partners and other USG agencies in country – a synergy that the evaluation team believe should be supported. The evaluators will explore what makes these IPs particularly successful or innovative in their respective focus areas. In order to comply with the USAID Evaluation Policy,31 the Team Leader has developed the rules of engagement for data acquisition and the data analysis strategy. These steps will ensure that conclusions, judgments, and recommendations are impartial. The integrity of the process is further ensured by maintaining independence in decisions. The evaluation team’s KIIs will be targeted to understand the web survey’s findings, including the possible reasons for the trends found, and to better understand the stakeholders’ experiences as they relate to the web survey’s findings. As such, the KII￾specific questions can only be developed once the web survey has concluded to ensure that the KIIs are focused on the most salient points of the web survey’s results. The selection of several KIIs will be made from a pre-established list of host organizations provided by the IPs. If a host organization cannot be visited due to logistical limitations, an additional organization will be randomly selected. The agencies selected for KIIs can only be random since the survey results are anonymous. The selection of host organizations to be interviewed will use the same set of data as the survey, with the exception of countries where the majority of the host organizations do not have the necessary contact information. In these cases, the consultants will make additional efforts to increase the number of hosts with information provided by the IPs in country. The regions and countries for these visits are detailed in Table 2. Table 2: KII and Field Visit Sampling Region Country Implementing Partner Africa Senegal Winrock International, ADCI/VOCA Malawi CNFA Kenya Catholic Relief Services (CRS) Asia and Middle East Tajikistan ACDI/VOCA Lebanon Land O’Lakes Bangladesh Winrock International, Tennessee State University32 Caribbean, Central and South America Jamaica Special Programs: Florida Association for Volunteer Action in the Caribbean and the Americas (FAVACA) Guatemala Partners of the Americas/Caribbean E. Methodology for Quantitative and Qualitative Data Analysis The results of the evaluation’s web survey will be analyzed in Stata and Microsoft Excel, with the latter used for tables and data visualizations. Stata will be used to construct frequency tables and cross-tabulations. The analysis will be stratified by stakeholder group with outcomes presented for IPs, volunteers, and host organizations. Further 31 USAID’s Automated Directives System (ADS) Chapter 201, Program Cycle Operational Policy, released September 2016. 32 TSU did implement a VEGA Small Grant Project in Bangladesh and the hosts can be contacted, but this activity has ended. 97 stratification, such as results according to the volunteer’s years of service with F2F or the individual’s role in the IP, may also be possible, depending on participation rates for the web survey. The evaluation team will assess the impact of language based on the responses and their relationships to host characteristics, such as the number of staff or the ability of the organization to implement volunteer recommendations, as reported by the web survey. In doing so, the evaluation team will be able to investigate if hosts who respond from countries where one of the three survey languages are not commonly spoken are similar to responding hosts in other countries. This will allow the evaluation team to qualify the overall representativeness of the study’s respondents when interpreting and reporting the study’s results. The evaluation team will keep a record of the number of host organizations that do not have email addresses according to IPs. These details will be included in the final evaluation report and will be used in qualifying the results of the evaluation and the representativeness of its sample. The evaluation team can estimate the number of host organizations that will not be available by email as the process of collecting contact information from implementing partners is ended. The qualitative data collected through the web survey will undergo thematic coding and content analysis in Stata. This will be used for the limited number of open-ended questions in the web survey, specifically the questions related to new indicators that are recommended by IPs. The answers will be reviewed and then categorized based on common themes that appear. This will allow summary statistics even for these open-ended questions. However, since the process will be done manually, Stata will not be used for this. Qualitative data collected through the KIIs will undergo discourse analysis in Microsoft Word in order to identify key themes and common experiences of IPs and host organizations. The questions for the KIIs will be focused on understanding the context for the findings of the web survey, especially the experiences of the host organizations and IPs as they relate to these findings and their impressions as to why respondents answered in a given way. Lastly, the evaluation team will collect administrative data from IPs and host organizations during their field visits. These data will be compared to the project data reflected in annual project reports as a validation check on F2F’s impact and reporting mechanisms. Summary statistics on these validation checks will be prepared using Microsoft Excel and Stata. F. Outcome Measures IPs and local and regional actors play a central role in USAID’s efforts to be a learning organization. Although Collaborating, Learning and Adapting (CLA) is not required by the ADS, it is recommended that programs incorporate CLA processes and practices into their work. Knowledge and learning should be documented, disseminated, and used to help spread effective practices widely for improved development.33 The outcome measures for the evaluation are detailed in Table 3 below. This table includes the original evaluation questions, the outcome measures related to these, the data sources used, and the relevant variables. 33 ADS 201.3.5.22 – Planning for CLA. 98 Table 3: Outcome Measures Evaluation question Outcome measure Data source Variable(s) (if the data is taken from the web survey) Implementing Partner reports on: How strong and effective is the F2F Program Theory of Change? Volunteer effectiveness Web survey ● Recruitment ● Preparation ● Project matching for skills ● Project matching for interest ● Motivation ● Capacity of IP ● Capacity of host for learning ● Level of effort for training activities with hosts ● Level of effort for training non-hosts Volunteer outputs Web survey ● Percent of deliverables completed ● Constraints to completing all deliverables ● Impacts of work on host activities Host impacts Web survey ● Implementation of host recommendations ● Constraints to implementing recommendations ● Satisfaction with host trainings ● Satisfaction with volunteer recommendations ● Perceptions of impacts of host’s recommendations ● Volunteer’s level of effort for training activities with hosts ● Engagement in developing volunteer Statement of Work ● Engagement in developing goals for F2F participation ● Ability to oversee volunteer work Practices for volunteer management Web survey ● Ability to share/access negative volunteer reviews ● Ability to oversee volunteer work Any of the above KIIs N/A How well do F2F standard indicators capture the effects of F2F activities (particular emphasis Indicator usefulness Web survey ● F2F indicators which are difficult to collect34 ● Indicators to be eliminated ● Indicators to be added ● Value of current outreach indicators ● New outreach indicators ● Value of current capacity development indicators 34 This statement was shared by several IPs during the preliminary interviews. 99 Evaluation question Outcome measure Data source Variable(s) (if the data is taken from the web survey) Implementing Partner reports on: on indicators for outreach/cultural exchange, capacity development, environmental/ natural resource management) and provide useful information for program management? ● New capacity development indicators ● Value of environmental/natural resource management indicators ● New environmental/natural resource management indicators Data validation Web survey Host organization reports on: ● Validation practices for data used in F2F reporting Any of the above KIIs N/A Administrative data for validation Field visits Administrative data from hosts organizations and IPs Report data for validation Annual reports Project outcomes from annual reports How well does the F2F Program learn and integrate innovation to encourage quality improvements and good practice in agricultural volunteer programs? Impact of F2F innovations Web survey ● Impacts of SPSP activities on programs ● Sectors of most relevant innovations ● Methods for program learning ● Methods for sharing best practices ● Factors affecting implementing innovations ● Methods for sharing volunteer reviews ● Negative impacts of SPSP sub-award projects ● Satisfaction with LWA mechanism Any of the above KIIs N/A 100 G. Limitations of the Evaluation ● ● ● ● ● ● The F2F Program currently operates in about 30 countries under multiple IPs and project types. Pre-defining the number of KIIs to eight countries, before a sample size calculation is completed determines the number of participants involved in the KIIs given a particular confidence interval. This could affect the quality and precision of the results. Ideally, samples should be defined based on the size of the Program and level of precision required. The wide number of issues tackled and the uniqueness of each of the assignments could create issues when accounting for the F2F pre-established indicators. The intensity of the survey to be carried out could be limited by the amount and quality of the contact information available from IPs. A challenge is the presence of several IPs with different institutional missions and visions, implementing F2F projects in several countries, each with unique socio￾economic and agriculture conditions. Limiting the languages used in the web survey to English, French, and Spanish may result in less coverage of the survey, especially with regards to host organizations. o The evaluation team cannot estimate with strong confidence the number of non-responses that language limitations will cause, as the language capabilities of all host organizations are not currently known. o In countries where one of these three languages is not commonly spoken, the survey may only be completed by host organizations with higher capacity, to the degree in which such language skills are associated with overall organizational capacity. This may cause an overrepresentation of higher capacity organizations in the overall results, and, likewise, may skew the results away from the experiences of lower capacity organizations. The evaluation team cannot estimate the degree of such limitations at this time, as the language capabilities of all host organizations are not currently known. The use of a web survey will exclude respondents who do not have a valid email address from participation. The evaluation team does not expect this to have a significant impact on the response rates for implementing partners or volunteers. 4. Management Plan A. Team Members and Roles and Responsibilities The team members’ ultimate responsibilities are included in the attached Expression of Interest (Appendix 1). Besides being able to contribute specific technical insights and expertise, all the team members will undertake some/all of the duties detailed below: • • In line with evaluations, the assignment is anticipated to include conducting desk reviews, field visits to eight designated countries. Each team member will undertake the KIIs in the chosen region and report writing. This assignment will include meetings and consultations with the USAID F2F and mission staff, counterparts (including IPs), and other relevant actors from civil 101 society, international community, academia, and government in countries advised by USAID. • • • For KIIs, the team members will prepare and draft questions, devise a schedule, and carry out the interviews. They will include USAID Missions, counterparts, donors, and host government officials. The evaluation team will finalize the details of the design in the Evaluation Plan. A mixed-methods approach rooted in robust qualitative and quantitative data collection and analysis will be used. Due to the geographically dispersed nature of the intervention, a creative approach involving case studies can be used to expand the KIIs. The evaluation shall generate rigorous findings and relevant actionable results for all stakeholders. The evaluation team will submit draft and final reports and revise final reports based on USAID feedback. B. Logistics and Supplies Most of the specific support needed must come from the IPs, host organizations, and from the Program Evaluation for Effectiveness and Learning (PEEL) project. 1. Implementing Partners • • • The evaluation team needs the contact information of volunteers and host organizations in order to interact with the individuals and organizations who will have the unique opportunity to participate in the survey. When appropriate and available, in each country, the IPs, will provide a list with contact information of the government officials who could have voluntarily followed up the results from the F2F interventions. In each country visited, each IP shall assist with the arrangement of 45- minute meetings with targeted host organizations. The evaluation team will define the targeted hosts. The expectation is to visit three to four local host organizations and/or government officials per day. 2. Local Host Organizations • Whenever possible, and only when it is relevant, the host will arrange meetings with government officers. All key personnel are requested to participate actively in the KIIs. Off-the record conversation will be allowed (no information is recorded and the information is not to be disclosed). 3. PEEL The PEEL staff will: • • • Provide technical and logistic support. Make travel (air, ground transportation, visa, and hotel) arrangements as scheduled and modifications as needed. Assist to arrange meetings with host organizations. 102 • • • Review, provide comments, edit, and format deliverables before they are presented to USAID. Provide the resources necessary to successfully implement the evaluation. As needed, support the presence of interpreters. C. Data Collector/Enumerator Training The data used for this evaluation will be: taken from existing documents, collected through a web survey, and collected through KIIs. As the use of existing documents and a web survey do not include a data collector, no enumerator training will be held for these parts of the evaluation. Respondents for the web survey will be responsible for completing the web survey without any outside assistance. The evaluation team will be responsible for developing the data collection guide for the KIIs. These guides will include the full list of questions for the KIIs and a statement of verbal consent for the respondent. Through use of these guides, the evaluation team will be able to collect comparable data while minimizing errors that can occur through enumerator effects. The evaluation team will collaborate in the development of these guides and will hold a separate meeting to discuss the final version of these guides to ensure a common understanding of the questions and protocols. D. Quality Control Primary data will be generated through KIIs, and from a web-based survey. Data quality for the web survey will begin through the use of a participatory method in the development of the questionnaire. By consulting existing documents and USAID staff and through the exploratory meetings with partners, the evaluation team will develop a questionnaire that is comprehensive and clear to all stakeholders. This development will involve an iterative process between the Program’s stakeholders and targeted questions to IPs for their insights on key and potentially unclear questions before the questionnaire is finalized. Since the questionnaire will be implemented in a variety of countries and contexts, and will be self-administered, it is critical that the web survey be thoroughly tested for clarity and relevance before being finalized. This will be followed up with KIIs to verify the data collected in the survey. Once finalized, the questionnaire will be programmed in Liberty—a proprietary software developed by NORC. The programming for the web survey will include automated skips, which will ensure that the web survey is administered correctly every time and that respondents only see questions which are relevant to their particular stakeholder group. This programming has been pre-tested by NORC staff before the launch of the survey to ensure that all questions and skips have been correctly implemented. In addition, Liberty allows for multiple languages, which will allow the respondent to complete the survey in English, French, or Spanish. This will increase the respondents’ comprehension of the questionnaire and, thus, the quality of the answers they are able to provide. These translations will be done through a double translation and adjudication method to ensure their quality and the consistency of the survey across 103 languages. This translation method involves two translators working independently and then reconciling the differences in their work. This is the standard method for translation used by NORC to ensure high-quality translations of data collection tools. The survey translations will be reviewed by the evaluation team before the launch of the survey as a further check on their quality and consistency with the original text. This review will be conducted in Spanish and in French. Feedback from these reviews will be integrated into the web survey before launch. In order to create the highest response rate possible to the web survey, the evaluation team will conduct multiple rounds of targeted communications with respondents. These will include: 1) an introductory email to introduce the evaluation and its purpose; 2) an email to invite respondents to participate in the survey, which will include a link to the web survey; 3) a follow-up email sent to respondents who have not yet completed the web survey; and 4) a second follow-up email to those respondents who have not yet completed the survey. These emails will be sent in English, French, or Spanish— depending on the respondent—in order to ensure understanding of all project communications. From NORC’s past experience, it is expected that the response rate for the web survey for all stakeholders will be approximately 25 percent. For the first of these, data quality will be maintained through the use of a KII guide,35 which will be developed by the evaluation team and reviewed by USAID before implementation. The KII guide will provide the respondent with: details on the purpose of the discussion; any risks or benefits; and their right to refuse the interview, specific questions, or to end the interview at any time. The guide will also detail the list of questions to be asked to the respondent in order to ensure that comparable data is collected and that enumerator effects are minimized. The KII guide will be translated into French and Spanish by the evaluation team to ensure consistent administration of the KIIs across countries. The content of the letters will be shared with the IPs before they are issued. The core translations are contracted by PEEL; and translation reviews into Spanish and French will be undertaken. Even though the evaluators have no need to extensively test the KII questions before they are used, each evaluator will make few runs before traveling to make sure that the interview questions can be answered within the allowed time. The initial test can be done among the evaluation team, and/or can be done by asking NORC staff if they understand the questions. Questions can then be revised accordingly. E. Data Management & Security Data collected through the web survey will be sent to NORC’s secure servers which host Liberty—the software used in programming the web survey. Data from the web survey will only be shared through WinSCP, a Secure File Transfer Protocol (SFTP) interface. No data will be shared by email at any point. No one outside of the evaluation team will be given access to the raw data from the web survey, in order to ensure 35 Annex 6: Standardized Key Informant Interview Guide. 104 confidentiality of all responses. Data stored locally by the Research Analyst will be saved on an encrypted and password protected computer for analysis, in order to ensure the security of these data at all times. Data analysis will be performed locally, through Microsoft Excel and Stata, on the Research Analyst’s computer. Data collected through KIIs and exploratory interviews will be recorded in Microsoft Word and managed through the evaluation team’s Google Drive folder. The existing documents for review will be managed through the Dropbox folder established by PEEL and which has been shared with the evaluation team. F. Preparation of Datasets for Public Use The data resulting from the evaluation’s web survey will be prepared for public use, once approved by USAID, on the Development Data Library. The dataset will be shared with USAID in Stata format and will include metadata in the form of variable and value labels, which will clearly link the data with the question asked and any categories used in the answer choices. The evaluation team will also make available the questionnaire used for this data collection in Microsoft Excel or Word format. Further metadata related to the web survey will be shared as a Nesstar file or in USAID’s preferred format. This will include notes on data limitations, data gathering methodologies, and explanations of any necessary redactions. Before making the web survey data available, the evaluation team will redact some answers to ensure respondent confidentiality. While no single answer will provide personally identifiable information, a combination of responses could be used to identify the individual and/or the organization serving as the respondent. Therefore, some redactions will be necessary and will include a combination of the questions related to the individual's role in the organization, the funding level of the organization, the number of years that the organization has been engaged with F2F, and the volunteer history for current and past program volunteers. Qualitative data will not be prepared to be shared for public use. Likewise, data obtained from the evaluation team’s review of existing documents will not be prepared to be shared for public use. 5. Ethical Considerations A. Ethics Standards Research ethics govern the standards of conduct for scientific researchers. It is important to adhere to ethical principles in order to protect the dignity, rights and welfare of research participants.36 36 According to the World Health Organization (WHO). 105 As such, all researchers involved in human research must be trained and certified before the process begins. Two main considerations govern the need for certification: • • Are the human subjects exposed to manipulation, intervention, observation, or other interaction with investigators either directly or through alteration of their environment? or Are the human subjects individually identifiable through investigator's collection, preparation, or use of biological material or medical or other records? In the case of the evaluation of the F2F Program, the main source of concern is the survey. In the case of the F2F evaluation only the interactions and observations apply. The online survey, which is the main source of new information, will be conducted anonymously. The data extracted from the reports are not in any way linked to individual names. However, with regard to the KIIs and exploratory interviews names of individuals and contact details are needed for the purpose of making appointments. The links to names and contact details will be eliminated in the analysis and will not appear in the final report. B. Approvals Since the research takes place mostly in countries other than the United States, the HHS regulation (45 CFR 46.101(h)) applies. The participation in the interviews and in the anonymous online surveys, is voluntary; the online survey is preceded by a letter of invitation explaining the complete freedom individuals have to participate or not and to select the answers. No children will participate. To ensure that there are no ethical issues that could create potential impacts to the research subjects, all the members of the evaluation team must take the online training designed as Protecting Human Research Participants provided by the NIH Office of Extramural Research.37 All the members of the evaluation team must reach the certification before the online survey is implemented. The individual certifications will be placed in an annex of the final report. The evaluation protocol and web survey has been reviewed by NORC’s Institutional Review Board (IRB) for ethics clearance prior to the start of data collection. NORC’s IRB follows a formal process for examining all research projects to assure human subjects protection. NORC’s IRB is registered with the HHS Office of Human Research Protection and has a Federal-wide assurance (FWA 00000142). The IRB takes an active role in helping guide protocols to meet the highest standards for human subject protections. NORC’s IRB requires that research protocols provide sufficient detail to ensure that: the selection of subjects is equitable, subjects’ privacy is protected, and data confidentiality is maintained; informed consent is written in language that study participants can understand and is obtained without coercion or undue influence; and appropriate safeguards protect the rights and welfare of vulnerable subjects. No data 37 The training and certification is provided by the NIH Office of Extramural Research Protecting Human Research Participants located at http://phrp.nihtraining.com 106 collection will take place with the web survey until approval is given by NORC’s Institutional Review Board. This approval has been granted. 6. Data Collection Instruments Data from F2F Annual and Special Reports; Indicators Reports from Exploratory Meetings Web Survey Reports from KIIs 7. Annexes Annex 1: USAID Task Order AID-OAA-I-15-00024/AID-OAA-TO-16-00008 Annex 2: Evaluation Timeline Annex 3: Guidelines: Exploratory Interviews with F2F Stakeholders USAID Farmer-to￾Farmer Evaluation 2016-2017 Annex 4: Template for Exploratory Interviews Annex 5: US Department of State and USAID Joint Strategic Goal Framework Annex 6: Standardized Key Informant Interview Guide Annex 7: Announcement/Letter of Introduction from Implementing Partners Annex 8: F2F Partners, Countries, and Projects. Annex 9: Standard Indicators Annex 10: Feed the Future Goals 107 Annex 1: Task Order - AID-OAA-I-15-00024/AID-OAA-TO-16-00008 PEEL TASK ORDER EXPRESSION OF INTEREST – PERFORMANCE EVALUATION I. BACKGROUND INFORMATION A) Identifying Information1. Project/Activity Title: John Ogonowski and Doug Bereuter Farmer-to-Farmer Program 2. Award Number: OAA-L-13-00002, 3, 4, 5, 6, 7, 8, 9; OAA-A-13-00053 3. Award Dates: FY14-18 4. Project/Activity Funding: 8 LWAs, ~$8M; 1 SPSP Cooperative Agreement, ~$5.6M 5. Implementing Organization(s): LWAs: ACDI/VOCA (2), CNFA, CRS, Land O’Lakes, Partners of the Americas, Winrock (2); SPSP: VEGA 6. Project/Activity COR/AOR: Gary Alex B) Development Context 1. Problem or Opportunity Addressed by the Project/Activity being Evaluated The F2F Program goal is to generate rapid, sustainable and broad-based food security and economic growth in the agricultural sector. A secondary goal is to increase the American public’s understanding of international development issues and programs and international understanding of the U.S. and U.S. development programs. 2. Target Areas and Groups F2F Programs are global in nature, although most programs have core countries within a single region: ● OAA-L-13-00002 – East Africa: Uganda, Kenya, Tanzania, Ethiopia ● OAA-L-13-00003 – Southern Africa: Malawi, Mozambique, Angola ● OAA-L-13-00004 – West Africa: DRC, Senegal, Liberia, Ghana ● OAA-L-13-00005 – Caribbean: Dominican Republic, Nicaragua, Haiti, Guatemala ● OAA-L-13-00006 – Asia: Bangladesh, Burma, Nepal ● OAA-L-13-00007 – Europe, Caucasus, Central Asia: Armenia, Georgia, Tajikistan, Kyrgyzstan 108 ● OAA-L-13-00008 – Middle East, North Africa: Egypt, Lebanon ● OAA-L-13-00009 – Agricultural Education and Training: Senegal, Nigeria, Guinea ● OAA-L-13-00053 – Uganda, Ethiopia; Colombia; Jamaica; and others In furthering F2F’s secondary goal of increasing public understanding and involvement in international development efforts and improved international understanding, F2F Programs design volunteer assignments and scopes of work to maximize opportunities for volunteer work and interactions with rural people in the host countries. C) Intended Results of the Project/Activity being Evaluated See Attachment A: F2F Description and Theory of Change D) Approach and Implementation See Attachment A: F2F Description and Theory of Change II. EVALUATION RATIONALE A) Evaluation Purpose This Evaluation of the F2F Program has two purposes. First and foremost, it will provide guidance for planning future F2F Program activities. The last Farm Bill was delayed from 2013 to 2014. Current budget deficit talks make legislative processes unpredictable. But, it is expected that a 2018 Farm Bill will fund the F2F Program for the period FY2018-2022 (implementation period FY2019 – 2023). This Evaluation will review the goals and implementation of the Program and ensure that the program operations are efficient and effective. Information from the evaluation will be used in shaping future F2F programs. Past evaluations and subsequent analytical and design work have provided a basis for adjustments to the F2F Special Program Support Project (SPSP) and to simplify F2F standard indicators. The 2005 F2F Program Manual was revised in FY15 and FY16 to reflect updates in current implementation practices. This Evaluation will assess implementation of F2F in light of changes to the program and the operating environment. The second purpose is to identify best practices and areas in which the program can make mid-course corrections. Evaluation recommendations will be shared with current IPs and adjustments in implementation made to the extent possible. Documentation on the recommendations will be provided to applicants for implementation of future F2F programs. This will enable implementing agencies to take stock of implementation approaches, strategies, organization, and results and make any changes that are indicated to improve performance. The Evaluation is being carried out following an extensive GAO Assessment of the F2F Program as completed in 2015 (see: http://www.gao.gov/products/GAO-15-478). This GAO review thoroughly examined (1) how USAID administers the program, (2) how partners implement volunteer assignments and screen volunteers, and (3) the extent to which USAID 109 uses monitoring and evaluation to manage the program. The Evaluation is not expected to duplicate the GAO review of the entire Program implementation, but to address selected issues in greater depth. The F2F Program has had positive reviews in the past and has a strong positive reputation. However, the operating environment for the program is continuously evolving, with a changing pool of potential volunteers, different needs and priorities for hosts, and challenges in promoting broad change based on time-limited volunteer assignments. There may be value in reassessing and identifying good practice to strengthen the rationale underlying the Program theory of change. Past evaluations have noted the heavy management burdens inherent in reporting on F2F Standard Performance and Impact Indicators. The 2012 evaluation recommended simplifying the reporting indicator set. The current indicator set has eliminated a few indicators, but as a result of consultations with implementing partners, some new indicators have been included. Indicators for impacts on institutional capacity and environmental objectives are particularly challenging. There may be value in reassessing and identifying good practice to further streamline the standard indicators for the Program and identifying different indicators for organizational capacity and environmental impacts that capture Program results more effectively. The SPSP is now in a second phase of implementation. The project is intended to strengthen the overall F2F Program by involving new implementing partners, testing new approaches to use of volunteers, supporting knowledge management in implementing agricultural volunteer programs, and facilitating special studies and activities on behalf of the F2F Program. There may be value in assessing and identifying good practice to improve support to the F2F Program for relevant knowledge management, quality improvement, and outreach. The Evaluation is expected to provide input related to the abovementioned purposes in a cost-efficient manner. The team will assess the IPs’ progress toward meeting their objectives and make recommendations on the future design of the program. The program, in general, appears to be operating well, as reflected by annual and semi-annual reports and site visits. Implementation issues identified during the 2012 Evaluation have been addressed and are not expected to require extensive assessment by the team. This, however, is to be confirmed by the Evaluation team’s own assessment. While this SOW may focus the Evaluation team on specific issues, the team is encouraged to provide comments and recommendations on any issues identified during the course of the Evaluation. B) Audience and Intended Uses The USAID F2F team (BFS/CSI/Technical Division) and the F2F implementing partners will use the evaluation to identify potential for changes to program strategies, implementation arrangements and approaches as appropriate to achieve F2F objectives. USAID also will use the evaluation to inform design of future F2F programs. C) Evaluation Questions 110 The Evaluation Team will evaluate progress with the implementation of the current F2F Cooperative Agreements against the stated objectives of the F2F Program. The Evaluation team will need to be creative in using existing data and reports to assess expected effectiveness and efficiency of the program. In this the Evaluation Team will need to draw on documented experience from earlier F2F programs, especially the FY09-13 F2F Program, and from preliminary results achieved to date. Using that information, the report will assess F2F Program experience and strategies and provide empirical evidence to support conclusions and recommendations, with focus on the following questions. 1. How strong and effective is the F2F Program Theory of Change? How effectively do Program activities (i.e., volunteer assignments) lead to desired impacts? How effectively are F2F interventions (country projects, hosts, host development plans, volunteer SOWs) selected and developed to lead to change? How effectively are volunteers recruited, selected, matched to SOWs, motivated, and prepared for their assignments? How well do volunteers complete SOWs and provide desired deliverables? To what extent do hosts change behavior and/or perceptions as a result of F2F interventions? To what extent are persons trained/directly assisted drivers of change in relevant host organizations? 2.How well do F2F standard indicators capture the effects of F2F activities (particular emphasis on indicators for outreach/cultural exchange, capacity development, environmental/natural resource management) and provide useful information for Program management? How well do the indicators for F2F serve purposes for program management, monitoring and evaluation? Are there indicators that present particular difficulties for collection and management interpretation? Are there indicators that might be simplified or eliminated or are any others needed? Are the Program’s performance and impact on outreach/cultural exchange, capacity development, environmental/natural resource management being captured effectively? 3.How well does the F2F Program learn and integrate innovation to encourage quality improvements and good practice in agricultural volunteer programs? How well do F2F IPs engage in a community of practice to improve the program? In what ways could the program improve learning innovation to become more efficient and effective? What learning and innovation activities or community of practice support are particularly effective or needed and what may be of lesser effectiveness or utility? Are there unaddressed needs for support to core F2F Program quality and good practice? What are key issues for future success of the F2F program? How effectively have SPSP projects (including sub-awards) supported F2F learning, innovation and community of practice and how is that support perceived by the F2F IPs? What impacts have SPSP sub-award projects had on the F2F program and its activities? What factors might limit or facilitate the two SPSP projects’ effectiveness and impact? How effective has the LWA mechanism been in promoting F2F objectives? 111 For each of the three above questions, the evaluation team should attempt to identify ways in which current processes can be improved. The team should discuss good practices for agricultural volunteer program planning and implementation, discuss any limitations of current practices, and suggest methods to address any limitations. The evaluation team is not required, but is encouraged, to comment on any other aspects of the F2F Program planning and implementation that they encounter that may be relevant to improving the Program’s efficiency, effectiveness, and benefit sustainability. III. TIMEFRAME & TRAVEL A) Timeframe The Evaluation should start on/about October 2016 and shall be completed no later than June 30, 2017. Work of the Evaluation Team members does not have to be continuous and may be phased as appropriate to the most efficient, effective and convenient completion of the required Evaluation. B) Travel TBD – The team shall visit selected country programs to obtain insights to program operating procedures, reporting and activities, for a total of 3 weeks. Selection should attempt a balance among different IPs. Since country conditions must be considered and USG country clearance will be required, the team shall select proposed sites to be visited and the composition of the teams for each country and obtain USAID approval for the required travel. It is expected that the Evaluation Team will split up to cover as many countries as possible within the limited timeframe and limited resources. IV. DELIVERABLES & DESIGN A) Deliverables All deliverables listed in section F. 7 of the PEEL-Task Order contract. B) OPTIONAL: Evaluation Design The Evaluation will have to be carried out within a limited resource envelope. The Program operates in about 30 countries under multiple IPs and project types. The Evaluation team will be able to visit a few selected F2F program countries and will have to draw from multiple sources for data and insights in carrying out the Evaluation. The evaluation team will finalize and elaborate the evaluation approach in the evaluation plan. However, a mixed methods or process evaluation approach to collecting and analyzing qualitative and quantitative data is recommended to arrive at robust findings and conclusions and fully address all the evaluation questions. The evaluation should review the experience and impacts of the FY2009-13 F2F Program (core programs and SPSP) as reflected in final reports, as well as the experience to-date in the current FY14-18 program. The programs are largely similar, such that the FY09-13 Program 112 may suggest results that can be expected from the current Program. Assessment of potential impact from any changes between the programs, if any, would be of interest. V. TEAM COMPOSITION USAID envisions a team of four Key Persons including an Evaluation Specialist/Team Lead, a Research Analyst, an Agricultural Development Specialist, and a Technical Agriculture Specialist. Taken together, the team must have familiarity/experience in six areas: a) agricultural development, technology transfer, organizational and capacity development; b) gender analysis; c) quantitative and qualitative evaluation, including formulating questionnaires or surveys, and knowledge of cultural exchange measures; d) USAID-agricultural projects; e) the US agricultural sector; and f) agricultural volunteer program management, implementation or service work. All team members must also have the ability to analyze issues and formulate concrete recommendations orally and in writing. A mix of resources is proposed below but the team composition and individual skill/experience mix may vary, as determined by the Team Leader and in collaboration with USAID. Evaluation Team Leader – 65 work days A senior-level evaluator with a minimum of 10 years of experience designing, managing and/or evaluating multifaceted international development teams, involving farmers, private sector, and public sector involvement. S/he will provide leadership in the conceptual design of the project and must have strong project management skills. The Team Leader must have experience with agricultural development and capacity building activities and experience with monitoring and evaluation and project/activity design experience. S/he should understand how to use the private sector as the engine of economic growth, have experience in agribusiness or agricultural economics, understand the dynamics of institution building, human capacity building, the value chain, and/or have experience working with volunteers and consultants, monitoring and evaluation experience, project/activity design experience, and an appreciation of the need for and complexities of applied agricultural research. The candidate will also have: a) a demonstrated capacity to conduct independent program evaluation; b) project/activity design experience; and c) an understanding of USAID’s foreign assistance goals, and its particular objectives related to agricultural development and food security. Research Analyst – 50 work days Must be an experienced analyst (5 years or more) with research and/or socio-economic analysis in international development related to agriculture. S/he will take the lead in data collection and analysis as well as conducting surveys. This team member will have an excellent understanding of the tools and research methodologies for monitoring and evaluating agricultural development programs. S/he should have experience assessing the impacts of technology transfer and capacity development on various types of beneficiary organizations, as well as in assessing the adequacy of monitoring and evaluation indicators in reflecting the accomplishments and impacts of development and capacity building activities. The qualified candidate must also demonstrate experience recommending practical and pragmatic development indicators. S/he must have some familiarity with the US agricultural community, NGO programs, and volunteer programs. 113 Agricultural Development Specialist – 50 work days Must be experienced expert (5 years or more) in international development related to agriculture. Team member must have experience developing and implementing technical approaches to agricultural development and capacity building activities, particularly related to technology transfer, capacity development, and value chain development. S/he should understand production systems, producer and community organizations, the dynamics of institution building, human capacity building, and agricultural research/extension/education institutions. Team member will also have demonstrated experience in effectively conducting outreach and dissemination to policy makers, development practitioners and/or the private sector. S/he should have experience with production technology and production projects oriented toward market demands. Experience with agribusiness, agricultural value chains, and working with volunteers and consultants is desirable. Technical Agriculture Specialist – 40 work days Must be experienced agricultural professional (5 years or more) with knowledge and contacts in the US agricultural sector. Experience with international agriculture is desirable. S/he should have experience in agribusiness, value chains, and farm production. S/he should understand institutional and human capacity needs for an effective agricultural sector. S/he should understand the science and technology behind current agricultural systems, and have knowledge of international agricultural standards and market entry requirements. The Level of Effort (LOE) and professional qualifications of the specialists may be inter￾changed. The LOE also may be allocated differently, with for example a fifth Specialist included with the same total LOE (205 work days). Specialists should have university degrees in fields relevant to agriculture and rural development. No specific degree qualification is indicated as this requirement will be subsumed by the requirement for experience in international development. 114 Annex 2: Evaluation Timeline November 2016 December 2016 January 2017 February 2017 March 2017 April 2017 May 2017 June 2017 Kick-off meeting (KOM) 11/23 Document review 11/22 2/15 Evaluation questions 12/1 1/19 Interview guides 12/30 TOCOR approval of concept paper (5 pg.) 12/30 TOCOR approval of evaluation protocol (20 pg.) 3/6 Evaluation Phase Implementation of the survey 2/19 3/11 Visits to selected countries 3/19 4/15 Analysis & Reporting Phase Draft report 4/30 Receive USAID comments 5/12 Presentation of findings 5/20 Final report 5/29 115 Tentative timeline (for use only by the Evaluation Team for planning) Activity # Name of the activity Begin Date Ro n Eri c Mi ch ell e He rm an Da vid Observations 1 First evaluation protocol draft for edition Dec 20th Jan 15th X X X X X 2 Evaluation protocol submitted to USAID Feb 28th X 3 Survey Jan 2nd Jan 23rd X 4 Translation of the survey into 9 languages Jan 2nd Jan 23rd TBD 5 Exploratory meetings with IPs Dec 2nd Jan 30th X X X X 6 Invitation letter to participate in the survey; Corrections Jan 16th Jan 30th X 7 Implementation of the survey Feb 20th Mar 18th X 8 Initial analysis of the survey - Minimum: General summary Mar 8th X 9 Advance analysis of the survey Mar 14th X 10 Final analysis of the survey Mar 25th X 11 First travel week Mar 25th Mar 31st X X Each traveler is responsible for proposing the best travel route to Phelisha. Final determination done by David. 12 Second travel week Apr 1st Apr 6th X X 13 Consolidated data from Reports: Indicators w/outputs Mar 28th X 14 Third travel week Apr 9th Apr 15th X X 15 First week for final report Apr 17th Apr 22nd X X X 16 Second week for final report Apr 24th Apr 29th X X X 17 Third week for final report May 1st May 6th X X X 18 Final draft of the report for editing May 9th X X X X 116 19 Submission of the final draft for suggestions by the USAID May 11th X 20 Preparation of the PowerPoint Presentation (PPT) May 12th May 14th X X X 21 Editing and formatting of the PPT May 15th X 22 Submission of the PPT to USAID and to IPs May 17th X 23 Evaluation Team travel to Washington DC May 17th X X X 24 Evaluation team internal discussion & preparation for presentation May 18th X X X 25 PPT of the final report to IPs & USAID May 19th X X X X 26 Presentation final report with IPs and USAID suggestions May 23rd X X X NOTE: A red X indicates who is leading the process 117 Annex 3: Guidelines - Exploratory Interviews with F2F Stakeholders EXPLORATORY INTERVIEW GUIDELINES: Exploratory Introductory Interviews with F2F Stakeholders USAID Farmer-to-Farmer Evaluation 2016-2017 Purpose: The goal of these exploratory interviews is to: 1) introduce F2F Stakeholders to the evaluation team, and process; 2) review the USAID F2F Three Themes for Evaluation 2017, and to 3) get initial thoughts and guidance on F2F Program and the IPs role; priorities and focal points for the IP, to guide the team on its next steps for research, field visits, and planning. Audience: The Evaluation Team members will interview Implementing Partner [IP] HQ Staff, IP Country Staff, and USAID F2F staff. Date, Time, Duration: Interviews will be 30-45 minutes in length, depending on the familiarity the interviewer has with the project. Interviews will be conducted in the month of December 2016 and January 2017, ending by first week of February 2017. Methodology: The Evaluation Team member will be assigned IP point people to interview; each person traveling will focus first on the IPs for country projects he/she will visit. Interviews will be informal but uniform; each interviewer will use the same script and questions and the attached template for documenting the results of the interview (see next pages). Interviews will be conducted in-person to the extent possible, and if not possible, then via Skype or phone. The Evaluation Team will work in consultation with USAID and the IP to determine who to interview and other related guidance. INTERVIEW SCRIPT (Interview Duration: 30-45 minutes) Interviewer: “Thank you for agreeing to do this interview with me. My name is: ___. I am part of the F2F Evaluation Team, and my role is ___, and I have a background in ___.” Purpose of the Interview: “To optimize our time together, this interview serves multiple aims. The purpose of this interview today is to: 1. To introduce the evaluation team and parameters of the evaluation (including the “3 evaluation questions (and subset questions)” 2. To learn about your unique experience with and recommendations related to F2F 3. To request your support needed to have a successful implementation Program Goals – Our Assumptions: 118 “Before we get into my questions I just want to be clear on our assumptions and understanding of the program’s mission and role within USAID/USG: As you may know, the F2F Program is the result of legislation and is funded through the US Farm Bill 2014 Just so we’re all on same page, I will now explain briefly how F2F fits into the goals outlined in the US Farm Bill, to which F2F is accountable. HIERARCHY OF PROGRAM GOALS: US Farm Bill 2014 38 “further assist development countries, middle-income countries, emerging markets….. to increase farm productions and farmer incomes, the President may: establish and administer a program, to be known as the “John Ogonowski and Doug Bereuter Farmer-to-Farmer Program to assist in (1) establish and administer a program, to be known as the "John Ogonowski and Doug Bereuter Farmer-to-Farmer Program", of farmer-to-farmer assistance between the United States and such countries to assist in- (A) increasing food production and distribution; and (B) improving the effectiveness of the farming and marketing operations of agricultural producers in those countries; (2) use United States agricultural producers, agriculturalists, colleges and universities (including historically black colleges and universities, land grant colleges or universities, and foundations maintained by colleges or universities), private agribusinesses, private organizations (including grassroots organizations with an established and demonstrated capacity to carry out such a bilateral exchange program), private corporations, and nonprofit farm organizations to work in conjunction with agricultural producers and farm organizations in those countries, on a voluntary basis- (A) to improve agricultural and agribusiness operations and agricultural systems (B) to strengthen cooperatives and other agricultural groups in those countries; USAID Mission US Dept. of State and USAID Combined Mission:39 USAID (only) Mission: “We partner to end extreme poverty and promote resilient, democratic societies while advancing our security and prosperity.” 40 38 US Farm Bill 2014, Farmer to Farmer Legislation, Full text is here: http://uscode.house.gov/view.xhtml?path=&req=%28title%3A7+section%3A1737+edition%3Apreli m%29&f=&fq=&num=0&hl=false&edition=prelim 39 US Department of State and USAID Strategic Plan FY 2014-2017. Retrieved from: https://www.state.gov/s/d/rm/rls/dosstrat/2014/html/225784.htm#StartLongDescription 40 USAID Mission (USAID only). Retrieved from: https://www.usaid.gov/who-we-are/mission￾vision-values 119 USAID F2F Mission Three (3) Themes for Evaluation (‘the Big 3’): USAID Farmer-to-Farmer Below are the three major questions or themes (‘Big 3’) that frame the entire Evaluation. Each interviewee will receive these questions in advance of the interview, so they may begin to prepare and organize their thoughts, in view of a productive interview. 1. How strong and effective is the F2F Program Theory of Change? 2. How well do F2F standard indicators capture the effects of F2F activities (particular emphasis on indicators for outreach/cultural exchange, capacity development, environmental/natural resource management) and provide useful information for Program management? 3. How well does the F2F Program learn and integrate innovation to encourage quality improvements and good practice in agricultural volunteer programs? USAID F2F Three Themes for this Evaluation: “In the big picture view of this 2017 Evaluation, USAID has asked these three central questions, which will frame and guide the entire evaluation. Here are those 3 major questions….. [READ QUESTIONS FROM BLUE BOX BELOW]” Q&A – The Interview Questions: (Editor’s Note: These questions were initially developed for the purpose of informational meetings with F2F IPs, in preparation for initial USAID and IP meetings; and for exploratory research purposes. Due to time limitations [evaluation proceeding 120 at accelerated pace], questions serve multiple purposes and are therefore question types are in a mixed sequence for ease of conversation and flow.) 1. Tell us about your experience: your name, your role/relevance to F2F, number of years’ experience with the program 2. As a F2F ________ (type) Stakeholder, what would you call your biggest success? 3. What were the factors contributing to this great success? 4. What would be required programmatically (F2F) to achieve even greater success? 5. What are the strengths and weaknesses of the F2F Program? 6. How many F2F projects do you have? How are your F2F projects managed? What is your role and what do you do on a day-to-day basis? Do you manage other programs? If so, which? How many staff do you supervise? What do they do? 7. What is your project’s strategic competitive advantage or value-addition to the F2F Program overall? What is your “elevator pitch” and what makes you stand out among F2F stakeholders? 8. Tell me about your reporting processes and how you report on indicators. What resources and/or tools for data and knowledge management do you have access to? What is missing? What would make your work easier (besides staff and time and money)? 9. Name the top 3 documents we must read, or top resource recommendations. 10. Please identify 1-3 people the evaluation team should interview? Their role and their knowledge area. 11. Let’s discuss field visits, travel scope of work (if applicable/if the IP manages one of the nine countries to be visited) 121 Annex 4: Template for Exploratory Interview (Rapid) Reporting TEMPLATE NOTES FROM F2F EXPLORATORY INTERVIEWS Date and Time: Ex. December 18, 2016 (spell out month) Organization Name: Name of the IP or other stakeholder type Respondent Name & Title: Name of the person being interviewed (if more than one￾person list names, and titles) Interviewer Name: Evaluation Team member name Location of Interview: Physical location or mode of communication (Skype, etc.) Country Project Summary: Quick summary of: what countries, what sectors, and key info about this person and/or organization. KEY INFORMATION AT-A-GLANCE Findings Key factors, events, statements … (etc.) discussed. Conclusions Might be interviewer conclusion or conclusion identified with interviewee. Just bullets. Can write in shorthand. 3 major points, for ex. Recommendations & Next Steps List of actionable next steps, including key ideas and insights to follow up on – or topics to note and explore further). Include titles of reports and documents the respondent sends interviewer. More information/details: (The guidelines above should cover it all, but if more elaboration or detail is useful put it here and in the next pages. This might include: direct quotes and testimonials, and very specific information that is relevant to what was discussed in the interview. Attach any relevant documents.) 122 Annex 5: US Department of State and USAID Joint Strategic Goal Framework DEPARTMENT OF STATE/USAID JOINT STRATEGIC GOAL FRAMEWORK Strategic Goal 1: Achieving Peace and Security • Counterterrorism • Weapons of mass destruction and destabilizing conventional weapons • Security cooperation and security sector reform • Conflict prevention, mitigation, and response • Transnational crime • Homeland security Strategic Goal 2: Governing Justly and Democratically • Rule of law and human rights • Good governance • Political competition and consensus building • Civil society Strategic Goal 3: Investing in People • Health • Education • Social services and protection for especially vulnerable populations Strategic Goal 4: Promoting Economic Growth and Prosperity • Private markets • Trade and investment • Energy security • Environment • Agriculture Strategic Goal 5: Providing Humanitarian Assistance • Protection, assistance, and solutions • Disaster prevention and mitigation • Orderly and humane means for migration management Strategic Goal 6: Promoting International Understanding • Offer a positive vision • Marginalize extremism • Nurture common interests and values Strategic Goal 7: Strengthening Consular and Management Capabilities • Consular services (Visas, Passports, American Citizen Services) • Major management functions 123 Annex 6: Standardized Key Informant Interview Guide This document is to be developed once the results of the web survey are available. The results will define the key questions to raise in the KII process and to develop the standardized Key Informant Interview Guide. 124 Annex 7: Announcement/Letter of Introduction from Implementing Partners FROM: Farmer-to-Farmer Program Web Survey [FarmerToFarmer@norc.org] SUBJECT: Upcoming evaluation of USAID’s Farmer-to-Farmer Program Dear [Firstname] [Lastname]: USAID, in partnership with Mendez England & Associates, have contracted with NORC at the University of Chicago, an independent research organization, to evaluate the Farmer-to-Farmer (F2F) Program. In particular, the evaluation team will examine the experiences of F2F Program stakeholders, their challenges and their activities as part of the program. Results from this evaluation will provide the Program with insights on the effectiveness of the F2F Program, best practices, and how the Program can be improved. As part of this evaluation, NORC will seek input from individuals, like yourself, who have participated in the F2F Program. One week from now, you will receive an email from NORC at the University of Chicago inviting you to participate in a short web survey. The survey will ask you questions about your work with the F2F Program, the challenges you have faced, and the types of activities that you or your organization have engaged in. We would very much appreciate your participation in this survey as the information you provide will help USAID and the F2F stakeholders to learn from and improve the Program. Your answers to the survey will be kept completely confidential and the researchers will not include your name or affiliation in any of their reports. Only summary data will be shared with USAID and included in the evaluation report. We highly encourage you to participate in this important effort, and thank you in advance for your time to respond to the survey NORC sends to you. In addition, if there is someone else from your organization who should be sent this survey or would be better able to answer questions about your work with the F2F Program, please send their name and email address to the research team at FarmerToFarmer@norc.org. You do not need to do anything at this time – the survey will be emailed to you in a few days’ time. If you have questions or concerns regarding this survey or our request, please send them to the research team at FarmerToFarmer@norc.org. Thank you in advance for your time. Sincerely yours, [SignatoryName] [SignatoryTitle] [SignatoryOrg] [IP Logo] 125 Annex 8: List of IPs, Countries, and Projects Partner Country Organizational Development Rural Enterprise Development Ag. Education & Training Financial Services Staple Crops41 Horticulture Livestock42 & Dairy Environmental Conservation Women & Youth Food Safety & Quality Climate Change Adaptation ACDI/VOCA – Europe, Caucus & Central Asia Tajikistan X X Kyrgyzstan X X X Georgia X X X Armenia X X ACDI/VOCA – West Africa Ghana X X X Liberia X X X Senegal X X X DRC X X CRS – East Africa Ethiopia X X X Kenya X X Tanzania X X Uganda X CNFA – Southern Africa Malawi X X Mozambique X X Angola X X Land O’Lakes – MENA Egypt X X Lebanon X X X Partners of the Americas – Caribbean Basin Nicaragua X X Haiti X Dominican Republic X Guatemala X X Winrock International – Ag. Education & Training Guinea X Nigeria X Senegal X Winrock International – Asia Bangladesh X X Burma X Nepal X X Not included in the chart are the following SPSP projects: • VEGA/PDP: FAVACA – Jamaica, Jamaican Sustainable Farm Enterprise. • VEGA/PDP: Purdue University – Colombia, F2F in Colombia’s Orinoquia Region. • VEGA/PDP: Vets Without Borders – Ethiopia and Uganda, Syndromic Surveillance for Livestock Health. • VEGA/PDP: IESC – Ghana, Improving Food Safety Systems Project. 41 Staple crops include legumes, oilseeds, and grains (wheat, maize, rice, millet etc.). 42 Livestock includes projects focused on poultry and aquaculture. 126 127 Annex 9: Standard Indicators 128 Annex 10: Feed the Future Goals 129 ANNEX C: WEB-BASED SURVEY QUESTIONS Farmer-to-Farmer Program Evaluation for Effectiveness and Learning (PEEL) [January 2017] This survey is part of an independent evaluation of the John Ogonowski and Doug Bereuter Farmer-to-Farmer Program, a program funded by the United States Agency for International Development (USAID). NORC, a research center associated with the University of Chicago, is a conducting this independent survey in partnership with Mendez England & Associates (ME&A) and on behalf of USAID to learn how the Farmer-to-Farmer (F2F) Program is operating from the perspective of host organizations, government stakeholders, volunteers and implementing partners. This survey is an important method for receiving truly representative opinions of the F2F Program. We value your insights, which will be used to help to improve the F2F Program and the experiences of its stakeholders. Your responses will help us to evaluate the effectiveness of the F2F Program and will guide USAID to improve the F2F Program in the future. This survey should take no more than 20-30 minutes to complete. Your participation in this survey is voluntary. If you are unable to answer a question, you may skip it. This survey is confidential. The information you provide will be used in summary form only and will not identify you as a participant of this survey. If you have questions or need assistance in any way, please e-mail the NORC research team at [FarmerToFarmer@norc.org] so that we may assist you. Thank you very much for your participation. By clicking the ‘Next’ button at the bottom right of the screen, you acknowledge your acceptance to participate in this survey. 130 Variable name Question Type Answer choice Volunteers Implementer Host Org. Government language How would describe your ability to read English/French/Spanish? [Only the name of the language assigned for this respondent will appear in the text for this question. For example, respondents who receive the survey in English will only see “English” in the text for the question.] Select one 1=Very poor [Go to end of survey--no more questions asked] 2=Poor [Go to end of survey--no more questions asked] 3=Fine 4=Good 5=Very good/fluent 1 1 1 1 relation What is your relationship to the F2F program? PROG: REQUIRED. Select one 1=Implementing Partner (i.e. An American NGO that is working on the Farmer-to-Farmer program) 2=Current volunteer 3=Former volunteer 4=Government Officer 5=Host organization (Directly receiving advice and services from the volunteer) 88=None of these [Go to end of survey--no more questions asked] 1 1 1 1 role What is your role with the Implementing Partner? Select one 1=Recruiter 2=In-country F2F implementer 3=F2F implementer in USA 4=Country Director 1 131 Variable name Question Type Answer choice Volunteers Implementer Host Org. Government prevvol [Only ask if relation==3] How long ago were you a Farmer￾to-Farmer (F2F) volunteer? Select one 1=Previous 0-2 years 2=Previous 3-5 years 3=6 or more years ago 1 hosttype What type of host organization do you belong to? Select one 1=Farmer cooperative or association 2=Individual private farmer 3=Other private enterprise 4=NGO 5=Educational institution 6=Public sector agency 7=Rural financial institution 98=Don't know 1 gender What is your gender? Select one 1=Male 2=Female 99=Prefer not to answer 1 1 1 1 ethnicity What is your ethnicity? Select one 1=Caucasian/not Hispanic 2=Caucasian/Hispanic 3=African American/not Hispanic 4=African American/Hispanic 5=American Indian or Alaskan Native 6=Asian American 7=Indian American 8=Hawaiian 96=Other: Specify[OPEN-ENDED] 99=Prefer not to answer 1 1 132 Variable name Question Type Answer choice Volunteers Implementer Host Org. Government volheard How did you first hear about the USAID Farmer to Farmer program? Select one 1. List serve/email 2. Social media (Facebook, twitter other) 3. Website 4. F2F Implementer 5. Former F2F volunteer 6. Program partner/stakeholder (not a F2F Implementer) 7. Event 96=Other: Specify [OPEN-ENDED] 1 voljob What was your primary employment previous to your most recent volunteer opportunity with F2F? Select one 1=Agribusiness 2=Educational institution employee 3=Retired 4=Student 5=Farmer 6=NGO 7=Government organization 8=Farm cooperative/association 9=Rural financial institution 1 volregion In which regions have you served as a volunteer for F2F? [Please select all that apply.] Select multiple 1=Asia 2=East Africa 3=Former Soviet Union 4=Latin America and Caribbean 5=Middle East and North Africa 6=Southern Africa 7=West Africa 8=Europe (including Balkan states) 96=Other: Specify[OPEN-ENDED] 1 volfreq How many times have you volunteered with F2F? Integer Range 1-10 1 133 Variable name Question Type Answer choice Volunteers Implementer Host Org. Government volfinish For all of the following questions in this survey, please think only about your most recent assignment with the Farmer-to-Farmer Program. How long ago did you finish your most recent F2F assignment? Select one 1=0-6 months 2=7-12 months 3=More than 1 year-3 years 4=More than 3 years-5 years 5=More than 5 years 1 volhelp What was the main focus of your activities in your most recent assignment? Select one 1. Increase agricultural sector productivity and profitability 2. Improve conservation and sustainable use of environmental and natural resources 3. Expand agricultural sector access to financial services 4. Strengthen agricultural sector institutions 5. Administrative/logistic development 96=Other: Specify[OPEN-ENDED] 1 voldays How many volunteer days was your most recent assignment? (Please only count travel days and days spent in the country of your volunteer work. Do not count days spent preparing for your departure or days spent following up after your assignment.) Select one 1=0-7 days 2= 8-14 days 3= 15-21 days; 4= 22-28 days; 5= More than 29 days 1 134 Variable name Question Type Answer choice Volunteers Implementer Host Org. Government voladvice How would you classify the main recommendations that you provided to the host organization for your most recent assignment? Select one 1=Economic 2= Organizational 3= Environmental 4 = Financial 96=Other: Specify[OPEN-ENDED] 1 voliphq_times How many times did you speak with the Implementing Partner's Head Office in the United States for the sake of your most recent assignment? Select one 1= 0 [Go to volsatis_IP] 2=1 - 3 3= 4-6 4= More than 7 times 1 voliphq_method [Only ask if voliphq_times!=1] How did you communicate with your Implementing Partner's Head Office in the United States? [Please select all that apply.] Select multiple 1=In person 2=Phone 3=Video communications (e.g. Skype, Face Time, etc.) 96=Other: Specify[OPEN-ENDED] 1 volsatis_IP How satisfied were you with the overall competency shown by the implementing partner? Select one 1= Very Dissatisfied, 2=Dissatisfied, 3=Neither Satisfied nor Dissatisfied, 4 = Satisfied, 5= Very Satisfied 1 135 Variable name Question Type Answer choice Volunteers Implementer Host Org. Government volassign What assistance/deliverables were due as part of your most recent assignment? [Please select all that apply.] Select multiple 1= Training/mentoring 2=Demonstrations 3= Educational materials/manuals 4= Business strategy plans 5=Business assessment 6= Equipment assembling/calibration/fix 7=Creation of a video, media or a website 96=Other: Specify[OPEN-ENDED] 1 volassign_numb How many deliverables were due? Integer Range 1-20 1 volassign_done What percentage of these deliverables did you complete? Select one 1=Less than 25%; 2=25-50%; 3=51-75%; 4= 76-99%; 5= 100% [Go to volsatis_recruit] 1 volassign_reason [Only ask if volassign_reason!=5] What were the inhibiting factors that prevented you from completing all of the deliverables? [Please select all that apply.] Select multiple 1=Length of assignment was too short 2=Project activities were delayed 3=Deliverables were not clearly stated 4=Lack of support from host organization 5=Lack of support from program/implementing agency 6=Activities competed were different than those listed in Statement of Work 96=Other: Specify[OPEN-ENDED] 1 136 Variable name Question Type Answer choice Volunteers Implementer Host Org. Government volsatis_recruit Please indicate the extent to which you agree or disagree with the following statements about your experience with the F2F program. The recruitment process was clear and well -organized. (For the recruitment process, please consider the entire period since you applied to the F2F Program until the time that you were official accepted into the Program for your most recent assignment.) Select one 1=Strongly disagree 2=Disagree 3=Neither disagree nor agree 4=Agree 5=Strongly agree 1 volsatis_skill I was matched with a project that fit my skills. Select one 1=Strongly disagree 2=Disagree 3=Neither disagree nor agree 4=Agree 5=Strongly agree 1 volsatis_pref My preferences were taken into account when matching me with a project. Select one 1=Strongly disagree 2=Disagree 3=Neither disagree nor agree 4=Agree 5=Strongly agree 1 volsatis_info I received all of the information I requested before beginning my assignment. Select one 1=Strongly disagree 2=Disagree 3=Neither disagree nor agree 4=Agree 5=Strongly agree 1 137 Variable name Question Type Answer choice Volunteers Implementer Host Org. Government volsatis_prep I felt well -prepared before beginning my assignment. Select one 1=Strongly disagree 2=Disagree 3=Neither disagree nor agree 4=Agree 5=Strongly agree 1 volsatis_motive I was strongly motivated for the full duration of my assignment. Select one 1=Strongly disagree 2=Disagree 3=Neither disagree nor agree 4=Agree 5=Strongly agree 1 volsatis_host The host activity that I supported was significantly improved. Select one 1=Strongly disagree 2=Disagree 3=Neither disagree nor agree 4=Agree 5=Strongly agree 1 volsatis_idea My opinions were valued and respected by my host organization. Select one 1=Strongly disagree 2=Disagree 3=Neither disagree nor agree 4=Agree 5=Strongly agree 1 volsatis_rec I made suggestions to the host organization on how to improve the activities related to my direct assignment Select one 1=Strongly disagree 2=Disagree 3=Neither disagree nor agree 4=Agree 5=Strongly agree 97=Not applicable 1 138 Variable name Question Type Answer choice Volunteers Implementer Host Org. Government volsatis_change The host organization made significant changes based on the suggestions I made Select one 1=Strongly disagree 2=Disagree 3=Neither disagree nor agree 4=Agree 5=Strongly agree 97=Not applicable 98=Don't know 1 volsatis_learn In my opinion, the host organization is well -equipped to learn from their projects over time. Select one 1=Strongly disagree 2=Disagree 3=Neither disagree nor agree 4=Agree 5=Strongly agree 97=Not applicable 98=Don't know 1 volsatis_tech I received all the technical support that I needed to succeed in my duties. (Technical support includes information, links to local technology programs, teaching aids, and equipment.) Select one 1=Strongly disagree 2=Disagree 3=Neither disagree nor agree 4=Agree 5=Strongly agree 97=Not applicable 98=Don't know 1 139 Variable name Question Type Answer choice Volunteers Implementer Host Org. Government volsatis_log I received all the logistical support that I needed to succeed in my duties. (Logistical support includes transport, housing, translators, compensation arrangements, and other services.) Select one 1=Strongly disagree 2=Disagree 3=Neither disagree nor agree 4=Agree 5=Strongly agree 97=Not applicable 98=Don't know 1 volsatis_bens My discussions with the host organization included careful consideration of how potential benefits might be distributed differently between men and women. Select one 1=Strongly disagree 2=Disagree 3=Neither disagree nor agree 4=Agree 5=Strongly agree 1 volsatis_gender Women and men are equal participants in discussions related to program design and implementation Select one 1=Strongly disagree 2=Disagree 3=Neither disagree nor agree 4=Agree 5=Strongly agree 1 voljobin What percentage of your working hours were dedicated to on -the -job training to members of your host organization during your assignment? (Please enter 0 if you did not spend any time on this activity) Percent ______ % 1 140 Variable name Question Type Answer choice Volunteers Implementer Host Org. Government voltrainin What percentage of your working hours were dedicated to giving formal training to members of your host organization during your assignment through workshops or other formal training opportunities? (Please enter 0 if you did not spend any time on this activity) Percent _______ % 1 volmat [Only ask if voltrainin>0] Were you responsible for preparing most of the materials for this formal training? Select one 1= Yes 2= No 98= Don’t know 1 volmattime When where these materials prepared? Select one 1=Before traveling to the host country 2=Prepared once in the host country 3=Drafted before travel and finalized once in the host country 98=Don't know 1 voltot Did your assignment involve training of trainers (TOT) or preparation of training materials or other activities that would result in continued training in country after your assignment? Select one 1= Yes 2= No 98=Don’t know 1 voljobout What percentage your working hours were dedicated to on-the-job training of individuals outside of your Percent ___________ % 1 141 Variable name Question Type Answer choice Volunteers Implementer Host Org. Government host organization during your assignment? (Please enter 0 if you did not spend any time on this activity) voltrainout What percentage of your working hours were dedicated to giving formal training to individuals outside of your host organization during your assignment? (Please enter 0 if you did not spend any time on this activity) Percent ___________ % 1 volcom How often have you communicated with your host organization since you left? Select one 1=Daily 2=Weekly 3=Monthly 4=Less than 1 time a month 5=Never [Go to volexitint] 1 volcom_reason [Only ask if volcom!=5] You mentioned that you communicated with your host organization since you left. What was the primary purpose of this communication? Select one 1=Finalizing deliverables 2=Providing ongoing guidance related to your assignment 3=Providing ongoing guidance unrelated to your assignment 4=Personal communications 96=Other: Specify[OPEN-ENDED] 1 142 Variable name Question Type Answer choice Volunteers Implementer Host Org. Government volcom_method [Only ask if volcom!=5] What was the most common method you used for these communications? Select one 1=Email 2=Phone call 3=F2F web portal 4=Through the implementing organization 96=Other: Specify[OPEN-ENDED] 1 volexitint Did you have a meeting with the implementing organization to discuss your reports and recommendations at the conclusion of your most recent volunteer assignment? Select one 1= Yes 2= No 98= Don’t know 1 voloutr Have you been involved in any outreach activities related to F2F since returning to the United States? (Outreach includes events like press releases to local media, media events, social media posts on behalf of F2F, and group or individual presentations.) Select one 1= Yes 2= No [Go to volreason] 98= Don’t know [Go to volreason] 1 volpublic [Only ask if voloutr==1] What outreach activities have you taken part in? Select multiple 1=Gave presentation or participated in panel related to F2F 2=Spoke to local media regarding F2F 3=Wrote press release or other article for local media 4=Social media posts on behalf of F2F 96=Other: Specify[OPEN-ENDED] 1 143 Variable name Question Type Answer choice Volunteers Implementer Host Org. Government voloutr_none [Only ask if voloutr==1] What reasons prevented you from taking part in any outreach activities related to F2F since returning to the United States? [Please select all that apply.] Select multiple 1=Lack of personal interest in conducting outreach 2=Was not aware of any outreach events or opportunities 3=Had not previously considered conducting any outreach 4=Had a negative impression of F2F 5=Lack of time to give to outreach events 6=Was not aware that outreach related to F2F was allowed 96=Other: Specify[OPEN-ENDED] 1 volreason What were the reasons why you wanted to volunteer with the F2F Program? [Choose up to 3 most important reasons] Select multiple 1=Had volunteered with F2F previously 2=Personally know others who volunteered with F2F 3=Interest in cultural exchange 4=Interest in supporting host organizations and countries 5=Previous (non-F2F) experience in developing countries 6=Personal relationship to NGOs who are implementing F2F activities 7=Desire to travel 8=Interest in "giving back" 96=Other: Specify[OPEN-ENDED] 1 volchallenge What were the main challenges you faced as a volunteer? [Please select all that apply.] Select multiple 1=Cultural barriers 2=Language barriers 3=Recruitment process was unclear or poorly organized 4=Poor communication from host organization 5=Poor communication from program/implementing agency 6=Assigned to project outside of area of interest/expertise 7=Expectations for assignment not clearly explained 1 144 Variable name Question Type Answer choice Volunteers Implementer Host Org. Government 8=Expectations for assignment changed frequently 9=Poor support from host organization 10=Poor support from program/implementing agency 11=Logistical challenges 12=No significant challenges faced 96=Other: Specify[OPEN-ENDED] volagain How likely are you to volunteer again with F2F? Select one 1=Not at all likely 2=Somewhat unlikely 3=Unsure 4=Somewhat likely 5=Very likely 1 volinfo [Only ask if volsatis_prep==1 or 2 or 3] You previously mentioned that you did not feel fully prepared before beginning your most recent assignment. Which of the following topics would you have liked to receive more information about before your assignment? [Please select all that apply.] Select multiple 1=Travel information 2=Health and security issues 3=Logistics and support 4=Country and cultural information 5=Information on the organization and individuals you would be working with 6=Specific tasks of the assignment 88=Received all necessary information 96=Other: Specify[OPEN-ENDED] 1 iporg How do you classify your organization? Select one 1= Cooperatives and Associations 2= Individual Private Farmers 3= Other Private Enterprises 4= Non-Profit, Public Interest NGOs 5= Public & Private Education Institutions 6= Rural Financial Institutions 7= Public Sector Agencies 1 145 Variable name Question Type Answer choice Volunteers Implementer Host Org. Government ipyears Approximately, how many years has your organization been an implementer for the F2F Program in any country? (Please consider all years, not only those for the current program and include time spent for any type of implementing arrangement, including sub -agreements.) Select one 1=0 -2 years 2=3 -5 years 3=6 -10 years 4=More than 10 years 98=Don't know 1 ipcontr What is the primary type of F2F award that you manage? Select one 1= Core programs 2= PDPs programs 3= Small grants 98=Don't know 1 ipcontr_amt What is the approximate size of your organization's current grant with the F2F Program? Select one 1= $100,000 -$150,000 2= $150,000 -$600,000 3=$600,001 -$1,500,000 3= $1.5 million -$6 million 4=More than $6 million 98=Don't know 1 ipyears_abroad How many years has your organization been operating in the country or region where you are based? Select one 1=1 -3 years 2=4 -6 years 3=5 -10 years 4=More than 10 years 98=Don't know 1 ipempl How many full time individuals in your organization spend the majority of their time working on the F2F program in the Select one 1=Less than 3 2= 4 - 7 3= 8 -12 4= 13 -20 1 146 Variable name Question Type Answer choice Volunteers Implementer Host Org. Government country/region where you are based? 5=More than 20 98=Don't know ipf2f Is F2F the only activity that your organization is implementing in the country/region where you are based? Select one 1= Yes 2= No 97=Not applicable 98=Don't know 1 ipproj How many other project/programs do you personally manage, other than F2F projects? Integer Range 0-10 1 ipsector What sectors is your organization currently engaged in for the F2F Program in the country where you are based? [Please select all that apply.] Select multiple 1= Agricultural Education & Training 2= Horticulture 3= Field Crops 4= Livestock & Dairy 5= Rural Enterprise Development 6= Youth in Agriculture 7= Financial Services 8= Producer Organization Development 9= Environment & NRM 10= Market Systems Development 11= Food Safety 97=Not applicable, there are no F2F programs in the country where I am based 1 ipext How often do you personally use information from the USA Extension Services, USDA, CGIAR International Research Select one 1=Never 2=Rarely 3=Sometimes 1 147 Variable name Question Type Answer choice Volunteers Implementer Host Org. Government Centers, or other science - based sources for your personal work? 4=Often 5=Very often ipind Please indicate the extent to which you agree or disagree with each of the following statements. The information provided by the F2F indicators is used by your organization to make program management decisions Select one 1=Strongly disagree 2=Disagree 3=Neither disagree nor agree 4=Agree 5=Strongly agree 97=Not applicable 98=Don't know 1 ipind_me The information provided by the F2F indicators is important for monitoring and evaluating the project's work at your organization Select one 1=Strongly disagree 2=Disagree 3=Neither disagree nor agree 4=Agree 5=Strongly agree 97=Not applicable 98=Don't know 1 ipind_dcoll Which indicators are difficult to collect? [Please select up to 3 indicators] Multiple selection 3max List of all F2F indicators 1 ipind_dmgmt Which indicators provide valuable information for making management decisions related to your program? [Please select up to 3 indicators] Multiple selection 3max List of all F2F indicators 1 148 Variable name Question Type Answer choice Volunteers Implementer Host Org. Government ipind_simp Which indicators could be simplified? [Please select up to 3 indicators] Multiple selection 3max List of all F2F indicators 1 ipind_elim Which indicators might be eliminated? [Please select up to 3 indicators] Multiple selection 3max List of all F2F indicators 1 ipind_add Are there other indicators needed, in addition to the current indicators? Select one 1= Yes 2= No [Go to ipind_outr] 98= Don’t know [Go to ipind_outr] 1 ipind_new [Only ask if ipind_add==1] You said that new indicators are needed. Which indicators should be added? List no more than 3 new indicators Text 1=_________________________2=__________________________ 3=___________________________ 1 ipind_outr F2F's current indicators related to outreach include the number of press releases, media events and group presentations completed. In your opinion, do these indicators effectively capture the program's performance related to outreach? Select one 1= Yes [Go to ipind_cult] 2= No 98= Don’t know [Go to ipind_cult] 1 ipind_outrc [Only ask if ipind_outr==2] What changes would you recommend to these indicators to more effectively capture the program's performance related to Select multiple 1=Add some new indicators 2=Remove some old indicators [Go to ipind_cult] 3= No changes needed [Go to ipind_cult] 1 149 Variable name Question Type Answer choice Volunteers Implementer Host Org. Government outreach? [Please select all that apply.] ipind_outrn [Only ask if ipind_outrc==1] Which additional indicators would you recommend adding to more effectively capture the program's performance related to outreach? Text 1=__________________________2=__________________________ 1 ipind_cult Is F2F effectively capturing the performance and impact on cultural exchange? Select one 1= Yes [Go to ipind_cap] 2= No 3= Don’t know [Go to ipind_cap] 1 ipind_cultc [Only ask if ipind_cult==2] What changes would you recommend to these indicators to more effectively capture the program's performance and impact on cultural exchange? [Please select all that apply.] Select multiple 1=Add some new indicators 2=Remove some old indicators [Go to ipind_cap] 3= No changes needed [Go to ipind_cap] 1 ipind_cultn [Only ask if ipind_cultc==1] Which additional indicators would you recommend adding to more effectively capture the program's performance and impact on cultural exchange? Text 1=__________________________ 2=__________________________ 1 150 Variable name Question Type Answer choice Volunteers Implementer Host Org. Government ipind_cap F2F's current indicators related to capacity development include the number of persons trained, the number of persons directly assisted, the number of volunteer recommendations, new or improved products/services, and the organizational development index for organizational strengthening objectives. In your opinion, do these indicators effectively capture the program's performance related to capacity development? Select one 1= Yes [Go to ipind_env] 2= No 3= Don’t know [Go to ipind_env] 1 ipind_capc [Only ask if ipind_cap==2] What changes would you recommend to these indicators to more effectively capture the program's performance related to capacity development? [Please select all that apply.] Select multiple 1=Add some new indicators 2=Remove some old indicators [Go to ipind_env] 3= No changes needed [Go to ipind_env] 1 ipind_capn [Only ask if ipind_capc==1] Which additional indicators would you recommend adding to more effectively capture the program's performance Text 1=__________________________ 2=__________________________ 1 151 Variable name Question Type Answer choice Volunteers Implementer Host Org. Government related to capacity development? ipind_env F2F's current indicator related to environmental/natural resource management includes the area under improved environmental/natural resource management. In your opinion, does this indicator effectively capture the program's performance related to environmental/natural resource management? Select one 1= Yes [Go to ipind_comm] 2= No 3= Don’t know [Go to ipind_comm] 1 ipind_envc [Only ask if ipind_env==2] What changes would you recommend to more effectively capture the program's performance related to environmental/natural resource management? [Please select all that apply.] Select multiple 1=Add some new indicators 2=Remove some old indicators [Go to ipind_comm] 3= No changes needed [Go to ipind_comm] 1 ipind_envn [Only ask if ipind_envc==1] Which additional indicators would you recommend adding to more effectively capture the program's performance related to Text 1=__________________________2=__________________________ 1 152 Variable name Question Type Answer choice Volunteers Implementer Host Org. Government environmental/natural resource management? ipsatis_comm How satisfied are you with the level of collaboration and information sharing among F2F implementers? Select one 1= Very Dissatisfied 2=Dissatisfied 3=Neither Satisfied nor Dissatisfied 4 = Satisfied 5= Very Satisfied 1 iplearn In your opinion, in what ways could the program improve learning to become more efficient and effective? [Please select all that apply.] Select multiple 1. Listen to stakeholders' perceptions and ideas 2. Provide feedback to suggestions 3. Analyze/discuss issues w/ stakeholders 4. Facilitate sharing of practices between organizations 5.Update the F2F Good Practice Manual w/ a practical quick guide 88: No improvements needed 96=Other: Specify [OPEN-ENDED] 1 153 Variable name Question Type Answer choice Volunteers Implementer Host Org. Government iplearn_prac Which are the most effective current practices by F2F for encouraging learning among F2F stakeholders? [Please select all that apply.] Select multiple 1=F2F Program Implementers' Meeting 2=Web-based training for F2F Program Implementers for performance and indicator monitoring and reporting 3=Website for F2F information and good practice exchange 4=Seminars on volunteer program operations and management 5=Best practice manual 6= Informal networking 7 =None are particularly effective 96=Other: Specify [OPEN-ENDED] 1 ipshare How should the F2F Program facilitate the sharing of best practices and innovations between organizations? [Please select up to three.] Select 3 max 1. Additional USA/Regional/Country partner meetings to exchange innovations 2. Facilitate the presence of specialist to IPs in on-line workshops. 3. Produce, disseminate, and analyze case studies among IPs. 4. Webinars 5. Require Implementing Partners to report their innovations initiatives. 6. Country visits by USAID staff including agriculture development specialists 88: No changes needed 96=Other: Specify[OPEN-ENDED] 1 154 Variable name Question Type Answer choice Volunteers Implementer Host Org. Government ipprior Which of the following methods would be most helpful for you to learn about best practices and innovations? [Please select up to two.] Select 2 max 1. Innovations/reports posted on the internet 2. Discussion groups 3. Volunteer chat rooms 4. Shared innovations from IP's categorized by subject matter 96=Other: Specify[OPEN-ENDED] 1 ipinnov What factors affect your ability to implement relevant, innovative practices which you learn about? Multiple selection 1= Technical limitations/ lack of knowledge/ issues too complex 2= Budget constrains 3= Recommended materials not available locally 4= Cultural barriers 5= Logistical constrains 6= Legal constrains 7=No difficulties in implementing new practices 96=Other: Specify [OPEN-ENDED] 1 ipinnov_use How satisfied are you with your ability to find and use innovation practices for your work on the F2F Program? Select one 1= Very Dissatisfied 2=Dissatisfied 3=Neither Satisfied nor Dissatisfied 4 = Satisfied 5= Very Satisfied 1 155 Variable name Question Type Answer choice Volunteers Implementer Host Org. Government ipinnov_prior Which areas of innovation should be prioritized to improve the effectiveness of the F2F Program? Multiple selection 1. Policy and data 2. Research 3. Extension and advisory services 4. Education 5. Agribusiness/value chains 88: No changes needed 96=Other: Specify [OPEN-ENDED] 1 ipprac Which of the following areas do F2F host organizations most commonly ask for information and assistance? [Please select up to three.] Multiple selection; 3 max 1= Food Security 2=Economic Development 2= Improved plant varieties 3= Low-energy agricultural systems 4= Post-harvest technologies 5= Use of best agricultural practices 6= Use/improve of labor saving devices 7= Management practices 8= Youth enterprise (e.g. youth-related activities) 9= Access new market information and linkages 10= Develop targeted marketing/ ICT technologies 11= Branding strategies 12= Regional trade development 13= Development of rural financial system 88= None of the above 96=Other: Specify [OPEN-ENDED] 1 ipneeds_n Which of the following concerns have you personally experienced with the F2F Program? [Please select all that apply.] Multiple selection 1. Lack of responsiveness to implementing partner concerns/questions 2. Lack of support in volunteer recruitment 3. Delays in distributions of funds 4. Lack of facilitation of learning 88= None of the above; 96=Other: Specify [OPEN-ENDED] 1 156 Variable name Question Type Answer choice Volunteers Implementer Host Org. Government ipkeys What are key issues for future success of the F2F program? [Please select up to three.] Multiple selection 1= Improvements to M&E system/indicators 2= Recruitment of volunteers with more specialized expertize 3= Recruitment of more diverse volunteers 4= Better volunteer orientation 5= Involvement of new host organizations 6= Collaboration with local governments 7= Outreach programs by past volunteers 8= More staff training 9= More country technical staff 10= Outreach programs in country 11=Continued PDP and small grant programs 12=Continued SPSP knowledge management activities 96=Other: Specify [OPEN-ENDED] 1 ipspsp Based on your experience of the current and completed SPSP projects, are the SPSP (Special Programs Support Project) Small Grant and PDP (Program Development Project) projects an effective way to support learning, innovation and community of practice?" (Please note that the SPSP activities include all activities carried out by VEGA for the sake of the F2F Program.) Select one 1= Yes 2= No 98= Don’t know 1 157 Variable name Question Type Answer choice Volunteers Implementer Host Org. Government ipspsp_pgm Which of the following types of activities by a F2F Special Program Support Project (SPSP) have been most helpful for your work? (Please choose up to 3) Select multiple (Max of 3) 1=Test new ways of using volunteers 2=Experience gained from funding PDPs 3=Experience gained from funding Small Grants (SGs) 4=Web-based training for F2F Program Implementers for performance and indicator monitoring and reporting 5=Website for F2F information and good practice exchange 6=Seminars on volunteer program operations and management 7=SPSP activities are relevant to my work but have not been helpful 98=Don't know 1 ipspsp_satis How satisfied are you with the support you have received from the SPSP programs? Select one 1= Very Dissatisfied 2=Dissatisfied 3=Neither Satisfied nor Dissatisfied 4 = Satisfied 5= Very Satisfied 97=Not applicable 1 158 Variable name Question Type Answer choice Volunteers Implementer Host Org. Government ipspsp_imp What impacts have SPSP sub￾award projects had on your F2F program and its activities? Multiple selection 1= Generated new ideas and innovations 2= Supported knowledge management between and within the organization 3= Involved new implementing partners 4= Provided new approaches for using volunteers 5= Facilitated special studies 6= Provide training 7= No significant impacts 8=SPSP sub-award projects have had negative impacts on your F2F program/activities 96=Other: Specify [OPEN-ENDED] 1 spsp_neg [Only ask if ipspsp_imp==8] What negative impacts have the SPSP sub-award projects had on your F2F program and its activities? Text 1 ipspsp_limit In your experience, what factors are likely to limit the impact of the SPSPs? (Choose up to 3) Multiple selection 1= Legal constraints; 2= Budget constraints; 3= Logistic constraints; 4= Lack of coordination w/ F2F program; 5=Lack of response to stakeholder feedback 96=Other: Specify [OPEN-ENDED] 98= Don't know 1 159 Variable name Question Type Answer choice Volunteers Implementer Host Org. Government iplwa_satis How satisfied are you with the LWA mechanism in promoting F2F objectives? Select one 1= Very Dissatisfied, 2=Dissatisfied, 3=Neither Satisfied nor Dissatisfied, 4 = Satisfied, 5= Very Satisfied 97=Not applicable 98=Don't know 1 ipvol_neg How satisfied are you with the current ability to share and access information about negative volunteer assignments? Select one 1= Very Dissatisfied 2=Dissatisfied 3=Neither Satisfied nor Dissatisfied 4 = Satisfied 5= Very Satisfied 97=Not applicable 98=Don't know 1 ipvol_share What system(s) would be the most effective for improving how this information about negative volunteer assignments is shared? [Please select all that apply.] Select multiple 1=Central database accessible to all IPs 2=Ability to make direct requests to USAID regarding each volunteer 96=Other: Specify [OPEN-ENDED] 88=No changes needed 1 ipsatis_fem How satisfied are you with your organization's ability to recruit female volunteers? Select one 1= Very Dissatisfied [Go to ipsatis_femr] 2=Dissatisfied [Go to ipsatis_femr] 3=Neither Satisfied nor Dissatisfied [Go to ipsatis_femr] 4 = Satisfied 5= Very Satisfied 97=Not applicable 98=Don't know 1 160 Variable name Question Type Answer choice Volunteers Implementer Host Org. Government ipsatis_femr [Only ask if ipsatis_fem==1 or 2 or 3] What factors prevent your organization from recruiting more female volunteers? [Please select all that apply.] Select multiple 1=Lack of qualified applicants 2=Lack of support from USAID 3=Host organizations less interested in female candidates 4=Lack network to women-focused organizations to identify female volunteers 96=Other: Specify [OPEN-ENDED] 97=Not applicable 98=Don't know 1 ipsatis_min How satisfied are you with your organization's ability to recruit minority volunteers? Select one 1= Very Dissatisfied [Go to ipsatis_minr] 2=Dissatisfied [Go to ipsatis_minr] 3=Neither Satisfied nor Dissatisfied [Go to ipsatis_minr] 4 = Satisfied 5= Very Satisfied 97=Not applicable 98=Don't know 1 ipsatis_minr [Only ask if ipsatis_min==1 or 2 or 3] What factors prevent your organization from recruiting more minority volunteers? [Please select all that apply.] Select multiple 1=Lack of qualified applicants 2=Lack of support from USAID 3=Host organizations less interested in minority candidates 4=Lack network to minority-focused organizations to identify minority volunteers 96=Other: Specify [OPEN-ENDED] 97=Not applicable 98=Don't know 1 161 Variable name Question Type Answer choice Volunteers Implementer Host Org. Government ipsatis_monit How satisfied are you with your organization's ability to monitor the work of its volunteers? Select one 1= Very Dissatisfied 2=Dissatisfied 3=Neither Satisfied nor Dissatisfied 4 = Satisfied 5= Very Satisfied 1 ipbens Please indicate the extent to which you agree or disagree with each of the following statements. My discussions with the host organization and volunteer included careful consideration of how potential benefits might be distributed differently between men and women. Select one 1=Strongly disagree 2=Disagree 3=Neither disagree nor agree 4=Agree 5=Strongly agree 97=Not applicable 98=Don't know 1 ipsatis_gen Women and men are equal participants in discussions related to program design and implementation Select one 1=Strongly disagree 2=Disagree 3=Neither disagree nor agree 4=Agree 5=Strongly agree 97=Not applicable 98=Don't know 1 ipsatis How satisfied have you been with your involvement in the F2F Program overall? Select one 1= Very Dissatisfied 2=Dissatisfied 3=Neither Satisfied nor Dissatisfied 4 = Satisfied 5= Very Satisfied 1 ipchallenge What were the main challenges you have faced as part of your participation with Select multiple 1=Cultural barriers 2=Language barriers 3=Recruitment process of volunteer was unclear or poorly organized 4=Poor communication from volunteer 1 162 Variable name Question Type Answer choice Volunteers Implementer Host Org. Government the F2F program? [Please select all that apply.] 5=Poor communication from host 6=Volunteer lacked necessary skills 7=Volunteer's duties were not clearly explained 8=Expectations for assignment changed frequently 9=Poor support from host 10=Poor support from USAID 11=Logistical challenges 12=No significant challenges faced 96=Other: Specify [OPEN-ENDED] hostemp How many full time, contract staff does your organization employee? Select one 1=Less than 10 2=10-20 3=20-40 4=40-60 5=More than 60 98=Don't know 1 hostsect What type of assistance has your organization received as part of the F2F Program? [Please select all that apply.] Select multiple 1=Technology transfer 2=Organizational development 3=Business/Enterprise development 4=Environmental conservation 5=Financial services 6-Administrative 96=Other: Specify [OPEN-ENDED] 1 hostsatis How satisfied have you been with your involvement in the F2F Program overall? Select one 1= Very Dissatisfied 2=Dissatisfied 3=Neither Satisfied nor Dissatisfied 4 = Satisfied 5= Very Satisfied 1 163 Variable name Question Type Answer choice Volunteers Implementer Host Org. Government hostsatis_goal How satisfied were you with your organization's involvement in developing a plan for support from the F2F program, including identification of your organizations goals and expected results from participating in the program? Select one 1= Very Dissatisfied 2=Dissatisfied 3=Neither Satisfied nor Dissatisfied 4 = Satisfied 5= Very Satisfied 1 hostsatis_sow How satisfied were you with your organization's involvement in developing the Statement of Work for the volunteer, which detailed the volunteer's responsibilities and activities? Select one 1= Very Dissatisfied 2=Dissatisfied 3=Neither Satisfied nor Dissatisfied 4 = Satisfied 5= Very Satisfied 1 164 Variable name Question Type Answer choice Volunteers Implementer Host Org. Government hostsow At what stage was your organization involved in developing the Statement of Work for the volunteer? Select one 1=Your organization created first version of the Statement of Work 2=Your organization and the implementing partner created first version of the Statement of Work together 3=Implementing partner created first version of the Statement of Work, which your organization made changes to 4=Your organization received finalized Statement of Work without the opportunity to provide ideas 5=Your organization never received Statement of Work for volunteer 96=Other: Specify [OPEN-ENDED] 98=Don't know 1 hostsatis_rec Please answer the following questions related to your satisfaction at your organization. How satisfied were you with the recommendations made by the volunteer? Select one 1= Very Dissatisfied 2=Dissatisfied 3=Neither Satisfied nor Dissatisfied 4 = Satisfied 5= Very Satisfied 1 hostsatis_otj How satisfied were you with the quality of the on-the-job training your organization received from the volunteer? Select one 1= Very Dissatisfied 2=Dissatisfied 3=Neither Satisfied nor Dissatisfied 4 = Satisfied 5= Very Satisfied 1 165 Variable name Question Type Answer choice Volunteers Implementer Host Org. Government hostsatis_train How satisfied were you with the quality of the formal training your organization received from the volunteer? Select one 1= Very Dissatisfied 2=Dissatisfied 3=Neither Satisfied nor Dissatisfied 4 = Satisfied 5= Very Satisfied 1 hostsatis_monit How satisfied are you with your organization's ability to monitor the work of its volunteers? 1= Very Dissatisfied 2=Dissatisfied 3=Neither Satisfied nor Dissatisfied 4 = Satisfied 5= Very Satisfied 1 hostvol_done Have any volunteers already completed their assignment and left the country? Select one 1= Yes 2= No [Go to hostrec_type] 3= Don’t know [Go to hostrec_type] 1 hostvol_comm [Only ask if hostvol_done==1] How often have you communicated with this/these volunteers since they left? Select one 1=Daily 2=Weekly 3=Monthly 4=Less than 1 time a month 5=Never [Go to hostrec_type] 1 hostvol_purp [Only ask if hostvol_comm!=5] You mentioned you communicated with these volunteers since they left. What was the primary purpose of this communication? Select one 1=Finalizing deliverables 2=Provided ongoing guidance related to their assignment 3=Provided ongoing guidance unrelated to their assignment 4=Personal communications 96=Other: Specify [OPEN-ENDED] 1 166 Variable name Question Type Answer choice Volunteers Implementer Host Org. Government hostvol_method [Only ask if hostvol_comm!=5] What was the most common method you used for these communications? Select one 1=Email 2=Phone call 3=F2F web portal 4=Through the implementing organization 5=Social media (e.g. Facebook, Twitter) 96=Other: Specify [OPEN-ENDED] 1 hostrec_type What types of recommendations did you receive from the volunteer? Select one 1=Economic 2= Organizational 3= Environmental 4 = Financial 96=Other: Specify [OPEN-ENDED] 1 hostrec Approximately what percentage of the volunteer's recommendations have you already implemented? Percent 1 hostrec_lim What factors have limited your ability to implement all of the recommendations? [Please select all that apply.] Select multiple 1=Volunteer left recently, so have not had time to implement all recommendations 2=Lack of financial resources 3=Lack of local access to goods/services mentioned in recommendations 4=Legal restrictions on enacting recommendations 5=Did not find recommendations practical or helpful 96=Other: Specify [OPEN-ENDED] 1 167 Variable name Question Type Answer choice Volunteers Implementer Host Org. Government hostsatis_vol Please read the following statements and indicate the extent to which you agree or disagree. The work which was supported by the volunteer was significantly improved because of their contributions. Select one 1=Strongly disagree 2=Disagree 3=Neither disagree nor agree 4=Agree 5=Strongly agree 97=Not applicable 98=Don't know 1 hostsatis_resp The opinions of my organization were valued and respected by the volunteer. Select one 1=Strongly disagree 2=Disagree 3=Neither disagree nor agree 4=Agree 5=Strongly agree 97=Not applicable 98=Don't know 1 hostsatis_bens My discussions with the volunteer included careful consideration of how potential benefits might be distributed differently between men and women. Select one 1=Strongly disagree 2=Disagree 3=Neither disagree nor agree 4=Agree 5=Strongly agree 97=Not applicable 98=Don't know 1 hostsatis_gend Women and men are equal participants in discussions related to program design and implementation Select one 1=Strongly disagree 2=Disagree 3=Neither disagree nor agree 4=Agree 5=Strongly agree 97=Not applicable 98=Don't know 1 168 Variable name Question Type Answer choice Volunteers Implementer Host Org. Government host What percentage of the volunteer's working hours were dedicated to on-the-job training of your personnel? (Please enter 0 if the volunteer did not spend any time on on-the-job training of your personnel) Percent ___________ % 1 hossatis_otj How satisfied were you with the quality of the on-the job training you received from the volunteer? [Only ask if host!=0] Select one 1= Very Dissatisfied 2=Dissatisfied 3=Neither Satisfied nor Dissatisfied 4 = Satisfied 5= Very Satisfied 1 hosttrainin What percentage of the volunteer's working hours were dedicated to formal training to members of your organization through workshops or other formal training opportunities? (Please enter 0 if the volunteer did not spend any time on formal training of your personnel) Percent __________ % 1 hostsatis_form How satisfied were you with the quality of the formal training your organization received from the volunteer? [Only ask if hosttrainin!=0] Select one 1= Very Dissatisfied 2=Dissatisfied 3=Neither Satisfied nor Dissatisfied 4 = Satisfied 5= Very Satisfied hostchallenge What were the main challenges you have faced as part of your participation with Select multiple 1=Cultural barriers 2=Language barriers 3=Recruitment process of volunteer was unclear or poorly organized 4=Poor communication from volunteer 1 169 Variable name Question Type Answer choice Volunteers Implementer Host Org. Government the F2F program? [Please select all that apply.] 5=Poor communication from implementing agency 6=Volunteer lacked necessary skills 7=Volunteer's duties were not clearly explained 8=Expectations for assignment changed frequently 9=Poor support from implementing agency 10=Logistical challenges 11=No significant challenges faced 96=Other: Specify [OPEN-ENDED] hostvalid Which of the following systems do you use to verify the data that is used in F2F reports? [Please select all that apply.] Select multiple 1=Written records or written copies 2=The same data collected by 2 or more staff 3=Data collected by someone with no association to the F2F Program 4=No verification system is used [END SURVEY] 96=Other: Specify [OPEN-ENDED] 1 hoststore How do you store your verification records? [Only ask if hostvalid!=4] Select one 1=Location of records changes frequenty 2=Records kept in a designated area that is unlocked 3=Records kept in a designated area that is locked 4=Do not keep any documents or records related to your verification process 98=Don't know 1 170 Variable name Question Type Answer choice Volunteers Implementer Host Org. Government govjob What is your primary job? Select one 1= Central government official for agriculture / rural development / environment 2=Central government official for economics / development / Planning/ gender 3= Central government official for irrigation/ water management 4= Other central government official 5= Decision maker level at Provincial / District/ Municipality 6=Sub-national government official for agriculture/rural development/environment 7=Sub-national government official for economics/development planning/gender 8=Sub-national government official for irrigation/water management 9=Other sub-national government official 10= University Professor / Researcher / Administrator 11= Educator for secondary school/vocation school 12= Agriculture Extension Officer/ Agriculture Outreach Officer 13= Private sector 96=Other: Specify [OPEN-ENDED] 1 171 Variable name Question Type Answer choice Volunteers Implementer Host Org. Government govlearn When did you learn about the F2F Program? Select one 1=Less than 1 year ago 2=1 -3 years ago 3=4 -6 years ago 4=More than 6 years ago 98=Don't know 1 govlearn_how How did you learn about the F2F Program? Select one 1=Local NGO/CBO 2=International NGO 3=Colleague 4=Friend or Family member 5=Private company 96=Other: Specify [OPEN -ENDED] 98=Don't know 1 govcomm How often have you communicated with the F2F personnel? Select one 1=Daily 2=Weekly 3=Monthly 4=Once a year 5=Never [Go to govsatis_bens] 1 govcomm_method [Only ask if govcomm!=5] You mentioned you communicated with F2F personnel. What was the most common method you used for these communications? Select one 1=Email 2=Phone call 3=F2F web portal 96=Other: Specify [OPEN -ENDED] 1 govcomm_purp [Only ask if govcomm!=5] What was the primary focus of this communication? Select one 1= Sharing accomplishments 2= Logistic, technical support / Guidance 3= Personal communications 4=Requesting support from F2F volunteer 96=Other: Specify[OPEN -ENDED] 1 172 Variable name Question Type Answer choice Volunteers Implementer Host Org. Government govsatis_bens Please indicate the extent to which you agree or disagree with each of the following statements. My discussions with the host organization and volunteer included careful consideration of how potential benefits might be distributed differently between men and women. Select one 1=Strongly disagree 2=Disagree 3=Neither disagree nor agree 4=Agree 5=Strongly agree 97=Not applicable 98=Don't know 1 govsatis_gen Women and men are equal participants in discussions related to program design and implementation Select one 1=Strongly disagree 2=Disagree 3=Neither disagree nor agree 4=Agree 5=Strongly agree 97=Not applicable 98=Don't know 1 govperc How satisfied have you been with the F2F Program overall? Select one 1= Very Dissatisfied 2=Dissatisfied 3=Neither Satisfied nor Dissatisfied 4 = Satisfied 5= Very Satisfied 1 govmore Do you think more local organizations could benefit from support from F2F? Select one 1= Yes 2= No 98= Don’t know 1 173 ANNEX D: KEY INFORMANT INTERVIEW (KII) GUIDE AND QUESTIONS Interview Guide: Key Informant Interviews (KIIs) Prepared for USAID Farmer-to-Farmer Evaluation By Mendez England & Associates April 5, 2017 Washington, DC Purpose of this Guide The purpose of this document is to succinctly describe the approach and methods to be used by the USAID Farmer-to-Farmer (F2F) Evaluation Team for Key Informant Interviews (KIIs) to be conducted among F2F stakeholders, in both the US and in eight countries where F2F operates. It was prepared for use by not only interviewers, but also for others involved with the KIIs, to better understand the approach (USAID, IPs), to maximize productivity for quality results. This Interview Guide for KIIs starts with what is common to all audiences, and explains the “why” and “how” that is important to understand for conducting the interviews. It also rolls up several “mini” interview guides from different respondents into a one-stop user-friendly document. This Guide includes these components: Rationale; Goals and Objectives, Consent and Interview Questions by Respondent Type. Rationale: Why Key Informant Interviews? The aim of the F2F Evaluation KIIs is to verify findings from a survey administered to 2,021 F2F participants in February and March 2017 and uncover what cannot be revealed through virtual methods. The Key Informant Interviews (KII) are one of multiple research methods used for this evaluation. The KII approach and questions build on emerging insights and learning thus far. The data collection plan and interview guides tie back into the three evaluation questions, and are based, in part, on a preliminary analysis of the survey data. In reviewing the survey data, the evaluation team focused on frequency results for individual survey questions in relation both to 174 three respondent groups individually (implementing partner, host organization, volunteer) and across these three respondent groups. The table below denotes the number of frequency tables that were reviewed per respondent group and across respondent groups. In a separate PDF file, all 83 frequency tables are presented. Respondent Group N = Number of Survey Questions Asked Number of Survey Frequency Tables Included in Survey Analysis Summary Implementing Partner 102 53 35 Host Organization 273 27 17 Volunteers 604 49 20 Across Respondents n/a n/a 11 Goals and Objectives (KIIs): Goal: The goal of Key Informant Interviews is to probe, clarify and mine first-hand, the knowledge and data from F2F Stakeholders; using questions informed primarily from the web-based survey but also include other sources of data. Our larger goal is to advance our holistic understanding of the F2F Program, in the framework of our Three Evaluation Questions (EQs), and to experience and learn first-hand what we would not have known without witnessing. Objectives: 1. Validate, verify, ground truth data from the survey, exploratory interviews, document reviews and desktop research. 2. Uncover new information not offered or able to be captured in the survey and other sources of information. 3. Gather research from those who could not respond to the survey, due to barriers such as language, education, access to Internet. Audience There are many stakeholders in the F2F knowledge and skills “value chain,” each with its own perspective working under the umbrella of a common set of goals and indicators. What cannot be derived are the nuances and processes involved with this “value chain,” and with each bringing his/her unique perspective on the bigger, shared, vision we begin to understand the Program more comprehensively. Our target audience of respondents is comprised of: 5. Implementing Partner (IP): These are the US-based organizations who with staff in countries served, manage individual F2F Country Projects. They are accountable for the performance of their respective country projects, from award management to recruitment and placement of volunteers, and liaise with hosts. 175 6. Host Organization: These are the organizations that are directly assisted by the program and receive face-to-face or hands-on technical assistance, training or advice from a F2F volunteer. They interface with the IP staff in country. 7. Volunteer: The person who is recruited by the IP in response to a request for assistance from a host. They embody a range of expertise areas, and provide face-to-face or hands-on direct technical assistance to the host. 8. USAID Country Mission: The USAID “branch” office based in each country that overseas USAID programs in the country, and liaises with USAID￾Washington and F2F Program in country. Sample Selection: The evaluation team will travel to 8 countries: Kenya, Malawi, Senegal, Bangladesh, Lebanon, Tajikistan Jamaica and Guatemala. In each of the 8 countries, we will attempt to interview 17 respondents per country. In country KII sample selection will be done through a combination of random selection, targeted, and convenience sample. This allows us to get a holistic and deeper knowledge of certain themes and still work efficiently. Our sample selection is strategically comprised of a cross section of geographies, IPs, sectors, and evaluation themes. The specific countries were chosen to ensure the largest number of IPs would be represented. The diverse selection enables the evaluation team to compare and contrast evaluation questions (EQs) across all countries. It is important to note that IPs differ in a number of significant ways: how long they have been implementing the F2F program, the size of the IP itself (staff size, company revenues); and the size and diversity of other IP in-country programs. (Most IPs are involved in additional programs outside of F2F). The “convenience sample” comes in when we look at who is available and able to host a team member, and ensuring that one IP doesn’t bear too much of the hosting burden. Random selection (not proposed by IP), combined with much targeted selection for specific reasons (e.g. respondent is a topical expertise, or plays a critical role, or has depth of knowledge on a specific EQs, such as use of innovation and focus on women. Randomization of host organizations will be done through an online list randomizer tool: https://www.random.org/lists/. However hosts found to be far from the center will likely be replaced by more accessible hosts to maximize the time resources available in country. Research Methodology/Process: KIIs will be conducted with a variety of different individuals from IPs, Host Organizations, volunteers, both in country and US based. Further, USAID Mission staff in the countries to be visited will also be interviewed. Individual interviews are anticipated to take 30 - 60 minutes, allowing the 17 interviews to be completed in five days in country. US-Based 176 IPs Initial arrangements will be made to establish the date and time for KII’s. If the IP is Washington based, they will be visited in person. If elsewhere they will be contacted by phone. Following a short introduction, the interviewer will proceed with the questions outlined below. Although the interview will follow the theme of the questions, the actual format will be by a free flow conversation. There will be time at the end for informal discussions and questions from the IPs. Volunteers Initial arrangements will be made to establish the date and time for KII’s. The volunteers will be contacted by phone. Following a short introduction, the interviewer will proceed with the questions outlined below. Although the interview will follow the theme of the questions, the actual format will be by a free flow conversation. There will be time at the end for informal discussions as well as questions from the volunteers. Global Prior to travel, the in-country IPs will notify USAID staff and arrange for an interview/in-brief. The host organizations to be interviewed will be agreed with the relevant IP. The selection of hosts will be done in the manner described above. The first day will be spent interviewing USAID and IP staff based at their HQ. Although the interview will follow the theme of the questions, the actual format will be by a free flow conversation. There will be time at the end for informal discussions as well as questions. The next three and half days will be spent in the field conducting interviews with the agreed upon host organizations. The final half day will be used for an out-brief with USAID Mission and the staff of the IPs (separately) to share findings from the field and discuss, probe and clarify. TENTATIVE DATA COLLECTION PLAN FOR EVALUATION TEAM DC SITE ET IP DC NUMBER OF INTERVIEWS TOTAL IP HO VOL USAID Senegal MM Winrock, ACDI/VOCA 5 days 4 10 1 2 17 Malawi MM CNFA 5 days 4 10 1 2 17 Kenya MM CRS 5 days 4 10 1 2 17 Tajikistan EJ ACDI/VOCA 5 days 4 10 1 2 17 Lebanon EJ Land O Lakes 5 days 4 10 1 2 17 Bangladesh EJ Winrock 5 days 4 10 1 2 17 177 Jamaica SP Partners 5 days 4 10 1 2 17 Guatemala SP VEGA/FAVAC A 5 days 4 10 1 2 17 USA ET All 5 days 8 n/a 5 3 16 TOTALS 9 45 40 80 13 19 152 Note: Not all DC Sites will have volunteers in-country Legend: DC = Data Collection; ET = Evaluation Team; VOL = Volunteer; IP = Implementing Partner; HO=Host Organization F2F Basic Theory of Change Input: Volunteer technical support with some additional staff support lead to learning through mentoring, workshops, and hands-on demonstrations. Strengthen the human capital. Output: Volunteer recommendations and training leading to improved efficiency in agricultural processes. Agribusiness with opportunities to be successful. Outcome: Adoption of recommendation or host change in behavior, management practice and/or technologies (innovation) leading to increase in profits and disposable income among the stakeholders. Impact: Productivity and income increases; improved social, human, and environmental capital. Three (3) Themes for USAID Farmer-to-Farmer Evaluation also known as “Evaluation Questions [hereafter referred to as EQ]) Below are the three major questions or themes (‘Big 3’) that frame the entire Evaluation and provide the overarching framework for the KIIs. Every question was designed to illuminate and advance our understanding of each EQ. 1. Evaluation Question (EQ1): How strong and effective is the F2F Program Theory of Change? 2. Evaluation Question (EQ2): How well do F2F standard indicators capture the effects of F2F activities (particular emphasis on indicators for outreach/cultural exchange, capacity development, environmental/natural resource management) and provide useful information for Program management? 3. Evaluation Question (EQ3): How well does the F2F Program learn and integrate innovation to encourage quality improvements and good practice in agricultural volunteer programs? 178 KEY INFORMANT INTERVIEW QUESTIONS BY RESPONDENT TYPE FOLLOWS ON THE NEXT PAGES. Interview Questions: Key Informant Interviews (KIIs) The following pages contain the set of interview questions by respondent type, followed by the standardized data collection forms we will use to record the data from our interviews. 1. Consent Form 2. Questions for Implementing Partners (IPs) 3. Questions for Host Organizations 4. Questions for Volunteers 5. Questions for USAID Country Missions 179 Farmer-to-Farmer Evaluation 2017 Respondent Consent Form Hello, my name is______________________. I have been contracted by USAID to conduct a study on the Farmer-to-Farmer (F2F) Program in which your organization participates. I am interviewing people like you in order to better understand the successes and challenges that organizations like yours and individuals like yourself have faced as part of the F2F Program. The information from the discussion will be used to assess needs of local partners and to improve the performance of the F2F Program in the future. Taking part in this interview is completely voluntary; you are free to choose to participate or not to participate. There is no risk to participation and your decision or refusal to participate will not affect your or your organization’s ability to participate in the F2F Program in the future. We expect the interview to last about forty-five minutes, and it will be anonymous; your name and organization will not be identified in any report, nor will you or your organization be associated with any response that is shared with the Farmer-to-Farmer Program. Participation in this effort by people like you is essential to the success of the study, so your time and cooperation is very much appreciated. If you have any questions, please contact the Team Leader on our project, Michelle F. Muldoon, at (+1) 571-344-4948; michelle@veritasdevelopment.biz. I agree to participate: ______________________________ Date: ______________________________ KII QUESTIONS: FOR IMPLEMENTING PARTNERS (IPs) [EQ1] Data from the survey suggest that you as IPs are generally quite satisfied with the Farmer to Farmer Program overall. Can you say a bit about what drives that satisfaction? 1. [EQ1] When you think about the specific technical expertise of your organization and the volunteers you recruit, do you think this aligns well with the assistance host organizations are requesting? Moving forward, what types of expertise will you be 180 looking for? What skill sets and attributes would like to see represented in volunteers hired for assignments? Do you see more roles for women and/or minority volunteers? 2. [EQ1] How has your IP and your volunteers managed cultural exchange issues? 3. [EQ2] What records does your organization keep on the data reported to F2F? How do you keep these records? 4. [EQ2] The survey data indicate that the greatest percent of respondents feel that area of potential environmental /natural resources impact is difficult to collect. Can you say a bit about your experience and what you think of this indicator and why is it hard to collect? 5. [EQ2] Our (non-survey) secondary research and exploratory interviews has revealed that indicators are used for multiple purposes besides reporting to USAID. How do indicators you collect drive your management plan? Which indicators that you collect are most important for the management plan? Which indicators that you collect do you feel do not enhance your management decisions? How is this data used for designing future programs? Is this data shared with volunteers after they have left? 6. [EQ2] The survey data indicate that the “Value of annual gross sales” and the “Annual net income” are both difficult to collect but are generally considered useful. What are the challenges with collecting each of these? How are these useful to your work? How do you propose F2F adjust these indicators to make them easier to collect? 7. [EQ3] The survey data suggest that F2F could improve learning to become more efficient and effective by listening to stakeholders’ perceptions and ideas and analyzing and discussing issues with stakeholders. How are stakeholders’ perceptions and ideas received by the Program currently? How can this communication be improved? Which stakeholders does the Program listen to? Which stakeholders are not able to share their perceptions/ideas with the Program? Why? 8. [EQ3] Data from the survey suggest that some implementing partners are dissatisfied with the level of collaboration and information sharing among F2F implementers. What prevents more collaboration and information sharing from taking place among F2F IPs? What information would you like to see shared between IPs? How should this collaboration take place? How should this information sharing take place? What happens when something goes wrong with a volunteer assignment? Are there established and effective ways to draw lessons from the negative experience? 181 9. [EQ1] The survey data suggest that logistical challenges are commonly faced by Implementing Partners. What types of logistical challenges have you faced as an Implementing Partner? What was the cause of these challenges? How can logistics be improved for IPs? Are these challenges mostly related to your own logistics or those of others (e.g. training participants)? If the latter, what types of challenges are faced and how can these be improved? How have logistical constraints impacted your ability to implement new practices in the past? 10. [EQ1] The survey data suggest that in-country F2F staff and Country Directors sometimes have challenges with language, poor communications from hosts and poor support from hosts. Have you experienced these challenges? When are these challenges the greatest? Have these challenges gotten better or worse over time? Probe: If better, what has been done to make these improvements? 11. [EQ1] For non-SPSP IPs: F2F has a program focused on special projects Based on the survey data, many IP staff are not familiar with the SPSP program. What is your impression of the SPSP program? How has the SPSP program impacted your work with F2F? 12. [EQ1] For SPSP IPs: What is the mission of the SPSP program, as you understand it? How does the SPSP program accomplish this mission? What other ways—outside of the SPSP program—could F2F use to accomplish this mission? What do you think can be done to improve the SPSP program? 182 KII QUESTIONS: FOR HOST ORGANIZATIONS 1. [EQ1] Have you participated/benefited from any other USAID programs or projects? 2. [EQ1]What has been your experience with how volunteers are selected and matched to the scope of work? What are some of the issues you’ve had with volunteers not completing their deliverables? 3. [EQ1] Our exploratory research revealed that development of scopes of work can be challenging. Are you confident that the scope of work is clear and stands a good chance of success? Do you ever feel unsure about what the outcome will be? Do you feel that volunteers do well in terms of being able to tailor training to specific farmers and to staff within your organization? 4. [EQ1] The survey data suggest that logistical challenges are commonly faced by host organizations. What types of logistical challenges have you faced as a host organization? What was the cause of these challenges? How can logistics be improved for host organizations? Besides logistics, what other challenges have you faced? 5. [EQ1] The survey data suggest that language barriers are common challenges faced by volunteers. What types of language challenges have you faced as a host organization? What was the cause of these challenges? How can these be improved? 6. [EQ1] Our exploratory interviews revealed that sometimes in order to implement the volunteer’s recommendations; the host organization will need to spend some money. This could be to hold a training or to purchase some equipment or some other cost that is necessary to implement the recommendation. How has your organization approached such recommendations in the past? Has your organization ever borrowed money to implement a recommendation? Please explain. 7. [EQ1] In your experience, what types of F2F projects include considerations of how potential benefits might be distributed differently between men and women? And how do they include gender considerations? 8. [EQ2] The F2F Program is interested in how you collect and report on required indicators. Please tell me about your reporting processes and how you report on indicators. What tools for data and knowledge management do you use? What is missing? What would make your work easier (besides staff and time and money)? Do you feel these indicators capture the impact of the F2F program? 9. [EQ2] The Farmer to Farmer program is interested in the relationship between farming 183 practices and environmental conservation. The survey data indicate a low percent of the recommendations from volunteers’ concern environmental conservation. Do you have any thoughts about why this is the case? Do you feel the volunteers are not knowledgeable about environmental conservation? 184 KII QUESTIONS: FOR VOLUNTEERS 1. [EQ1] The survey data suggest that logistical challenges are commonly faced by volunteers. What types of logistical challenges have you faced as a volunteer? What was the cause of these challenges and was the challenge resolved? 2. [EQ1] The survey data suggest that language barriers are common challenges faced by volunteers. Did you experience this and how you mitigate these challenges? 3. [EQ1] The survey data suggest that sometimes volunteers don’t know if the host organization is making changes based the suggestions of the volunteer? Can you say a bit about if this has been your experience? And to what extent, in your experience, that you come to know about what host organizations do with the recommendations you provide. 4. [EQ1] For volunteers who volunteered more than once with F2F: Despite a number of volunteers citing logistical challenges, the general trend is that logistical challenges are becoming less common. What improvements have you noticed from your older assignments to your more recent ones that might explain this? Have you been traveling less because of your assignments, requiring fewer logistics? Or have the logistical operations improved? If the latter, how have they improved? 5. [EQ1] In your experience, what types of F2F projects include considerations of how potential benefits might be distributed differently between men and women? What types of projects do not require these considerations? 6. [EQ1] When you think about the technical support you received and your experience overall, do you feel confident that both the host organization and the farmers you are working with can be effective drivers of change? 7. [EQ3] Can you say a bit about what happens after your assignment is complete? 8. [EQ3] Would you volunteer again? And if so, how would you integrate any lessons learned into a new assignment? 185 KII QUESTIONS: FOR USAID MISSION STAFF 1. [EQ1] Given that F2F is a Washington DC program, do you have any management responsibility? Are you able to integrate F2F activities into other EG/AG projects? In what ways? How can synergy be enhanced? 2. [EQ2] Does the F2F program get reported in the Mission’s annual report or other do? If so, do you use the standard F2F indicators or do they feed into other indicators used by the Mission? If you use F2F indicators, do you feel that they are comprehensive and accurately capture the impact of the F2F program? 3. [EQ3] Are lessons learned and/or innovations introduced by the F2F program integrated into other USAID Mission projects? 4. [EQ1] In what ways is the F2F Program contributing to your Feed the Future and Country Development Cooperation Strategy (CDCS)? 186 ANNEX E: TALLY SHEETS BY COUNTRY 187 FARMER TO FARMER PERFORMANCE EVALUATION DATA COLLECTION BY COUNTRY Code Type Date M F Total KENYA 1 KY-IP-INT-1 IP April 17, 2017 1 0 1 2 KY-IP-INT-2 IP April 17, 2017 0 1 1 3 KY-IP-INT-3 IP full week 1 0 1 4 KY-IP-INT-4 IP April 22, 2017 1 0 1 5 KY-IP-INT-5 IP full week 1 0 1 6 KY - HO-FGD-1 HO April 18, 2017 0 3 3 7 KY - GV-INT-1 HO April 18, 2017 0 1 1 8 KY-HO-INT-1 HO April 18,2017 1 0 1 9 KY-HO-INT-2 HO April 19, 2017 0 1 1 10 KY-HO-INT-3 HO April 19, 2017 1 0 1 11 KY-USAID-INT-1 US April 20, 2017 1 0 1 12 KY-HO-INT-4 HO April 20, 2017 1 0 1 13 KY-HO-INT-5 HO April 20, 2017 1 0 1 14 KY-HO-FGD-7 HO April 21, 2017 3 1 4 15 KY-HO-INT-6 HO April 21, 2017 1 0 1 16 KY-HO-INT-7 HO April 21, 2017 0 1 1 17 KY-HO-INT-8 HO April 21, 2017 0 1 1 18 KY-HO-INT-9 HO April 22, 2017 1 0 1 Sub-Totals (Kenya ) 18 14 9 23 MALAWI 1 MW-IP-INT-1 IP April 24, 2017 1 0 1 2 MW-IP-INT-2 IP April 24, 2017 0 1 1 3 MW-HO-INT-1 HO April 24, 2017 0 1 1 188 4 MW -USAID -FGD -1 US April 24, 2017 0 2 2 5 MW -HO -FGD -1 HO April 25, 2017 0 11 11 6 MW -HO -INT - 2 HO April 26, 2017 0 1 1 7 MW-HO-INT-3 HO April 26, 2017 1 0 1 8 MW-HO-INT-4 HO April 26, 2017 1 0 1 9 MW-HO-INT-5 HO April 26, 2017 1 0 1 10 MW -HO -INT - 6 HO April 26, 2017 1 0 1 11 MW -HO -INT - 7 HO April 27, 2017 1 0 1 12 MW -HO -INT - 8 HO April 27, 2017 0 1 1 13 MW -HO -INT - 9 HO April 27, 2017 0 1 1 14 MW -HO -INT -10 HO April 28, 2017 1 0 1 Sub -Totals (Malawi) 14 7 18 25 SENEGAL 1 SN -IP1 -INT -1 IP May 29, 2017 1 0 1 2 SN-IP1-INT-2 IP all week 0 1 1 3 SN-HO1-INT-1 HO May 29, 2017 2 0 2 4 SN-HO1-FGD-1 HO May 29, 2017 2 6 8 5 SN-IP2-INT-1 IP May 30-31, 2017 1 0 1 6 SN-HO2-INT-1 IP May 30, 2017 0 3 3 7 SN-HO2-INT-2 HO May 30, 2017 1 0 1 8 SN-HO2-INT-3 HO May 31, 2017 0 1 1 9 SN-HO2-INT-4 HO May 31, 2017 3 0 3 10 SN -HO2 -INT - 5 HO May 31, 2017 1 0 1 11 SN -USG -INT -1 USG May 31, 2017 1 1 2 12 SN -IP1 -INT - 3 IP June 1, 2017 1 0 1 13 SN -IP1 -INT - 4 IP June 1, 2017 1 0 1 14 SN -USG -INT - 2 USG June 1, 2017 1 0 1 15 SN -HO1 -INT - 2 HO June 2, 2017 3 0 3 16 SN -HO1 -INT - 3 HO June 5, 2017 1 0 1 17 SN -IP1 -INT - 5 IP June 6, 2017 1 0 1 18 SN -HO1 -FGD - 2 HO June 7, 2017 5 4 9 19 SN -HO1 -FGD - 3 HO June 7, 2017 2 2 4 189 Sub -Totals (Senegal) 19 27 18 45 TAGIKISTAN1 TG -IP -INT -1 IP April 24, 2017 1 0 1 2 TG-IP-INT-2 IP April 24, 2017 1 0 1 3 TG-USAID-INT-1 US April 24, 2017 2 0 2 4 TG-HO-INT-1 HO April 24, 2017 1 0 1 5 TG-HO-INT-2 HO April 25, 2017 1 0 1 6 TG-HO-INT-3 HO April 25, 2017 0 1 1 7 TG-HO-INT-4 HO April 25, 2017 1 0 1 8 TG-HO-INT-5 HO April 25,2017 1 0 1 9 TG-HO-INT-6 HO April 25, 2017 1 0 1 10 TG -HO -INT - 7 HO April 25, 2017 1 0 1 11 TG -HO -INT - 8 HO April 26, 2017 1 0 1 12 TG -HO -INT - 9 HO April 26, 2017 1 0 1 13 TG -HO -INT -10 HO April 26, 2017 1 0 1 14 TG -HO -INT -11 HO April 26, 2017 0 1 1 15 TG -HO -INT -12 HO April 26, 2017 0 8 8 16 TG -HO -INT -13 HO April 26, 2017 1 0 1 17 TG -VOL -INT -1 VOL April 27, 2017 1 0 1 18 TG -HO -INT -14 HO April 27, 2017 1 0 1 19 TG -HO -INT -15 HO April 27, 2017 1 0 1 20 TG -HO -INT -16 HO April 27, 2017 1 0 1 21 TG -HO -INT -17 HO April 27, 2017 1 0 1 22 TG -HO -INT -18 HO April 27, 2017 1 0 1 23 TG -HO -INT -19 HO April 27, 2017 1 0 1 24 TG -HO -INT -20 HO April 28, 2017 1 0 1 25 TG -HO -INT -21 HO April 28, 2017 0 1 1 26 TG -IP -INT - 3 HO April 28, 2017 0 1 1 Sub -Totals (Tajikistan) 26 22 12 34 BANGLADESH1 BD -IP -INT -1 IP April 30, 2017 3 0 3 2 BD -USAID -1 US April 30, 2017 1 0 1 190 3 BD -HO -INT -1 HO April 30, 2017 4 3 7 4 BD -HO -INT - 2 US May 1, 2017 2 30 32 5 BD -HO -INT - 4 HO May 1, 2017 0 2 2 6 BD -HO -INT - 5 HO May 1, 2017 1 10 11 7 BD -HO -INT - 6 HO May 2, 2017 1 0 1 8 BD-HO-INT-7 HO May 2, 2017 1 0 1 9 Bd-GO-INT-1 GO May 2, 2017 0 2 2 10 BD -HO -INT - 8 HO May 2, 2017 1 0 1 11 BD -HO -INT - 9 HO May 2, 2017 1 0 1 12 BD -HO -INT -10 HO May 3, 2017 1 0 1 13 BD -HO -INT -11 HO May 3, 2017 1 0 1 14 BD -HO -INT -12 HO May 3, 2017 9 1 10 15 BD -HO -INT -13 HO May 4, 2017 10 1 11 16 BD -HO -INT -14 HO May 4, 2017 10 5 15 17 BD -HO -INT -15 HO May 5, 2017 4 0 4 18 BD -HO -INT -16 HO May 6, 2017 4 0 4 Sub -Totals (Bangladesh) 18 50 54 104 LEBANON 1 LB -IP -INT -1 IP May 8, 2017 1 0 1 2 LB-IP-INT-2 HO May 13, 2017 0 1 1 3 LB-USAID-1 US May 8, 2017 1 0 1 4 LB-HO-INT-1 HO May 8, 2017 0 1 1 5 LB-HO-INT-2 HO May 8, 2017 2 0 2 6 LB-GO-INT-1 GO May 9, 2017 2 0 2 7 LB-HO-INT-3 HO May 9, 2017 1 0 1 8 LB-HO-INT-4 HO May 9, 2017 2 0 2 9 LB-HO-INT-5 HO May 9, 2017 1 0 1 10 LB -HO -INT - 6 HO May 9, 2017 1 1 2 11 LB -HO -INT - 7 HO May 10, 2017 1 1 2 12 LB -HO -INT - 8 HO May 10, 2017 1 0 1 13 LB -HO -INT - 9 HO May 10, 2017 0 1 1 14 LB -GO -INT - 2 GO May 11, 2017 2 0 2 191 15 LB -HO -INT -10 HO May 11, 2017 2 0 2 16 LB -HO -INT -11 HO May 11, 2017 2 0 2 17 LB -HO -INT -12 HO May 11, 2017 0 1 1 18 LB -HO -INT -13 HO May 12, 2017 2 0 2 19 LB -HO -INT -14 HO May 12, 2017 1 0 1 20 LB -HO -INT -15 HO May 13, 2017 1 0 1 Sub -Totals (Lebanon) 20 22 6 28 GUATEMALA1 GT -HO -INT -1 HO May 2, 2017 2 0 2 2 GT -HO -INT - 2 HO May 2, 2017 1 1 2 3 GT -HO -INT - 3 HO May 2, 2017 0 1 1 4 GT-HO-INT-4 HO May 2, 2017 1 0 1 5 GT-HO-INT-5 HO May 3, 2017 2 0 2 6 GT-HO-INT-6 HO May 3, 2017 2 0 2 7 GT-HO-INT-7 HO May 4, 2017 1 1 2 8 GT-HO-INT-8 HO May 5, 2017 1 0 1 9 GT-VOL-INT-1 VOL May 4, 2017 0 1 1 10 GT -IP -INT -1 IP full week 1 0 1 11 GT -IP -INT - 2 IP May 6, 2017 2 2 4 12 GT -USAID -INT -1 USAID May 19, 2017 0 2 2 Sub -Totals (Guatamala) 12 13 8 21 JAMAICA 1 JA -HO -INT -1 HO May 8, 2017 0 2 2 2 JA -HO -INT - 2 HO May 8, 2017 1 0 1 3 JA-HO-INT-3 HO May 9, 2017 1 0 1 4 JA-HO-INT-4 HO May 9, 2017 1 0 1 5 JA-HO-INT-5 HO May 9, 2017 2 0 2 6 JA-HO-INT-6 HO May 10, 2017 0 1 1 7 JA-HO-INT-7 HO May 10, 2017 1 0 1 8 JA-HO-INT-8 HO May 10, 2017 0 0 0 9 JA-HO-INT-9 HO May 10, 2017 1 0 1 10 JA -HO -INT -10 HO May 11, 2017 1 0 1 192 11 JA-HO-INT-11 HO May 11, 2017 1 0 1 12 JA-HO-INT-12 HO May 11, 2017 0 1 1 13 JA-HO-INT-13 HO May 11, 2017 0 1 1 14 JA-HO-INT-14 HO May 11, 2017 1 0 1 15 JA-IP-INT-1 IP full week 0 1 1 16 JA-IP-INT-2 IP May 9, 2017 0 1 1 17 JA-IP-INT-3 IP May 9, 2017 0 1 1 18 JA-IP-INT-4 IP May 9, 2017 0 1 1 19 JA-IP-INT-5 IP May 9, 2017 0 1 1 Sub-Totals (Jamaica) 19 10 10 20 USA 1 US-IP-INT-1 IP May 22, 2017 1 0 1 2 US-IP-INT-2 IP June 1, 2017 0 1 1 3 US-IP-INT-3 IP June 7, 2017 0 2 2 4 US-VOL-INT-1 VOL June 20, 2017 1 0 1 0 Sub-Totals (USA) 4 2 3 5 GRAND TOTAL 150 167 138 305 NOTE: A combined total of 146 FGD and KIIs conducted overseas with 4 formal KIIs conducted in U.S. Only interviews using the standard KII Interview Guide are counted here. Exploratory interviews and follow up discussions of a less formal nature are not included. 193 ANNEX F: VOLUNTEER SURVEY RESULTS 2 1 3 33 57 2 1 The recruitment process was clear and well-organized. (Volunteer) N=573 Strongly disagree Disagree Neither disagree nor agree Agree Strongly agree Not applicable Don't know 2 0 1 19 77 1 0 I was matched with a project that fit my skills. (Volunteer) N=574 Strongly disagree Disagree Neither disagree nor agree Agree Strongly agree Not applicable Don't know 194 2 1 3 20 67 6 1 My preferences were taken into account when matching me with a project. (Volunteer) N=574 Strongly disagree Disagree Neither disagree nor agree Agree Strongly agree Not applicable Don't know 22 4 21 71 0 0 I was strongly motivated for the full duration of my assignment. (Volunteer) N=574 Strongly disagree Disagree Neither disagree nor agree Agree Strongly agree Not applicable Don't know 195 3 5 5 55 32 1 0 I received all of the information I requested before beginning my assignment. (Volunteer) N=573 Strongly disagree Disagree Neither disagree nor agree Agree Strongly agree Not applicable Don't know 2 6 6 35 52 0 0 I felt well-prepared before beginning my assignment. (Volunteer) N=573 Strongly disagree Disagree Neither disagree nor agree Agree Strongly agree Not applicable Don't know 196 2 5 10 40 40 3 1 I received all the technical support that I needed to succeed in my duties. (Technical support includes information, links to local technology programs, teaching aids, and equipment.) (Volunteer) N=565 Strongly disagree Disagree Neither disagree nor agree Agree Strongly agree Not applicable Don't know 197 2 3 3 26 65 1 0 I received all the logistical support that I needed to succeed in my duties. (Logistical support includes transport, housing, translators, compensation arrangements, and other services.) (Volunteer) N=565 Strongly disagree Disagree Neither disagree nor agree Agree Strongly agree Not applicable Don't know 1 0 2 23 71 2 0 I made suggestions to the host organization on how to improve the activities related to my direct assignment. (Volunteer) N=573 Strongly disagree Disagree Neither disagree nor agree Agree Strongly agree Not applicable Don't know 198 1 4 7 40 35 1 11 The host activity that I supported was significantly improved. (Volunteer) N=573 Strongly disagree Disagree Neither disagree nor agree Agree Strongly agree Not applicable Don't know 1 3 13 29 15 2 38 The host organization made significant changes based on the suggestions I made. (Volunteer) N=564 Strongly disagree Disagree Neither disagree nor agree Agree Strongly agree Not applicable Don't know 199 2 1 3 24 67 1 2 My opinions were valued and respected by my host organization. (Volunteer) N=574 Strongly disagree Disagree Neither disagree nor agree Agree Strongly agree Not applicable Don't know 2 4 11 43 37 1 2 In my opinion, the host organization is well-equipped to learn from their projects over time. (Volunteer) N=565 Strongly disagree Disagree Neither disagree nor agree Agree Strongly agree Not applicable Don't know 200 201 1 1 7 20 71 How likely are you to volunteer again with F2F? (Volunteer) N=531 Not at all likely Somewhat unlikely Unsure Somewhat likely Very likely 202 Annex G: USAID Response to F2F Evaluation USAID commends the PEEL F2F Program Evaluation team its assessment of the Farmer-to￾Farmer (F2F) Program based on development of a thorough understanding of the Program objectives, structure, operating principles, issues and constraints. The Team has focused well on the key evaluation questions and has produced a report with useful findings and conclusions that should be useful to guide the current and future projects. The findings and recommendations provide a rich base of ideas for improvements to on-going and future F2F volunteer projects. However, many of the recommendations would tend to increase costs for the program and spread activity beyond a sole focus on the volunteer assignments. There is virtue in many of the recommendations, but USAID and implementing partners will have to consider and balance these against the cost increases and program creep that may have undesirable effects. We appreciate the opportunity to expand volunteer impacts by strengthening country F2F program staffing and leveraging volunteer contributions through expanded country level networking and partnerships. This can build on the demand-driven nature of the volunteer assistance responding to needs and opportunities identified by and owned by the local host organizations with which the volunteers work. We agree with the importance of increasing consistency throughout the program by sharing good practices and encouraging their adoption among implementers. USAID already provides regular training on indicators and data collection to implementing partners, in a format that’s accessible to the field and facilitates extensive collaboration and sharing of experience. Creating a standard model or enforcing specific implementation practices could diminish the F2F program flexibility, ability to build on different implementing partners’ unique capabilities, and ability of implementers and field staff to respond most effectively to specific host and volunteer needs and in-country circumstances. The evaluation recommends strengthening the statement of the Program's strategy and theory of change. Country F2F Project descriptions are intended to do this at the country level and are not highlighted in the report, but may be revisited to ensure clarity and a comprehensive systems approach. Survey findings are useful, though they may mask differences between core program operations and the special projects' activities. USAID will undertake further assessment of possible differences in this area. USAID will convene a meeting of implementing partners to review findings and adapt current program activities to the extent possible and will integrate findings into future programs as appropriate.