November 2015 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Rees Warne and Jim Dempsey of Social Impact. evaluationSeptember 2015 This publication was produced at the request of the United States Agency for International Development. EVALUATION Feed the Future Partnering For Innovation Midterm Performance Evaluation Report EVALUATION Feed the Future Partnering For Innovation Midterm Performance Evaluation Report Feed the Future Partnering for Innovation midterm performance Evaluation Report November 2015 AID-OAA-I-10-00013/AID-OAA-TO-14-00021 DISCLAIMER: The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Cover Page Image Source: AgTechXChange website: http://www.partneringforinnovation.org/program-updates.aspx Cover Page Image Source: AgTechXChange website: http://www.partneringforinnovation.org/program-updates.aspx P4I Mid-term Performance Evaluation Report | November 2015 i CONTENTS Contents ...........................................................................................................................................................................................i Acronyms....................................................................................................................................................................................... iii Acknowledgements ...................................................................................................................................................................... v Executive Summary ......................................................................................................................................................................vi Evaluation Purpose & Evaluation Questions .......................................................................................................................... 1 Project Background ...................................................................................................................................................................... 2 Evaluation Methods and Limitations ........................................................................................................................................ 5 Findings, Conclusions, & Recommendations ......................................................................................................................... 9 Evaluation Question 1 ................................................................................................................................................................. 9 Evaluation Question 2 ...............................................................................................................................................................17 Evaluation Question 3 ...............................................................................................................................................................31 Evaluation Question 4 ...............................................................................................................................................................43 Annexes ........................................................................................................................................................................................47 Annex I: Evaluation Statement of Work ...............................................................................................................................47 Annex II: Evaluation Methods and Limitations ....................................................................................................................49 Anex III: P4I Component 1 and 2 Sub-Grants………………………………………………………………….59 Annex IV: Summary of Evaluation Question 1 Strengths and Weaknesses ................................................................62 Annex V: Data Collection Instruments ................................................................................................................................. 66 Annex VI: Sources of Information ..........................................................................................................................................78 Annex VII: P4I Logical Framework .........................................................................................................................................80 Annex VIII: P4I Indicators and Performance ........................................................................................................................82 Annex IX: P4I Theory of Change ...........................................................................................................................................83 Annex X: Disclosure of Conflict of Interest........................................................................................................................84 Annex XI: Fintrac Statement of Differences……………………………………………………………………88 Annex XII: Social Impact Response to Fintrac Statement of Differences…………………………......................90 P4I Mid-term Performance Evaluation Report | November 2015 ii LIST OF TABLES Table 1: P4I Sub-Grants for Component 1 in Kenya and Component 2 in Mozambique…..…4 Table 2: Time between proposal submission and sub-grant signing………………………......18 Table 3: Component 1 Sub-grant in Kenya: Milestone Progress and Modifications………......23 Table 4: Component 1 Sub-grant in Kenya; Evaluation Questions 2A, 2B, and 2C……...….. 24 Table 5: Component 2 Sub-grants in Mozambique: Milestone Progress and Modifications.....26 Table 6: P4I Monitoring and Evaluation Plan Indicators and their Data……………...……......31 Table 7: Additional Indicators for Mozambique Component 2………………...…………......32 Table 8: Post-Award Reporting and Data Collection (from the P4I M&E Plan)……….……..35 P4I Mid-term Performance Evaluation Report | November 2015 iii ACRONYMS AATF African Agricultural Technology Foundation/Cooperative League of the United States of America ADS Automated Directives System ABEO Agriculture, Business and Environment Office AO Agreement Officer AOR Agreement Officer's Representative BOM Banco Oportunidade de Mozambique BFS Bureau for Food Security CIMMYT International Maize and Wheat Improvement Centre COI Conflict of Interest CSO Civil Society Organization DQA Data Quality Assessment EMCL Export Marketing Company LDA ETG Export Trading Group FOG Fixed Obligation Grant FTF Feed the Future GDA Global Development Alliances ICT Information Communication Technologies iDE International Development Enterprises IQC Indefinite Quantity Contract IP Implementing Partner IR Imazapyr-Resistant KAVES Kenya Agricultural Value Chains Enterprises KFIE Kenya Feed the Future Innovation Engine KII Key Informant Interview LOE Level of Effort M&E Monitoring & Evaluation P4I Mid-term Performance Evaluation Report | November 2015 iv MoV Means of Verification MPI Office of Markets Partnerships and Innovations MSME Micro, Small, and Medium Enterprises N/A Not Applicable NCBA/CLUSA National Cooperative Business Association/Cooperative League of the USA NGO Non-Governmental Organization OI Opportunity International P4I Partnering for Innovation PE Performance Evaluation PICS Purdue Improved Crop Storage PIRS Performance Indicator Reference Sheets PMP Performance Monitoring Plan PPP Public Private Partnership PPR Performance Plan and Report RFTOP Request for Task Order Proposals SI Social Impact SMOG Standard Milestone Obligation Grant SNAP Solicitation for New Agricultural Partnerships SOW Statement of Work TAP Transparency, Accountability and Performance TO Task Order TBD To Be Determined USAID United States Agency for International Development USG United States Government ZOI Zones of Influence P4I Mid-term Performance Evaluation Report | November 2015 v ACKNOWLEDGEMENTS The P4I Midterm Performance Evaluation Team wishes to acknowledge the cooperation of the staff at Fintrac who worked collaboratively with us to provide the initial insights and access to the data required to complete this mid-term evaluation report. In addition, the team thanks the many stakeholders in Mozambique and Kenya who welcomed us into their business places and generously shared their time and experiences with us over the course of our fieldwork. USAID Mission staff in Kenya and Mozambique generously gave their time and insights. Finally, we extend our thanks and gratitude to USAID for its guidance and support throughout the work. P4I Mid-term Performance Evaluation Report | November 2015 vi EXECUTIVE SUMMARY EVALUATION PURPOSE AND EVALUATION QUESTIONS Within the USAID Bureau for Food Security (BFS), the Office of Markets, Partnerships and Innovations (MPI) is charged with the mandate to promote new approaches to food security through innovative partnerships that improve market access for food insecure households in Feed the Future (FTF) countries. The Partnering for Innovation activity (P4I) is intended to provide BFS/MPI a platform to fulfill this role. USAID/BFS contracted Social Impact (SI) to undertake a midterm performance evaluation of P4I. This evaluation was carried out as part of the BFS Performance Evaluations Task Order held by SI under the Transparency, Accountability and Performance Indefinite Quantity Contract. The evaluation was designed to provide empirical evidence to support learning and continuous improvement for BFS’ work, and also assess implementation, and provide information and recommendations that BFS and P4I can use to improve activity effectiveness and better achieve intended outcomes. The evaluation covers the two primary P4I components: Component 1 Agricultural Technology Commercialization and Component 2 Partnership Development1 (all four components are listed below). The evaluation addresses four evaluation questions: 1. P4I as USAID interface with private sector: What are the strengths and weaknesses of the ways in which P4I works with the private sector (through Components 1 and 2) as a means to increase USAID’s private sector engagement in Mozambique and Kenya? 2. Effects of the SMOG2 funding model from the perspective of partners: What are the effects of the structure of the SMOG funding model on private sector firms’ ability to scale up smallholder access to their technologies? 3. Assessing Benefits to Smallholders: How can P4I improve its ability to assess whether P4I Component 1 and 2 grants and partner investments are increasing smallholder men and women farmers’ access to and use of targeted technologies? 1 Component 1: Agricultural Technology Commercialization: The purpose of this component is to identify and bring to scale promising “bottom of the pyramid” agricultural technologies that are ready for commercialization. This component is a core focus of P4I as it is expected to enable increased access to innovative technologies by smallholder farmers, women in particular. Component 2: Partnership Development: Component 2 provides a platform for developing 10-20 new, innovative partnerships between USAID missions and the private sector, with financial and technical support provided through P4I. It includes partnership management and development through outreach and communication with private sector partners to channel commercialization toward FTF priorities. It includes 1) support for relationship management; 2) support for partnership management; 3) partnership start-up funding; and 4) promotion of mission buy-in to partnerships. 2 “SMOG” is the acronym for “Standard Milestone Obligation Grant,” a grant mechanism designed through P4I. The SMOGs include information on the milestones sub-grantees must achieve, the means of verification for each milestone, the milestone due date, and the amount to be paid to the sub-grantee once each milestone is finished. P4I Mid-term Performance Evaluation Report | November 2015 vii 4. Management: How might management of P4I (by Fintrac, BFS, and/or Missions) be improved to maximize P4I’s outputs and outcomes? PROJECT BACKGROUND The Feed the Future P4I Program, awarded in September 2012 to Fintrac, is a five-year initiative to engage the private sector to meet the global food security challenges. The P4I goal is to find new approaches to food security through innovative partnerships, technologies, and methods that improve food security and nutrition for food insecure households in FTF countries. P4I has four components; Components 1 and 2 are the main components of this activity (based on amount of time and resources dedicated to them), and USAID’s request for this evaluation covers just those two components. Component 1: Agricultural Technology Commercialization: Identify and bring to scale promising bottom-of-the-pyramid agricultural technologies that are ready for commercialization. Component 2: Partnership Development: Provide a platform for developing innovative partnerships between USAID missions and the private sector financial and technical support through P4I. Component 3: Investment Design and Modeling: Provide modeling and analysis to support the development and establishment of innovative partnerships that will benefit smallholder agriculture. Component 4: Knowledge Management: Develop a central source of information and analysis on private sector engagement models in agriculture and food security for the Feed the Future Initiative. EVALUATION DESIGN, METHODS, AND LIMITATIONS The evaluation team applied a mixed methods approach to data collection and analysis. The team used robust qualitative techniques including key informant interviews, limited numbers of focus groups, and observations to elicit information from a variety of P4I stakeholders, including representatives from USAID/BFS, Missions, Fintrac, sub-grant recipients and their second-level sub-partners, retailers selling the targeted technologies, and farmers. This allowed for triangulating information on P4I processes and progress from multiple perspectives and from multiple stakeholders. The evaluation team also reviewed qualitative and quantitative secondary data from BFS, the Missions, P4I and sub-grantees (including monitoring records, etc.). Qualitative data analysis focused on identifying emergent themes and patterns related to each evaluation question and drawing conclusions verified by the data. Data collection was carried out between April 16 and July 22, 2015. Much of the primary qualitative data and some of the secondary data was collected during field work in Mozambique (May 18 – May 26) and Kenya (May 27 – June 5). There are some limitations to the evaluation design. Carrying out the evaluation work in just one country each for Components 1 and 2 limited the ability to generalize conclusions. Due to the limited field time (three weeks total for two countries and nine sub-grants) and the distances between implementation sites, the findings, although robust, do not provide a comprehensive representation of P4I’s work under Components 1 and 2. The evaluation does not cover Components 3 and 4. P4I Mid-term Performance Evaluation Report | November 2015 viii EVALUATION QUESTION 1: What are the Strengths and Weaknesses of the ways in which P4I Works with the Private Sector (through Components 1 and 2) as a means to Increase USAID’s Private Sector Engagement in Mozambique and Kenya? Findings Differing Mission Needs and Experiences: USAID/Mozambique staff appreciated P4I as a transitional FTF funding mechanism. It allowed the Mission to relatively quickly fund FTF programming and report on FTF indicators without investing substantive staff time or budget. In Kenya, Mission staff perceived less need for the Component 1-supported activities, and, while P4I attempted to engage Mission staff, there has been little coordination or cooperation between P4I-supported sub-grants and USAID/Kenya staff. Private Sector Engagement: Of the nine sub-grants covered in this evaluation of an activity aimed at partnering with the private sector, five (three in Mozambique and two in Kenya) were led by non-profit organizations (with private sector sub-partners). USAID/Mozambique staff participated in calls with sub￾grantees and provided limited support. Sub-grants in Kenya operated with almost no Mission engagement. Management Complexity & Roles and Responsibilities: While P4I staff are strong and responsive communicators, there are numerous stakeholders in the P4I assistance and communication process, particularly with non-profit sub-grantees and those based outside the country of operations. P4I is a highly complex globally dispersed activity covering a broad range of value chains. Staff do not (and cannot be expected to) hold all subject matter and country context expertise for effective management of all possible sub-awards. Conclusions Mission Needs: Different Missions have different perceptions of the utility and value of P4I Components 1 and 2 as a means of increasing their FTF engagement with the private sector. This appears to depend in part on the Mission’s current FTF programming. Private Sector Engagement: While strong non-profits can be an important support to weaker private firms or those lacking experience with smallholder farmers (and sometimes a necessary one for initial funding to a weak private sector firm), using P4I and non-profits as intermediaries between USAID and the private sector firms can weaken USAID engagement with the private sector. Management Complexity & Roles and Responsibilities: There is room for improving communication flows among P4I, USAID, and sub-grantees and their second-level sub-partners. P4I staff spend relatively little time with sub-grantees or second-level sub-partner. This was partially mitigated by staff responsiveness and good communication. The AOR was said to be responsive and helpful. Recommendations Mission Needs: Improve Component 1 coordination and integration with Mission FTF programs, including a strong point of contact or Activity Manager, substantive early involvement in solicitation design, selection, an action plan, and opportunities for engagement with sub-grantees. Private Sector Engagement: Increase the role of private sector firms sub-partnering to non-profit sub-grantees. Create role options for non-profits more appropriate for a private sector engagement activity while being flexible to accommodate private sector needs. P4I Mid-term Performance Evaluation Report | November 2015 ix Management Complexity & Roles and Responsibilities: Make it easier for sub-grantees to communicate with USAID staff and for Mission staff to communicate with and otherwise engage with sub-grantees and their second-level sub-partners. Assign a P4I point of contact in FTF Missions. Bolster USAID/BFS support by sharing the AOR workload with other staff with complementary expertise. EVALUATION QUESTION 2: What are the Effects of the Structure of the SMOG Funding Model on Private Sector Firms’ ability to Scale up Smallholder Access to their Technologies? Findings Utility of the SMOG Model: All P4I stakeholders supported the concept of performance-based award mechanisms. SMOG milestones were problematic and stressful for many sub-grantees, particularly where they had no control over needed inputs or faced high production risks. Challenging SMOG Negotiation Process & Means of Verification: SMOG negotiations were judged as difficult at times by private sector firms and non-profits. Many were unclear on how their milestones would work until late in the process. The means of verification (MoV) for some milestones were seen by some as more of a burden than the milestones themselves. Required data on technology buyers is difficult for some sub-grantees to get. Some SMOGs require more detailed data than others. Milestone Renegotiation & Failure to Meet Milestones: Five of the nine sub-grants received SMOG modifications - a total of 11 modifications were made to those five SMOGs, with more than 80 modifications made to milestones. While P4I willingness to make modifications is a strength; many modifications could have been avoided through better initial design. There was widespread lack of clarity on repercussions for missing milestone targets or for incomplete reporting of required data. Conclusions Utility of the SMOG Model: While all P4I stakeholders supported the concept of performance-based award mechanisms, the SMOG model has both strengths and weaknesses. High levels of risk, significant factors beyond awardees’ control and non-profit leads limit appropriateness of the SMOG mechanism. Challenging SMOG Negotiation Process & Means of Verification: As evidenced by the number and type of SMOG modifications, the SMOG milestone negotiation process could be improved. MoV with detailed information on buyers are more feasible to collect and more useful when costly and complex technology is being sold to a small number of purchasers. The cost per sale of collecting sales information is much higher for low cost/high volume technologies. Milestone Renegotiation & Failure to Meet Milestones: P4I demonstrated flexibility in modifying milestones. Some common types of milestone modifications indicate a need for improving the milestone design process. Implications of changing milestone due dates to “as completed” are unclear. Recommendations Utility of the SMOG Model: Improve SMOG Grants Manual, enhance alignment with FAA guidance and include risk and complexity criteria for SMOG use. Include options for other grant mechanisms. P4I Mid-term Performance Evaluation Report | November 2015 x SMOG Negotiation Process & Means of Verification: Improve the information provided during solicitation and negotiation to assure expectations for the milestone-based SMOG mechanism are fully understood by applicants from the beginning of the process. Provide examples of effective milestones. Simplify MoV where feasible, keeping focus on what is needed to verify milestones and report on USAID indicators and on consistency across sub-grants where appropriate. Milestone Renegotiation & Failure to Meet Milestones: Prepare guidance on milestone design. Review milestones to determine which are most effective under what circumstances and impacts of due dates “as achieved.” Make clear the implications of not fully achieving a milestone deadline. EVALUATION QUESTION 3: Assessing Benefits to Smallholders: How Can P4I Improve its Ability to Assess Whether P4I Component 1 and 2 Grants and Partner Investments are Increasing Smallholder Men and Women Farmers’ Access to and use of Targeted Technologies? Findings Targeting Smallholder farmers: P4I’s ability to report on whether smallholder farmers are being effectively targeted is limited by the type of data being provided by sub-grantees and the types of calculations performed on that data to convert it to indicator reporting for USAID. Gender: P4I does not effectively track sex-disaggregated data on technology purchase, use within the household, or benefits from its use and has no explicit means of tracking or demonstrating outcomes for women. Sub-grantee data on who purchases technologies does not track who uses the technologies. P4I Monitoring and Evaluation (M&E) Plan: Some of the P4I indicators do not accurately represent the number of smallholder men and of smallholder women farmers accessing and using P4I technologies. The overall P4I M&E Plan does not include Performance Indicator Reference Sheets (PIRS) and does not adequately address gender. (The P4I Mozambique M&E Plan does include PIRS and some gender disaggregation.) Means of Verification, Data Validation, Reporting, & Sub-grantee Interest in Data: The data on end users is reasonably strong where sales to the end users are under direct control of sub-grantees. For third-party sales and distribution, collecting information on end-users can be challenging. Sales validation surveys help manage risk of over-reported sales. There are no plans for final reports or evaluations of sub-grants of the type typically done on development projects. Representatives of the private sector firms indicated that data collected on end-users of their technology would be useful in growing their business. Conclusions Targeting Smallholder Farmers: Most of the reviewed technologies are designed to appeal to smallholder farmers. For other technologies, most (but not all) end-users are likely to be smallholders. Gender: It is difficult for P4I to monitor, make needed adjustments and demonstrate outcomes for women when women’s access to and use of the technologies is not being effectively monitored. P4I Mid-term Performance Evaluation Report | November 2015 xi P4I Monitoring and Evaluation (M&E) Plan: There are gaps in the indicators used to monitor P4I progress and results. The absence of PIRS and attention to gender compromise the overall P4I M&E Plan’s utility. Means of Verification, Data Validation, Reporting, & Sub-grantee Interest in Data: There are significant challenges to data collection by sub-grantees and to requiring parties with little stake in the sub-grant (such as small agrodealers) to collect the data. Sales validation surveys are important for verifying sales data and for determining P4I sub-grants’ effects on smallholder men and women farmers’ access to and use of targeted technologies. Private sector firms may not be as averse to collecting such data as P4I assert, as long as the data has clear utility and is cost-effective to collect. Recommendations Targeting Smallholder farmers: P4I should verify purchases and clearly attribute sales to smallholder farmers. A modified version of the phone survey P4I conducted is among the options. Gender: P4I should provide more explicit attention to gender in implementation, its M&E plan, and in monitoring. Collection and disaggregation of data by sex must be improved. P4I Monitoring and Evaluation (M&E) Plan: Add PIRS, explicitly incorporate gender, conduct a data quality assessment, and incorporate additional indicators. Means of Verification, Data Validation, & Reporting: Improve negotiations to assure feasibility, cost-effectiveness, and utility of MoV, particularly for high volume/low cost technologies. Incorporate costs of collecting data into sub-grant contracts. Conduct mid-line mini-surveys and endline mini-surveys or final assessments. Future SMOGs should include a final report on achievements. Fintrac should submit a final report on each sub-grant. USAID should consider an independent final evaluation. EVALUATION QUESTION 4: How Might Management of P4I (By Fintrac, BFS, and/or Missions) be Improved to Maximize P4I’s Outputs and Outcomes? Findings Management by Fintrac, USAID/BFS and Missions: USAID Component 1 was implemented as planned; various changes have improved its efficacy. Misunderstandings over Component 2 delayed its start by about one year. High staff turn-over limited progress during the first 18 months. Many staff worked on SMOGs. See also findings under other evaluation questions. Grant Selection Process: Round 1 Grant solicitation materials and selection criteria led to some sub￾grants not being aligned with Mission FTF strategies. Improvements were made during subsequent rounds. Conclusions Management by Fintrac, USAID/BFS and Missions: Component 1 implementation has improved. The Component 2 delay reduced the number of Missions that could participate, maximum possible length of sub-grants, and overall possible P4I impact. Staff composition and staff assigned to SMOGs appear to have stabilized. Differing USAID and Fintrac expectations for level of communication affect USAID collaboration and support. See also conclusions under other evaluation questions. P4I Mid-term Performance Evaluation Report | November 2015 xii Grant Selection Process: Changes in grant application and selection criteria led to better alignment with Mission strategy. The import of milestones was often poorly understood until well into negotiations. Recommendations Management by Fintrac, USAID/BFS and Missions: Reach shared understanding of USAID/Fintrac communication, collaboration and engagement with supported private sector firms. See also recommendations under other evaluation questions. Grant Selection Process: Assure full sub-grantee understanding of milestones and the SMOG. P4I Midterm Performance Evaluation Report | November 2015 1 EVALUATION PURPOSE & EVALUATION QUESTIONS EVALUATION PURPOSE Within the USAID Bureau for Food Security (BFS), the Office of Markets, Partnerships and Innovations (MPI) is charged with the mandate to promote new technologies and approaches to food security through innovative partnerships that improve market access for food insecure households in FTF countries. The Partnering for Innovation activity (P4I - originally called Agriculture Commercialization and Innovation) is intended to provide BFS/MPI a platform from which it can fulfill this role. BFS contracted Social Impact (SI) to undertake a midterm performance evaluation of P4I. This evaluation was carried out as part of the BFS Performance Evaluations task order held by SI under the Transparency, Accountability and Performance (TAP) Indefinite Quantity Contract (IQC) and commissioned by BFS in accordance with ADS 203.3.1.3: “Each USAID Mission/Office is required to conduct at least one evaluation of each large project it implements.” Consistent with the Office for Learning, Evaluation, and Research’s 2011 Evaluation Policy, the P4I performance evaluation is intended to provide empirical evidence to respond to evaluation questions (see the four evaluation questions listed below) designed to support learning and continuous improvement for BFS’ work. The evaluation was undertaken through a collaborative, utilization-focused approach. The primary audience for the P4I midterm performance evaluation is USAID’s BFS/MPI, the implementing partner (Fintrac), and USAID Mission staff in the countries that work with P4I or that are considering mechanisms for engaging private sector actors in furthering FTF and other Mission objectives. BFS is expected to use the findings, conclusions and recommendations from this evaluation to guide the implementation of the remainder of the P4I activity, through September 2017. Most immediately, findings from this evaluation report will be used by BFS and Fintrac as input for the design of the next P4I annual work plan. USAID/Mozambique and USAID/Kenya may find some information immediately applicable. Secondary users may use the results of the P4I evaluation to inform their decisions on programming and engaging private sector entities to increase smallholder access to technologies expected to improve food security. EVALUATION QUESTIONS The P4I midterm performance evaluation addresses the four central evaluation questions listed below, which are taken directly from the P4I Program Evaluation Scope of Work and the P4I Evaluation Design Plan, already submitted under this evaluation.3 The description in the paragraphs following the questions provides background on the decision to select these questions. The evaluation questions focus on P4I Component 1 “Agricultural Technology Commercialization” and 2 “Partnership Development” (see the 3 Illustrative evaluation questions were provided during the Question and Answer period of the Request for Task Order Proposals (RFTOP) Number SOL-OAA—14-000117 process of which this evaluation is a part. During discussions with USAID staff at the start of the evaluation design, it was determined that those questions did not fully fit USAID’s current needs and USAID/BFS and SI staff worked together to refine the questions. They retain the same general themes as the original questions. P4I Midterm Performance Evaluation Report | November 2015 2 description of the four P4I components below) and specifically on the implementation of those two components in Kenya and Mozambique, respectively. 1. P4I as USAID interface with private sector: What are the strengths and weaknesses of the ways in which P4I works with the private sector (through Components 1 and 2) as a means to increase USAID’s private sector engagement in Mozambique and Kenya? 2. Effects of the SMOG funding model from the perspective of partners: What are the effects of the structure of the SMOG funding model on private sector firms’ ability to scale up smallholder access to their technologies? 3. Assessing Benefits to Smallholders: How can P4I improve its ability to assess whether P4I Component 1 and 2 grants and partner investments are increasing smallholder men and women farmers’ access to and use of targeted technologies? 4. Management: How might management of P4I (by Fintrac, BFS, and/or Missions) be improved to maximize P4I’s outputs and outcomes? PROJECT BACKGROUND According to its cooperative agreement, the Feed the Future Partnering for Innovation activity is a five￾year initiative designed to engage the private sector in a meaningful, comprehensive way to meet the US government’s global food security challenges. USAID/BFS is funding this effort to promote small holder growth by leveraging resources and expertise in commercial partnerships. The P4I cooperative agreement was awarded in September 2012 to Fintrac and runs through September 2017. The program budget is $66.9 million, of which $52.5 million is allocated for commercialization of technology and mission partnerships. The goal of the P4I Program is to find new approaches to food security through innovative partnerships, technologies, and methods that improve food security and nutrition for food insecure households in FTF countries. The expected outcomes or objectives include the following: 1. Commercial availability of improved agriculture technology to small farmers increased. 2. USAID leverages support from private companies to meet individual program goals. 3. USAID is able to partner with the private sector to reach smallholder farmers effectively and within the parameters of private sector needs. 4. Private sector, Missions, stakeholders, and members of the development community have a central source of information and analysis on innovations reaching smallholder farmers. The P4I program provides the Bureau for Food Security a platform from which USAID can establish public private partnerships (PPPs) quickly and efficiently through a competitive selection process. P4I is designed to support the FTF Initiative by addressing constraints to increased access to inputs such as seed, feed, fertilizer, machinery, and irrigation systems at the right time, the right price, and in the right amounts in concert with partners to develop private input industries, organize private dealer networks, expand sustainable irrigation and water management, and strengthen farmer organizations. The program P4I Midterm Performance Evaluation Report | November 2015 3 facilitates partnerships between USAID Missions and the private sector and designs effective business development and engagement models and tools. MAIN PROGRAM COMPONENTS P4I has four components. As noted above, Components 1 and 2 are being covered by this evaluation. Component 1: Agricultural Technology Commercialization: The purpose of this component is to identify and bring to scale promising “bottom of the pyramid” agricultural technologies that are ready for commercialization. This component is a core focus of P4I as it is expected to enable increased access to innovative technologies by smallholder farmers, women in particular. Component 2: Partnership Development: Component 2 provides a platform for developing 10-20 new, innovative partnerships between USAID missions and the private sector, with financial and technical support provided through P4I. It includes partnership management and development through outreach and communication with private sector partners to channel commercialization toward FTF priorities. It includes 1) support for relationship management; 2) support for partnership management; 3) partnership start-up funding; and 4) promotion of mission buy-in to partnerships. Component 3: Investment Design and Modeling: P4I is developing a toolkit of adaptable models for engaging the private sector, including lessons learned and best practices for different types of partnerships and innovative ways of engaging the private sector. This component will be linked to Component 2 as the partnership models developed under this component are used to advise Missions on the most relevant design and institutional structures to facilitate successful partnerships. Component 4: Knowledge Management: Under Component 4, Fintrac is expected to develop a central source of information and analysis on models for engaging the private sector in agriculture and food security. It will also incorporate lessons learned and best practices from a literature review and a review and analysis of USAID‘s past experiences in agricultural commercialization, partnership seeding, and the design and modeling of private sector engagement in agriculture. DEVELOPMENT HYPOTHESIS The core problem that BFS/MPI intended the P4I activity to address is low crop yields, which lead to low income and poor nutrition for smallholder farmers. The P4I development hypothesis is that lack of technology use and private sector investment in smallholder farming are the main contributing factors. (See also the P4I Theory of Change diagram in Annex IX.) KEY INTERVENTIONS AND DELIVERY MECHANISMS: SUB-GRANTS The delivery mechanisms of interest for this evaluation are sub-grants for innovative technologies. Partnering for Innovation is focused on smallholder farmers in FTF countries. Private sector partnerships were the chosen mechanism to reach the target FTF population. P4I used a competitive process for PPP selection to select sub-grantees (private sector firms or non-profits with private sector sub-partners) for the PPP awards. This evaluation covers the five sub-grants that were awarded under P4I component 1 in Kenya during the first two rounds of sub-grants and the four sub-grants that were awarded under P4I Component 2 in Mozambique, the first country with Mission buy-in for Component 2 and the only one with fully operational sub-grants at the time of the data collection. See Table 1. P4I Midterm Performance Evaluation Report | November 2015 4 Table 1: P4I Sub-Grants for Component 1 in Kenya and Component 2 in Mozambique I. Kenya P4I Component 1 (Technology Commercialization) Sub-grants The five sub-grants total $5.4 million and leverage private sector contributions of $2.2 million. Purdue University: Crop Storage Bags: Purdue University (working with second-level sub-grantee Bell Industries) received a one-year (September 2013- September 2014) sub-grant of $386,201 (total with 8% sub-partner leverage was $420,666). Through the sub-grant Purdue Improved Crop Storage (PICS) bags were promoted and sold in Kenya. The one-year grant was completed in September 2014 with the successful commercialization (production and sale) of the hermetically sealed crop storage bags into the Kenyan market. During the grant period, Bell sold 60,000, more than three times the target of 17,500 set out in its SMOG milestones. Netafim: Small Scale Drip Irrigation Commercialization: Netafim received a two-year (September 2013- September 2015) sub-grant for $993,940 (total with 49% sub-partner leverage is $1,945,988). Netafim, one of the largest integrated drip irrigation companies in the world, holds a sub-grant to work with second-level sub-grantee Amiran and second-level sub-partner Connexus to provide access to finance to enable smallholders farmers to purchase drip irrigation systems. Netafim and Amarin provide a package of equipment, support, and after sales support designed to help smallholders successfully implement drip irrigation on their farm. African Agricultural Foundation (AATF): Imazapyr-Resistant (IR) Seed: AATF received a three-year two month (January 2014 – March 2017) $3,287,405 sub-grant from Fintrac (total with 23% sub￾partner leverage is $4,263,689) for work in Kenya, Tanzania and Uganda. The program in Kenya is partnering with Kenya Seed and Freshco seed companies to commercialize imazapyr-resistant improved maize varieties to combat Striga, a parasitic weed. The companies have expanded production, marketing and sale of IR seed in six FTF counties in western Kenya. Surehatch: Incubators: Surehatch received a one-year (July 2104 – July 2015) sub-grant of $260,289 (total with 25% sub-partner leverage is $346,744). Surehatch, a South African company, works with its local Surehatch affiliate in Kenya to sell small-scale incubators. Demonstrations, training and after sale support are being offered to promote purchase and use of small-scale incubators. Two hundred small scale incubators have been sold and over 1000 Kenyans, mostly women, have been trained on raising poultry as a business. MEA: Legume Inoculants: MEA received a 26-month (October, 2014 - December, 2016) sub-grant of $432,131 (total with 22% sub-partner leverage is $552,039). Through the sub-grant, fertilizer firm MEA is to expand production and sell BIOFIX®, a low-cost rhizobial inoculant for legumes. II. Mozambique P4I Component 2 (Partnership Development) Sub-grants The four sub-grants total $11 million and leverage private sector contributions of $18.5 million. NCBA/CLUSA and second-level sub-partners Oruwera and Phoenix Seeds were awarded a 31-month (December 2104 – July 2017) sub-grant to expand access to quality inputs and services for 10,000 smallholder farmers. The $1,705,067 sub-grant (total with 52% sub-partner leverage is $3,516,336) is designed to provide community-based extension services based on the model used by an existing NCBA/CLUSA program funded by Norway. Activities will support seed multiplication and link P4I Midterm Performance Evaluation Report | November 2015 5 EVALUATION METHODS AND LIMITATIONS METHODOLOGY FOR QUANTITATIVE AND QUALITATIVE DATA COLLECTION This mid-term performance evaluation used a process evaluation approach based on qualitative data. A process evaluation is designed to analyze how an activity operates and what factors appear to be most and least successful in forwarding the expected outcomes. As requested by USAID/BFS, this evaluation was designed to focus on P4I Components 1 and 2 as they have been implemented in Kenya and Mozambique respectively. Qualitative data was used to provide robust, objective evidence to answer the evaluation questions. Qualitative questions addressed the experiences, perceptions, opinions, motivations, and knowledge of the stakeholders interviewed, the context within which the activity is operating; and the factors (causal mechanisms) leading to observed results. In-depth conversations guided by structured and semi-structured key informant interview (KII) instruments provided the opportunity for a flow of information that is not constrained by pre-determined response categories. This allowed for exploration of subjects that surfaced during the interviews and elicited information that provided nuance and insight for addressing the evaluation questions. (See Annex II for full methods.) farmers to Oruwera dealers and 200 Phoenix Seed lead farmer agents and to land preparation and harvest services in 10 districts in Zambezia and Nampula provinces. Oruwera and Phoenix Seeds have developed input packages based on farm size, including seed, fertilizer, and inoculants. Opportunity International (OI) through its local financial services provider, Banco Oportunidade de Mozambique (BOM), is providing farmers in Manica and Zambezia with credit, savings accounts, and mobile money services oriented towards sesame and soybean production. The $1,754,072 sub-grant (total with 53% sub-partner leverage is $3,713,714) runs for 2.75 years (November 2104 - July 2017). To provide tools to capitalize on the newly offered financial services, BOM is providing agricultural extensionists to train farmers on good agricultural practices and financial literacy and improving farmer access to input and output markets. iDE received a 2.75 year (November 2014 - July 2017) sub-grant of $1,098,149 (total with 51% sub￾partner leverage is $2,229,162) . iDE will work with second-level sub-partners Lusosem Mocambique, Lda and HUB Assistancia Technica e Formacao to build the capacity of 40 agrodealers and establish 200 commercially oriented community-based extension advisors to reach smallholder farmers in Nampula and Zambezia provinces with production inputs an advice. Export Marketing Company LDA (EMCL) and its local office Export Trading Group (ETG) received a $6,458,484 (total with 68% sub-partner leverage is $20,101,906) 31-month (January 2015 – July 2017) sub-grant to establish 23 retail hubs, each with an agro input retailer, farm equipment dealer/rental agent (associated with Agro Tractors Limited), and an ETG operated warehouse where smallholder farmers can sell or store produce. Second-level sub-partner and affiliate TechnoBrain is implementing a mobile platform targeted at providing smallholder farmers with up-to-date production advise, forecasts, commodity pricing, pest control, and information on good agricultural practices. P4I Midterm Performance Evaluation Report | November 2015 6 The evaluation team applied qualitative techniques to elicit information from a variety of P4I stakeholders, including representatives from USAID/BFS, Missions, the implementing partner, sub-grant recipients and their second-level sub-partners, and retailers selling the targeted technologies. Interviewing this wide variety of stakeholders allowed for triangulating information on P4I processes and progress from multiple perspectives. The team also reviewed secondary data from sub-partner monitoring records. The team carried out data collection between April 16 and July 22, 2015. Much of the primary qualitative data and some of the secondary data was collected during field work in Mozambique (May 18 – May 26) and Kenya (May 27 – June 5). QUALITATIVE METHODS Qualitative data was collected primarily through key informant interviews and observation. Secondary data sources were also reviewed. See Annex II for a matrix summarizing the data collection methods used for each evaluation question. Key Informant Interviews: KIIs were the primary data collection technique. The KIIs captured information from the perspectives of the key stakeholders for P4I Component 1 and Component 2. The evaluation team designed a separate key informant interview guide for each type of stakeholder. (See Annex V.) Each guide included a core set of questions that were asked of all key stakeholders along with questions specific to the stakeholders’ role in P4I. The evaluation team conducted KIIs with the following stakeholders: 1) P4I staff; 2) staff of sub-grant recipients; 3) staff of second-level sub-partners (organizations that work directly with sub-grant recipients); 4) Retailers who sell technologies supported through P4I grants; 5) Farmers (some of whom used P4I supported technologies and some of whom did not); 6) USAID/Mission staff in Kenya and Mozambique; and 7) USAID/BFS staff. Observation Site Visits: The evaluation team traveled to selected activity sites. They observed parts of the P4I Technical Review Committee meeting where applicants to the fourth round of sub-grants were assessed and rated. The team also conducted observations at selected retail outlets where supported technologies are sold (as well as at others where they were not sold). The site visits in the field allowed the team to confirm data found in project reports and/or KIIs as well as to make observations of market dynamics. During observation visits to retailers, the team selected farmers (male and female) to ask a brief set of questions about interest and accessibility of technologies supported by P4I. The evaluation team also met with two groups of farmers and eight individual farmers who lived near or were involved in demonstration plots. Secondary Data Document Review: The bulk of the secondary data for this evaluation was gathered from existing documents from USAID/BFS, the Missions, and P4I: P4I award documents, implementation documents and progress reports; documents on sub-grant solicitations and awards for the four rounds of Component 1 and for Component 2 in Mozambique; and documents on and from sub-grant recipients and their sub-partners. (See Annex VI.) These sources included both quantitative and qualitative data. Monitoring Data: Data from the P4I performance monitoring system was reviewed in addition to data on sales that sub-grant recipients and their second-level sub-partners had available. ANALYTICAL APPROACH Qualitative Analysis P4I Midterm Performance Evaluation Report | November 2015 7 Qualitative data analysis involves the identification, examination, and interpretation of patterns and themes. The analysis focused on emergent themes as they related to each question. Opportunities were sought to interview women staff members of the sub-grant partners and second-level sub-partners. The evaluation team applied five qualitative data analysis steps to data from collected in the US, Kenya and Mozambique: Step 1) Process and record data immediately; Step 2) Begin analyzing as data is being collected; Step 3) Data reduction; Step 4) Identifying meaningful patterns and themes; Step 5) Conclusion drawing and verification. The analysis approaches for each evaluation question are summarized in the Evaluation Question Matrix found in Annex II. Quantitative Data Analysis While the bulk of the data for responding to the evaluation questions was qualitative, quantitative secondary data, such as data from the P4I monitoring and evaluation system and partner and sub-partner records, was also reviewed and analyzed. LIMITATIONS OF THE EVALUATION METHODOLOGY The evaluation methodology has significant strengths along with limitations that affect the ability to generalize based on findings. The evaluation design took these limitations into account. The scope of the evaluation was to cover two of the four P4I components, so findings, conclusions and recommendations are limited to these two components. While the team conducted interviews with as many sub-partners as possible, limited time in each country and the distance between locations where different sub￾partners operate resulted in the evaluation team using opportunistic sampling to prioritize meeting with sub-partners that are located close to each other geographically. Focus on sub-grants under two components in two countries: Because the opportunities and challenges are different in different country contexts, carrying out the evaluation work in just one country per component limits the ability to generalize. This evaluation is centered on the implementation of two components of P4I in two countries: Component 1 in Kenya and Component 2 in Mozambique. This is both a strength and a limitation. Both countries are sound choices. Home to five of the fifteen Component 1 sub-grants made by P4I, Kenya has the highest concentration of P4I sub￾grants of any FTF country.4 (See Annex III for details on the full set of P4I sub-grants.) Mozambique is a clear choice for Component 2 as the Mission in Mozambique was the first to participate in P4I’s 4 Between P4I inception and the time of the evaluation research in May and June of 2015, P4I had awarded 15 Component 2 grants. While two were multi-country grants, the majority were for work in just one country. Most countries where sub-grants were awarded had just one sub-grant. However, five sub-grants were made in Kenya and two sub-grants were made in Zambia. There were eight sub-grants in Round 1, covering countries 11 countries: Kenya (three sub-grants – one of which also included work in Tanzania and Uganda), India (one sub-grant), Rwanda (one sub-grant), Senegal (one sub-grant), Zambia (one sub-grant), and Cameroon, Côte d’Ivoire, Ghana, and Nigeria (one joint sub-grant). In Round 2 there were five sub-grants: Kenya (two sub-grants), Bangladesh (one sub-grant), Guatemala (one sub-grant), and Zambia (one sub-grant). Round 3 included two sub-grants: Ethiopia (one sub-grant) and Honduras (one sub-grant). See Annex III for details on the P4I sub-grants that had been awarded at the time of the research. Given that on-third of the Component 1 sub-grants were located in Kenya (and given that the sub￾grantee for one of the multi-country grants was in Kenya), Kenya was chosen as the site for evaluation field work. P4I Midterm Performance Evaluation Report | November 2015 8 Component 2 work and as there are four public private partnerships supported by P4I that are currently being implemented in Mozambique. Distance and time: Due to the limited field time (seven working days in each of two countries to cover a total of nine sub-grants) and the distances between implementation sites, the findings, while robust, do not provide a fully comprehensive representation of P4I’s Components 1 and 2 work. The relatively short amount of time in each country limited the evaluation team’s ability to meet with all of the stakeholders of interest, though the team was able to meet in person or by phone with nearly all of the sub-grantees and their second-level sub-partners. The evaluation team conducted opportunistic and purposive sampling to conduct interviews with a small sample of retailers and of users and non-users of the technologies supported by P4I. As agreed with USAID/BFS, given the amount of time allocated for fieldwork, the evaluation design did not include an assessment of outcomes at the smallholder farm level. The distances between some of the implementation sites, poor infrastructure and security considerations meant that the evaluation team did not have time to visit all implementation sites in each country. In coordination with USAID staff in Mozambique, the evaluation team decided to fly to Nampula, where there is the largest concentration of sub-grantees and second-level sub-partners. In Kenya, the evaluation team flew to Kisumu and drove to other nearby implementation sites. EVALUATION TEAM The primary evaluation team is composed of two senior-level key personnel: Team Leader and Senior Agricultural Development Technical Advisor, Rees Warne, and Senior Agricultural Innovations and Commercialization Expert, James Dempsey. Ms. Warne has served as evaluation team lead on impact, performance, and mid-term evaluations covering agricultural production and commercialization, business development services, and value chains in Africa and elsewhere. She has managed agriculture and commercialization projects as an implementer. Mr. Dempsey has technical expertise in agricultural value chains. He recently worked within the Kenyan Ministry of Agriculture developing and strengthening Kenya’s agriculture sector M&E plan and systems. Both have extensive experience conducting qualitative performance evaluations, including design of methods and instruments, implementation, analysis and reporting. They were supported by Dennis Wood (Social Impact’s Vice President for Evaluation Services and Technical Manager on this Task Order) and Dennis Marotta (Deputy Director and Senior Technical Advisor on this Task Order) who provided technical input and quality assurance. Program Manager Isadora de Latour provided evaluation management as well as report writing support, and Program Assistant Alessandra Barbiero provided coordination, editing, and logistical support. P4I Midterm Performance Evaluation Report | November 2015 9 FINDINGS, CONCLUSIONS, & RECOMMENDATIONS EVALUATION QUESTION 1: P4I As USAID Interface with Private Sector: What are the strengths and weaknesses of the ways in which p4i works with the private sector (through Components 1 & 2) as a means to increase USAID’s private sector engagement in Mozambique and Kenya? 1a) What worked well and what could be improved to maximize both the engagement of the private sector and the strength of those partners’ relationship with USAID? 1b) How well do the funded sub-grants align with the FTF strategies of USAID and of the Kenya and Mozambique missions? Findings: Evaluation Question 1 Please see Annex IV for a summary of strengths and weaknesses related to Evaluation Question 1. 1. Mission Needs and P4I Component 1 and 2 Sub-grants: Kenya and Mozambique There is a clear distinction between how staff from the Missions in Kenya and Mozambique perceive P4I. Kenya: Component 1: USAID/Kenya has strong and active FTF engagement of the private sector with a mature and comprehensive FTF program supporting numerous and strong local private sector partnerships. There was no need for or interest in accessing support through P4I Component 2. To date, there are five P4I Component 1 sub-grants in Kenya. The total planned expenditure for the Round 1 and 2 P4I sub-grants in Kenya is $5.5 million over 3.5 years. (Note: current and possible up-coming P4I award round(s) may bring further grants to Kenya.) For comparison, the Kenya Mission’s FTF funding is approximately $45 million per year. A key point raised by USAID/Kenya staff is that the Kenya FTF program has its own private sector innovation mechanism, the Kenya Feed the Future Innovation Engine (KFIE) program. Few Missions have similar programming. The $22 million KFIE activity (authorized in May 2012 and initiated in early 2014 for a five year period of performance) has essentially the same objectives as P4I.5 Kenya Mission informants indicated that the Mission was surprised to learn that the P4I program was authorized for Kenya given that KFIE already was approved. Once authorized, the Mission accepted P4I as an additional FTF resource for Kenya. The Mission approved new P4I awards in Kenya following the P4I procedures for local Mission clearance. Little post-award follow-up and coordination was done by Mission staff. 5 KFIE’s objectives are to enhance adoption of innovative agricultural technologies and practices, improve agricultural productivity and markets, and increase private sector investment in agriculture and nutrition-related activities. KFIE invests in selected products, services and business models that have potential for wide-scale adoption and marketplace sustainability. A competitive process identifies innovation partners who receive KFIE assistance to help them move to commercialization. KFIE supports promising innovations in agriculture and nutrition through three tiers, starting from “proof of concept” through “pilot roll-out” to “transition to scale-up.” A firm can be supported through KFIE at any one of these stages. P4I Midterm Performance Evaluation Report | November 2015 10 P4I Component 1 sub-partners and/or second-level sub-partners in Kenya are working with technologies ready to be commercialized to smallholders. Private sector engagement is about overcoming specific scale-up constraints for commercialization rather than for business capacity development. P4I places heavy emphasis on milestones directly tied to sales in all of its sub-grants, and some sub-grant payments are not made until sales targeting in the milestones are met. KFIE emphasizes business development and grant payments are based on expenditures and system development targets. Mozambique: Component 2: The situation in Mozambique is quite different from that in Kenya. USAID/Mozambique’s FTF program was at a transition stage in early 2013. The Mission was interested in having agile FTF programming to fill the gap until its planned FTF award was made. Mission staff saw P4I Component 2 as an attractive mechanism for providing bridge FTF programming and for addressing the USAID interest in engaging and partnering with the private sector. 1.1. Strengths a) Component 2 Agility: According to USAID/Mozambique staff, the P4I Component 2 sub-grant development and award process moved more quickly than an award through their Mission could have.6 USAID/Mozambique staff indicated appreciation for the opportunity for the Mission to take advantage of P4I as a transitional FTF funding mechanism for the Mission that awarded sub-grants as a program bridge to the next generation of FTF programming in Mozambique. USAID/Mozambique staff involved with the P4I work expressed appreciation for the timeliness of the availability of P4I’s Component 2 and of P4I’s ability to quickly put FTF awards and programming in place. b) Component 2 Low Cost and Low Level of Effort for Mission: USAID/Mozambique staff expressed appreciation that the P4I Component 2 allows USAID/Mozambique to fund FTF programming and report on FTF indicators while also alleviating the Mission of the need to allocate substantive staff time or budget for management of the FTF activities it funds. Fintrac performs most of the administration and management of the sub-grants. The Mission found it adequate for management to depend on Fintrac and on the USAID/BFS-based AOR and to simply assign a staff member to each of the four P4I sub-grants as an Activity Manager. Activity Managers indicated appreciation that little technical or management staff time was needed for sub-grant award and management. c) Component 1 and Smallholder Farmers:. USAID/staff in Kenya noted that there were benefits to expanding smallholder farmer access to promising technologies. 1.2 Weaknesses: Component 1 Missing Opportunities for Coordination to Improve Sub-grant Results: For Component 1 in Kenya, there has been little active coordination or cooperation between P4I-supported sub-grants and activities and USAID/Kenya staff after the sub-grants were awarded. While, in general, P4I is in line with the USAID/Kenya FTF strategy and supports USAID/Washington engagement with the private sector, there have been some disconnects with the Mission FTF strategy. For instance, in the 6 The P4I Mozambique request for Expression of Interest had as its goal to “support the development of public-private partnerships to support smallholder farmers in Mozambique, through expanding market access, providing improved inputs, and/or offering technical training to smallholder farmers in select value chains and the Feed the Future targeted districts.” P4I Midterm Performance Evaluation Report | November 2015 11 case of the P4I Surehatch grant, while its interventions and development of small scale chicken and egg production are moving forward well, poultry is not a priority value chain in the Kenya FTF strategy.7 As noted above, USAID/Kenya has a large FTF program and an FTF grant-making activity for innovations. Mission staff indicated that the independent operation of another FTF grant-making activity in Kenya that was not associated with the Mission introduced some complications into their relationships with private sector actors who conflated the two FTF grant mechanisms. This was seen as undermining the goal of a single, coherent FTF strategic approach. 2. Private Sector Engagement 2.1 The Mix of Non-Profit and For Profit Companies as P4I Sub-grant Awardees: The purpose of P4I Components 1 and 2 is to have private for profit companies make it part of their business to support smallholder production and livelihoods. While all sub-grants include at least one private sector partner and a private sector leveraged funding match (in Mozambique a minimum 50% match was required), many of the sub-grants awarded under P4I have been given to non-profits that are traditional USAID grantees rather than to private for profit businesses. Just four of the nine sub-grants that the evaluation team was assigned to review in Kenya and Mozambique were made to private sector firms. Of the four Mozambique Component 2 sub-grants reviewed, only one was awarded to a private sector firm, while the other three were to US NGOs (iDE, NCBA/CLUSA and OI). In Kenya, three of the five sub-grants were awarded to private sector firms, while two were to non-profits (Purdue University and African Agricultural Technology Foundation). 2.2 Perspectives on Private Sector Need for Non-profit Intermediation: The case was made by nearly all the non-profit sub-grantees that the local private sector firms needed their intermediation and assistance in order to win and successfully implement the P4I sub-grants. Non-profits’ pre-award activities include assisting the private sector firms with sub-grant application, negotiation, regulations, and compliance. Their post-award assistance includes planning, management and reporting, compliance with USAID regulations, and liaising with P4I and USAID, in addition to business and marketing operations, outreach to and education of farmers and input sales agents, and specific capacity-building related to the innovation. In Mozambique, two of the NGOs were working with smaller, newer private sector firms that did have significant operational and other challenges. Two of the private sector firms that were second-level sub-grantees indicated that they would have been able to apply for and manage a P4I award on their own. In Kenya, one private sector firm acknowledged the importance and central position of the non-profit’s assistance to its early work on the sub-grant. It then went on to apply on its own for an award from USAID, saying that it had learned about working with USAID through the non￾profit’s assistance on the sub-grant. 2.3 Strengths a) Support for private sector engagement: P4I supports private sector engagement under FTF. 7 Note P4I subsequently amended the criteria for selection of proposals to be funded to include a criterion on consistency with the FTF value chains prioritized in the country of proposed implementation. P4I Midterm Performance Evaluation Report | November 2015 12 b) Acceptance of private sector or non-profit sub-grant leads: While P4I was designed to support commercialization by private sector firms, P4I has created a flexible structure that accepts consortia of private sector firms and non-profits. c) Substantive Leverage: In Mozambique Component 2 sub-grants, private sector partners bring a match of 51-68%. In Kenya Component 1 sub-grants, the match is between 8% and 49%. 2.4 Weaknesses a) High Percentage of Non-Profit Sub-grantees: Of the nine sub-grants covered in this evaluation of P4I, an activity aimed at partnering with the private sector, five sub-grants were led by non-profit organizations. While all P4I sub-grants include private sectors firms at some level and the focus is on supporting them to commercialize technologies, overall, half of P4I’s 24 sub-grants have been awarded to non-profit organizations.) Six of the nine sub-grantees were based outside of the country in which the sub-grant was being implemented. Both of these situations limit Missions’ ability to engage effectively with the private sector firms being funded in their country. b) Lack of Awareness of USAID Role: Some key staff of entities awarded sub-grants through Component 2 that were interviewed were not aware of USAID/Mozambique’s role in funding their work. Some NGO sub-grantee staff confidently stated that they had a standard sub-partner arrangement with Fintrac, as P4I’s prime. While they could name the AOR in Washington, they indicated that they were not aware of any USAID/Mozambique involvement in their work. c) Low Levels of USAID Staff/Private Sector Engagement: While P4I grants in Kenya are technically USAID/BFS private sector engagement, there is almost no USAID/Kenya engagement with the sub￾grantees and little involvement with P4I beyond review of the short-listed application and concurrence for the sub-award (Fintrac staff note that they have repeatedly reached out to USAID/Kenya).The P4I sub-grants operate independently from the Mission. At the time the interviews were conducted, some people in leadership roles in private firms that were second-level sub-grantees (including some who were subbing to NGOs) had never met anyone from USAID or had only met USAID staff through an informal opportunity with no substantive discussion about the firm’s FTF work. Some second-level sub-grantees were said to be included in the periodic phone calls with P4I (reportedly largely as listeners, and none appear to be listed as participants in the call notes for three sub-grantees in Mozambique). Sub-grantee staff were told to go through Fintrac and not contact USAID staff directly. Mission staff indicated that they would like more opportunities to interact with P4I private sector sub-grantees and second-level sub-grantees. 3. Management Complexity The P4I competitive process accepts applications that have a commercial private sector firm as the primary awardee or second-level sub-partner. Particularly when non-profits lead a sub-grant consortium, there can be many links in the chain between USAID/BFS staff and the private sector firms. For instance, under Component 2, USAID/BFS grants funds through the P4I cooperative agreement to Fintrac, which then works with a Mission which awards funds through Fintrac to the competitively selected non-profit, which then passes grant funds to one or more private sector second-level sub-partners. 3.1 Cost-Effectiveness: Fintrac applies no direct overhead charge to the grant funds made available by Missions; the direct Fintrac costs of managing the competitive selection and the implementation support P4I Midterm Performance Evaluation Report | November 2015 13 and monitoring are paid by USAID/BFS as part of the P4I grant to Fintrac. Regarding the layers of implementation management (including Fintrac and non-profits), one USAID staff person asked whether the P4I approach is cost effective. 3.2 Strengths: P4I Strong and Responsive Communications: P4I staff’s on-going communications with the sub-grantees, USAID Missions, and USAID/BFS were strong and important in keeping all on board in this multi-layered program structure. 3.3 Weaknesses: Complex Lines of Communication: In Component 2, management and communications links are sometimes cumbersome. Two USAID/Mozambique staff pointed out that this creates distance between the Mission and some of the private sector implementers of the Mission-funded work. 4. Roles and Responsibilities Fintrac and BFS Washington, DC-based staff are responsible for managing a highly complex globally dispersed activity with sub-grants covering a broad range of value chains in 11 countries. With the fourth round of Component 1 sub-grants and negotiations with Missions for work under Component 2 both in progress, P4I could potentially have operations in many more. 4.1 Strengths a) P4I Communication: P4I staff are strong communicators during on-going sub-grant management. b) AOR Role: Stakeholders spoke highly of the USAID/BFS AOR. Mission staff in Mozambique accorded her particular recognition for the amount of work done and the difficulty of the assignment. 4.2 Weaknesses a) Washington, DC-based Management: The Washington, DC locus of management support and decision￾making is far away from where the sub-grants operate. P4I staff have limited ability to observe sub￾grant operations on-the-ground or to provide hands-on, face-to-face support. The USAID AOR for P4I is based at USAID/Washington. This is a difficult assignment, similar to that of P4I staff, but all under the responsibility of one person. Many, if not most, of the decisions on the cooperative agreement and particularly on the management of the SMOGs come from technical and business challenges in a diverse set of P4I countries and circumstances far removed from Washington. The lack of hands-on management of awards by people with a rich understanding of the context in which the awards operate was pointed out as a limitation by informants from Missions, sub-grantees, and second-level sub-grantees, and USAID/Mozambique staff strongly requested a P4I staff person based in-country. Fintrac staff reported that distance did not affect management. b) Very Wide Array of Expertise Needs: The P4I team does not have (and cannot be expected to have) knowledge and expertise in all local development contexts, political and economic structures, country business operating environments, country regulations and protocols, local markets, and product applications in all the FTF countries. The range of technical topics and types of business programs that P4I staff must manage is similarly broad. Staff does not have technical expertise in all of the sub-grantee’s technologies and innovations. In terms of both country and operating context, the range of proposals that P4I is required to review, assess, and prioritize is extremely wide. P4I Midterm Performance Evaluation Report | November 2015 14 c) Component 2 Mission Role: Mozambique: While indicating appreciation that Mission staff did not need to spend substantive time managing the FTF awards, most Mission staff interviewed said that they would prefer to have a greater role in oversight and support for the awards. 5. Other Strengths 5.1 Macro-level Action: P4I provides USAID/BFS with a vehicle for action at a macro scale. This gives BFS the ability to match opportunities to needs and to support innovations with potential for scaling up in ways that support FTF’s global objectives and with potential for replication in a variety of contexts. 5.2 Sub-grant Negotiations: Some (though not all) sub-grantee firms expressed satisfaction with the pace of the process.8 All private sector sub-grantees interviewed indicated that the negotiation process was not as fast and straight forward as contract negotiation with a private firm. (See Evaluation Question 2.) The USG does have many grant requirements beyond what is found in commercial contracts: there are approximately 40 pages of standard and other provisions in the SMOGs that cover up to 20 different requirements for certification, agreement, or operational procedures. Many private sector sub-grantees noted that they were not clear on the need for and relevance of these requirements. Conclusions: Evaluation Question 1 1. Mission Needs and P4I Component 1 and 2 Sub-grants: Kenya and Mozambique The relative attractiveness of both P4I Component 1 and Component 2 depends on the state of the FTF initiative in the Mission’s programing. Missions like USAID/Kenya that have strong programs engaging the private sector will likely find P4I Component 2 to offer less to the Mission’s FTF work than do Missions that face FTF program gaps or personnel shortages. Likewise, a Mission that is already of the opinion that it is addressing its FTF strategy and that it is managing complex relationships with FTF stakeholders may not wholly welcome additional FTF support from outside – particularly if the Mission doesn’t see the outside actor as offering support that can be easily integrated with its current program. 1.1 Agility: Under Component 2, P4I has been able to award USAID/Mozambique funds to implementing partners to further the Mission’s FTF objectives more quickly than a Mission can normally design, bid, and award cooperative agreements. Where Missions, for whatever reason, are not able to award FTF funding within their preferred timeframe, an activity like P4I mechanism can make the difference between having FTF programming and having none. P4I Component 2 functioned as an important transitional FTF funding mechanism for USAID/Mozambique, allowing the Mission to fund bridging activities while awaiting the completion of a large award for FTF programming. 1.2 Low Cost and Low Level of Effort for Mission: P4I Component 2 in Mozambique offers the Mission an option to engage the private sector in line with the country FTF strategy at a very low management cost. While the Mission’s budget provides funds for the sub-grants themselves, costs for staff working with contracts, accounting and technical support and management are minimal. However, cost effectiveness of the P4I grant management arrangement from the standpoint of total cost to USAID also needs to be taken into account. Without information on direct USAID management costs for grant 8 Note that the evaluation team did not compare the time period between emitting notice of the funding opportunity and release of funds for P4I sub-grants and other types of grants-under-contract mechanisms. P4I Midterm Performance Evaluation Report | November 2015 15 programs like P4I, the team could not judge the relative cost effectiveness of direct USAID awards versus that of the P4I approach. 1.3 Component 1 Missing Opportunities for Coordination to Improve Sub-grant Result: In Kenya, the procedure put in place between P4I and the Mission’s FTF program did not ensure direct and effective coordination and cooperation between the Mission and the sub-grantees. Not only did this result in some P4I resources not being in line with the Mission’s FTF strategy, it also may have limited some outcomes of a few P4I sub-grants. Recent revision of sub-grant selection criteria to assure that all awards are aligned to and support Mission FTF objectives is a positive step toward improving FTF outcomes. 2. Private Sector Engagement 2.1 The Mix of Non-Profit and For Profit Companies P4I Sub-grant Awardees: While P4I is intended to provide support to private sector firms and while the milestone grant agreements (including content of milestones and MoV) are designed for use by private sector firms, half of the P4I sub-grants follow a traditional (and successful) model of grants to NGOs for support to small farmers with the introduction of a new technology and donor subsidies to the private sector. While strong non-profits can be an important support to weaker private sector firms or those without much experience with smallholder farmers, the position of the non-profits as intermediaries between USAID and the private sector firms can weaken some elements of the desired USAID engagement with the private sector. 2.2 USAID engagement with the private sector: The primary engagement in P4I is between Fintrac and the sub-grantees (half of which are non-profits). a) Definition of engagement: P4I’s level of success in increasing USAID’s private sector engagement depends on the definition of “engagement.” If USAID engagement with the private sector is defined as provision of funds to private sector actors, then P4I has been relatively successful (discounting the funds that go to non-profit sub-grantees who are intermediaries between Fintrac and private sector sub-partners). If USAID engagement is defined as creating interactions and building relationships between Missions and cooperating country private sector firms, then P4I has been less successful. b) Weak Mission Role: Fintrac’s management of the application, award, and sub-grant management process has meant that little Mission involvement in the management of Component 1 sub-grants and their management has been required or requested and that relatively little has been required for Component 2 sub-grants. While Component 1 may strengthen USAID/Washington engagement with the private sector, it does not necessarily strengthen Mission engagement. 3. Management Complexity 3.1 Communication: There is room for improving communication flows among P4I, USAID, and sub￾grantees and their second-level sub-partners. 3.2 Cost-effectiveness: The cost effectiveness of channeling P4I awards through non-profits to private sector firms depends on the added cost of the non-profit compared to the outcome benefits achieved without the non-profit’s assistance. This may bear particular examination for non-profit entities based in the US that incur substantive travel costs as part of their participation in the sub-grant. 4. Roles and Responsibilities 4.1 Washington, DC-based Management: The P4I staff are located in Washington, DC and spend relatively little time with each sub-grantee or second-level sub-grantee or in the county contexts where P4I Midterm Performance Evaluation Report | November 2015 16 the sub-grants operate. This is a constraint that was partially, but not fully, mitigated by staff member responsiveness and good communication. 4.2 Mission Activity Manager Having a Mission staff person assigned as Activity Manager for each sub￾grant in that country is effective in Mozambique and is crucial for Component 2 sub-grants. It enhances the utility and efficiency of sub-grant implementation as well as the outcomes from the perspective of both BFS and the Missions. 4.2 AOR Role: The AOR was judged to be responsive and helpful by USAID Mission staff. Working largely alone with responsibilities for the same 24 sub-grants that the entire Fintrac staff managed as a group places a large management burden on the AOR. Recommendations: Evaluation Question 1 1. Missions and P4I Component 1 Sub-grants BFS and Fintrac should improve Component 1 coordination and integration with Mission FTF programs. Much of this is now done through Component 2, and it should be done through Component 1 as well. 1.1 Mission Point of Contact: Each Mission in a country eligible for P4I sub-grants under Component 1 should name a point of contact for the IP. 1.2 Early Mission Involvement: Mission staff (the point of contact or another delegated staff person familiar with the proposal’s sector) should be involved in the Component 1 solicitation design where possible and in the due diligence process as soon as an in-country sub-grant proposal is short-listed. 1.3 Activity Manager: A Mission-based Activity Manager for Component 1 grants (even with very low level of effort expectations) would help bring country-specific input, help keep sub-grants in line with Mission FTF strategies and programming and enable provision of field-based support if/when needed. 1.4 Mission Action Plan: Under Component 1 as well as Component 2, the IP and Mission staff should design annual action plans that go beyond the current IP/Mission Memorandum of Understanding and that include specific plans or options to foster coordination and synergy with on-going and anticipated Mission FTF (and other) programming as well as for level and type of Mission staff involvement with sub￾grant oversight and troubleshooting. Elements of these action plans should be included in SMOGs as appropriate. During sub-grant implementation, perhaps every six months, Fintrac and the Mission should review and revise actions and plans for coordination and synergy as needed. 1.5 Engagement: BFS and P4I should work with Mission staff to create additional opportunities for representatives of private sector sub-grantees to interact with Mission FTF staff. 2. Private Sector Engagement 2.1 Non-profit roles: In the solicitation phase, provide suggestions for different role options for non￾profits that increase the role of private sector firms. Options may include having non-profits serve as supporting sub-partners to private sector sub-grantees firms (such as helping the private sector firms with capacity-building, reporting to USAID, etc.). 2.2 Private Sector Roles: Increase the role of private sector firms that are second-level sub-partners on sub-grants – particularly those that are sub-partners to non-profit organizations – and increase USAID’s P4I Midterm Performance Evaluation Report | November 2015 17 engagement with them. This could include creating increased opportunities for private sector second￾level sub-grantees to participate in periodic phone calls and other communications with USAID. 3. Management Complexity: Communication: The IP should make it easier for Mission staff to communicate with, support, and otherwise engage sub-grantees and their second-level sub-partners in their country of operation by reducing barriers to communication and for private sector firms to communicate with USAID. See also Recommendations 2 and 4.1. 4. Roles and Responsibilities 4.1 IP: Future activities should consider increasing IP staff presence in key countries, particularly Component 2 countries. P4I should consider increasing staff travel. 4.2 Mission Point of Contact/Activity Manager: The role of the Activity Manager should be expanded beyond the role currently in place in Mozambique. Activity Managers should be given additional opportunities to 1) be in direct communication with sub-grantees and their second-level sub-partners (especially when non-profits are the sub-grantees) and 2) review and comment on sub-grantee milestone and monthly reports before approval by Fintrac. 4.3 AOR: Given the increasing size and complexity of P4I, adding or bolstering USAID management could provide additional support to P4I programming. Within BFS, the AOR workload could be shared with other staff with complementary expertise, perhaps divided by region or components. EVALUATION QUESTION 2: What are the effects of the structure of the Standard Milestone Obligation Grant (SMOG) funding model on private sector firms’ ability to scale up smallholder access to their technologies? Findings: Evaluation Question 2 1. Overall Utility of the SMOG Model: The Standard Milestone Obligation Grant (SMOG) is the P4I variation of the USG Fixed Amount Award, formerly known as a Fixed Obligation Grant (FOG) Agreement. As specified in ADS 303.3.25, a Fixed Amount Award is a category of USAID grant that is “appropriate for supporting specific projects when USAID is confident that a reasonable estimate of the actual cost of the overall effort can be established and USAID can define accomplishment of the purpose of the grant through defined milestones.” A Fixed Amount Award can be a relatively straightforward way to manage the disbursement of grant funds for an activity that can be broken into smaller parts, each with a clear result or product that can be submitted as proof that the part has been completed. Each of these parts is called a “milestone.” P4I uses the term “means of verification” for the information to be submitted as proof that the milestone has been completed. P4I sub-grantees are paid a set portion of the grant amount when they complete a milestone, present the MoV, and receive Fintrac’s approval. Under P4I, Fintrac uses the SMOG for both Component 1 and 2 sub-grants. 1.1 P4I Stakeholder Perception of Utility of Performance-based Awards and the SMOG: Among the P4I stakeholders, there was unanimity in support of the concept of performance-based award mechanisms. According to Fintrac staff, the SMOG milestone-based fixed amount award provides advantages for both USAID and the recipient: it focuses on outputs and results, establishes incentives, limits risk for both P4I Midterm Performance Evaluation Report | November 2015 18 parties, requires limited financial and management actions to monitor, and is specifically designed for private sector firms. Fintrac staff strongly indicated that the SMOG is appropriate for all its uses. USAID/Mozambique staff indicated that it was beneficial that NGOs were able to serve as consortium leads where private sector firms wanted an intermediary. Many stakeholders noted that private sector firms expect to have contracted results achieved, and thus performance-based awards are common. Support for the concept also came from non-profit sub-grantee representatives interviewed, though in some cases they expressed less interest in working under this type of agreement. Nonetheless, there was less enthusiasm for the SMOG negotiation process or the SMOG itself. SMOG milestones were problematic and stressful for many sub-grantees, particularly where they had no control over needed inputs or faced high production risks. One informant from a private sector sub-grantee said they “wouldn’t do this kind of contract again.” 1.2 The Risk of Unexpected or Uncontrollable Factors for Milestone Completion: Factors beyond their control were common concerns among the P4I sub-grantees. During due diligence, P4I does an initial risk assessment, though they do not use the USG Fixed Amount Award Entity Eligibility Checklist. There are two types of milestones. The “soft” milestones are business plans, marketing strategies, software development or the like, which are, for most sub-grantees, within their own control. Many hard milestones are sales and production targets. Private sector sub-grantees and second-level sub￾partners have a history of sales or production to go upon but do not have full control over outcomes. Other hard milestones, such as equipment purchase and importation, were seen as problematic because they depended on actors outside of the sub-grantee’s sphere of influence. Agricultural production risks are inherently high: many factors are beyond the control of the agricultural producer and predictability of an assessment. As an example, AATF partners with two seed companies in each of the three countries where it operates under its sub-grant and faces, through those sub￾partners, seed production problems that endanger its ability to meet milestones. Seed production has been an issue for Oruwera, an NCBA/CLUSA partner in Mozambique (as it was for Oruwera’s predecessor IKURU). Most sub-grantees passed on some risks through milestones in agreements with sub-partners for the same level of sales that they were required to meet in their P4I sub-grant. For the non-profits, the SMOG represented an increased level of risk in part because it is mechanism to which they were unaccustomed (see cost recovery risks below). Informants from two NGOs recounted lengthy negotiations with their internal leadership to get agreement on the exposure to risk posed by the SMOG. This was particularly a factor for sub-grants which require substantive upfront investments in capacity-building before high-value production or sales milestones could begin to be achieved. 1.3 The SMOG is the Only Type of Grant Agreement used in P4I: The SMOG is the only form of grant award used in P4I to date. USAID ADS Chapter 303.3.25 on FOG awards to NGO offers guidance relevant to the SMOG. It states The AO, with the advice of the technical office and M/CFO (Washington) or the Controller (overseas), must assess the risk factors that could prevent the proposed recipient from completing the activity or require a substantial change in the milestones. The operating unit must not use the Fixed Amount Award mechanism if there is an unacceptably high risk of failure or substantial changes in the milestones are expected as the recipient implements its program. P4I Midterm Performance Evaluation Report | November 2015 19 However, there appear to be substantive sub-grant performance risks and substantive milestone changes needed. As an example, the AATF SMOG has specific milestones for the sale of a total of 955 tons of IR seed and the establishment of a total of 4000 demonstration plots, with separate targets for each of three countries (Kenya, Uganda and Tanzania), yet AATF has no seed production or sales capacity. AATF depends on second-level sub-grants to seed companies for the production and sale of the IR seed. As mentioned above, the risks inherent in seed production are high. Weather risks are high for production. Maize necrosis has wiped out much of the seed production in some sites. Sales are dependent on seed company commitment to the IR product and, importantly, attractive returns to the company for IR seed sales. AATF has had trouble meeting its milestones, and there have been three major modifications to the milestones in its SMOG. The original 45 milestones have been adjusted and expanded to 57, and the dates for completion of most sales milestones have been extended. The only completed grant in the evaluation sample was Purdue/PICS Storage Bags in Kenya. Purdue’s second-level sub-grantee, Bell Industries, exceeded sales targets by over 200 percent. 2. Challenging SMOG Negotiation Process: Notwithstanding support for the performance-based agreements, the SMOG negotiations were judged as difficult at times by both private sector firms and non-profits. The negotiation process was seen by some firms as too long. While the Component 2 sub￾grants were awarded in less time than many full USAID RFA processes would take, the time between proposal submission and award signing for some Kenya sub-grants was relatively long (see Table 2). Table 2: Time between proposal submission and sub-grant signing Sub-Grant Time between Proposal Submission and SMOG Signing Kenya (Component 1) MEA (Sub-grant Round 2) 13 months Surehatch (Sub-grant Round 2) 10.5 months AATF (Sub-grant Round 1) 10 months Purdue University (Sub-grant Round 1) 6.5 months Netafim (Sub-grant Round 1) 6 months Mozambique (Component 2) EMCL/ETG 7 months NCBA/CLUSA 6 months Opportunity International 5 months iDE 5 months In MEA’s case, the long negotiations resulted in the equipment procurement bid being withdrawn, so when the award was made, the firm had to start the procurement process over. For several companies, the long negotiation meant that the start of a crucial growing season passed. In both cases, sub-grant plans then needed to be adjusted, at times significantly, sometimes though modifications. Some noted that others in their firm questioned USG-imposed requirements that seemed unrelated to business success. Cost verification was seen as onerous by several private sector informants and as intrusive by two. NGOs, however, were accustomed to providing the level and type of detail requested. It was the first time for each of the three NGOs with P4I sub-grants in Mozambique to go through a performance-based agreement under USAID funding. While each reached an agreement with Fintrac, all P4I Midterm Performance Evaluation Report | November 2015 20 three noted difficulty in the process. One NGO staff member noted that there was a steep learning curve for the contracting and implementing units in their organizations. Informants in two of the private sector firms recounted that there was a strong champion who convinced others to continue with the grant negotiations when they were frustrated by the long process, multiple requests for additional information, and perceived high risk of upfront costs with no guarantee of getting paid for milestones. One said, “We had a tough time with the negotiations. We almost dropped out a few times.” Informants from another firm and from one NGO told of similar situations. An informant from one firm said that they would not do this type of agreement again. Negotiating Milestones: Representatives from many of the sub-grantees indicated that they were surprised by the milestone structure, which were seen by some as being imposed after they thought agreements had been nearly reached. One private sector sub-grantee said, “When milestones came up, we had no idea what that meant” and said their team “used all kinds of energy” trying to quickly figure out what kinds of milestones might make sense. Several noted that they would have written their proposal and budget differently had they known what kinds of milestones they would be. Many saw it as difficult to try to retro-fit their budget into a milestone structure. While negotiations around milestones were described as difficult, in all cases agreement was reached. One NGO representative commented on the milestone structure saying, “Initially, I didn’t like this, but I have come to appreciate that it is a good way to ensure deliverables.” Several informants from both private firms and non-profits who were involved in milestone negotiations with P4I said that they had to work hard to convince their organization’s leadership or board to accept the milestone structure. Push-back was particularly strong in NGOs that were not accustomed to taking on the risk of doing development work and potentially, in the face of circumstances beyond their control, not getting reimbursed for the work they had done in good faith. This was particularly an issue for sub-grants that required substantive early investments (for training, promotion, etc.) and that would not receive the larger payments for meeting sales milestones until much later. Most informants indicated that, while Fintrac negotiators were tough, they were also fair and that they worked to take the applicants’ concerns into account. Several made comments similar to what one informant said: “We didn’t get what we wanted from the negotiations or the milestones, but Fintrac probably didn’t either, so the outcome was probably fair.” While one informant described Fintrac staff involved in the negotiations as “very strict and rigid,” others described relationships improving over time. 3. Some Means of Verification for Meeting Milestones seen as Burdensome: While P4I staff reported that private sector firms should not be expected to do monitoring, the MoV for some milestones appears to be similar to (if not more detailed than) output monitoring commonly used in other types of USAID contracts and cooperative agreements. For some sub-grantees, the MoV for some milestones were seen as a burden and as a source of stress well beyond the completion of the milestones themselves, and two informants labeled the detailed information requested as the MoV as “micro-management” of the process. There are a variety of different levels of detail required for MoV of purchasers in different SMOGs. It was not apparent to the evaluation team why this was so. In some cases the MoV were similar to those used by USAID to verify expenses under cost reimbursement contracts. In other cases sales targets were required to be verified with details about the buyer and their farming practices. As an example, in P4I Midterm Performance Evaluation Report | November 2015 21 one SMOG, the MoV for the set of milestones on the number of smallholder farmers accessing improved agricultural inputs includes “digital phots of input agent’s [sic] records indicating smallholder customer’s [sic] name, contact information, gender, location, volume and value purchased of each input.” The cumulative number of farmers for the last milestone is 10,000, and the information on those 10,000 farmers will need to be collected by people who are not employees of the sub-grantee or sub-partners. Some sub-grantees who dealt with low-cost/high volume technologies that are sold through agrodealers or others reported concern about getting those third parties to collect the detailed information required for MoV. As one person in Kenya put it, recording MoV for smallholder input purchasers during planting season “isn’t practical. Everyone pushes up to the counter at once, and there is no time to record data on farmers.” Another informant, whose MoV required name and contact information, said, “We can’t push [agrodealers] that far or they won’t sell our products.” In Mozambique, two sub￾grantees expressed concerns about how to get third parties – particularly those in remote areas – to accurately collect and record and then keep track of the required data. The above said, some sub-grantees indicated that some of the information they were being required to collect would be useful to them. Two of the sub-grantees, Surehatch and the Netafim/Amarin team, needed some of the information included in the MoV, such as name and phone number, so that they could provide training and support on the use of their products. ETG staff noted that a sub-partner is designing software to text farmers advertisements or encouragement to sell their harvest to ETG as well as information on commodity prices and weather. The other two private sector sub-grantees expressed interest in exploring using the collected contact information for push marketing. 4. Milestone Renegotiation: Milestones have already been renegotiated for two of the four Component 2 sub-grants in Mozambique (iDE and NCBA/CLUSA) as well as for three of the five Component 1 sub-grants in Kenya (MEA, Netafim, and AATF). A total of 11 modifications were made to the SMOGs of those five sub-grants. To give more time to achieve milestone outputs, the AOR and Fintrac have made several SMOG modifications that shift from having specific due dates to requiring milestone targets achieved “as completed” but prior to some later date (often the end of the sub-grant). Several sub-grantees underscored the importance and appreciation for P4I flexibility and responsiveness, noting, among other things, that “listening and flexibility are very important – the local partner knows the situation on the ground” and they “they were willing to work with us.” One sub-grantee informant said, “it would have been difficult if we had to stay with the original milestones” and that “P4I staff were “willing to listen to realities on the ground.” Another said, “Maybe we were just excited when we got the grant and didn’t think through everything” and went on to compliment P4I staff on quick turn￾arounds for agreed-upon changes to SMOG milestones. 5. Repercussions from Failure to Meet Milestones: Many of the sub-grantee representatives expressed concern that they would not be able to collect and report all of the information required for their SMOG MoV. Attitudes ranged from saying that they were “losing sleep” over it (two sub-grantee informants mention this) to saying they were sure that Fintrac would understand that the milestones and/or MoV were not feasible and that there would not be any actual consequences for not complying. The evaluation team asked managers from each sub-grantee what would happen if they missed a milestone deadline or did not provide all of the information required for the MoV. While P4I asserted that information on consequences of failure to meet milestones was provided to sub-grantees, in both P4I Midterm Performance Evaluation Report | November 2015 22 Mozambique and Kenya there was a widespread lack of clarity on this. Several said that they were unsure. Two said that this lack of clarity increased their level of stress. Others said that they were sure they knew what Fintrac would do, and each described a different scenario. The most common response was the expectation that Fintrac would pay for the percentage of the milestone that was reached; for instance, if 80 percent of the target sales were reached, Fintrac would provide 80 percent of the payment allocated to that milestone. Opinions differed on whether the remaining funds would be lost to the sub-grantee or whether they would be held temporarily and then paid upon subsequent attainment of the target numbers. Two sub-grantees thought that they would not get paid for missed milestones. A representative from one large firm that was awarded a sub-grant said that they assumed they would not get paid at all but that their firm could absorb the loss. Representatives from smaller firms and NGOs appeared to be more concerned about the repercussions of not being paid at all. Level of concern appeared to be based in part on how the sub-grant fit into the organization’s current cash flow and future plans. Representatives from three private sub-grantees said that, while it would be a cash flow problem if they were not paid for a milestone, they were focused on building a market for the future of their business. Two firms said that they could weather the loss. Some of the small second-level sub-partners expressed concern about implications for the sustainability of their business should they not receive the expected funds to reimburse them for the work they would have already done towards milestones. Two sub-grantees are large firms with offices in 25 or more countries. Representatives from non-profits expressed higher levels of concern about the risks of non-payment of milestones. The Fintrac modification to the Netafim sub-grant has had all four of its milestones on financed sales of irrigation systems modified to “as completed but prior to end date [of the sub-grant].” The final full sales target, divided across four milestones, is to have $10.0 million in financed sales by mid-September 2015. Present financed sales are under $350,000, a fraction of the overall target. A P4I staff member noted that the modification continued to give Netafim/Amiran incentives to achieve the milestones. However, it may also have allowed Netafim and P4I to reduce attention to milestone completion at a time when attention and possible grant adjustment to address the performance challenge is critical to avoid failure of the sub-grant. While P4I staff has done some proactive work on helping to address the challenges Netafim faced with facilitating access to credit for irrigation system purchasers, this has not been successful. 6. Unclear Language and Inconsistencies in some SMOGs: While P4I staff were almost universally complimented on responsiveness and fast turn-around times, there were cases where it appeared that additional attention to detail in the language of SMOGs, which are legal agreements, would have been useful. 6.1 Clarity: In a few cases, the language in an element of a SMOG does not provide a clear guide for performance. For example, the wording of a few MoV was unclear on what needed to be provided. As an example, it was unclear in two SMOGs whether the MOV required the sub-grantee to report the amount purchased, name, and detailed location and contact information of the household-level purchaser of the supported technology or of a middleperson or of a retailer seller of the technology. 6.2 Consistency: Several SMOGs reviewed contained apparent inconsistencies across milestones and/or MoV. For instance, in one SMOG the MoV for the six sales milestones are identical except that four of them include “customer’s name” (which can be burdensome to collect for low-cost/high-volume items), P4I Midterm Performance Evaluation Report | November 2015 23 and two of them do not. In one SMOG, the sub-grantee was directed to track sales in two different ways for two different types of reports. In several cases, the selection criteria listed in the SMOG were different from the selection criteria listed in the Negotiation Memo. 6.4 Calculations: One Negotiation Memo contained errors leading to an incorrect travel cost estimate. 7. New Rounds of P4I Grant Competitions: The P4I activity ends September 30, 2017. The fourth round of Component 1 sub-grant competitions is underway, and agreements are being explored and reached with additional Missions for Component 2. Given the length of time that SMOG negotiation has taken, a new Component 2 competition would face a sub-grant period perhaps as short as 12 months, depending on Mission mobilization and funding availability. Evaluation Questions 2a, 2b, and 2c  Evaluation Question 2a: To what extent have private sector grantees stayed aligned with their business plans?  Evaluation Question 2b: To what extent have private sector grantees been able to commercialize their technologies?  Evaluation Question 2c: What challenges are sub-grantees facing in meeting their milestones towards successfully and sustainability commercializing their technologies? Evaluation Questions 2a, 2b, and 2c cover the performance of the P4I sub-grants reviewed for this evaluation. Responses to these questions are summarized in Tables 3 and 4 for each Component 1 sub￾grant in Kenya. Mozambique Component 2 sub-grants had only been under implementation five to seven months at the time the interviews were conducted. Beyond noting that all had completed their initial “soft” planning milestones, there is little of substance to report on Questions 2a and 2b. For Question 2c, brief notes are provided in Table 5 on initial challenges faced by Mozambique sub-grantees. P4I Midterm Performance Evaluation Report | November 2015 24 Table 3: Component 1 Sub-grant in Kenya: Milestone Progress and Modifications Name Sub￾grantee is a private sector firm Concept Second￾level sub￾grantee Firm said needed non-profit lead Milestones # originally expected to reach by 6/1/159 # reached by 6/1/15 % of expected reached # of modifi￾cations # of mile￾stones modified10 % modified11 Component 1 Round 1 African Agricultural Foundation (AATF) No Commercialize improved maize variety to combat Striga BASF, CIMMYT For Kenya work: Kenya Seed, Freshco Yes 23 18 78% 3 63 12 added, 40 changed, 5 unchanged Purdue University No PICS crop storage bags to reduce pest-related post￾harvest losses Bell Industries Needed for first USAID grant, but now knows how to work with USAID 7 7 100% 0 0 0 Netafim Yes Drip irrigation kits financed through loans Amiran & Connexus N/A 5 212 40% 2 9 67% Component 1 Round 2 Surehatch Yes Small-scale poultry egg incubators Surehatch Kenya N/A 6 6 100% 0 0 0 MEA Yes Biofix legume inoculants WeRATE N/A 5 3 60% 2 4 22% 9 Based on the original SMOG 10 If a milestone was modified in more than one SMOG modification, each time it was modified is counted as a modification here. A milestone is only counted as changed once per modification (e.g., if date, milestone language and means of verification language are all changed, then it is counted as 1 changed milestone). 11 Percentage of the total number of original milestones that were modified. 12 While this milestone is counted as having been reached, the financing program design that met the milestone was not implemented by the subject financial institutions. P4I Midterm Performance Evaluation Report | November 2015 25 Table 4: Component 1 Sub-grant in Kenya; Evaluation Questions 2A, 2B, and 2C Sub￾grantee Notes on 2A Notes on 2B Notes on 2C Sub-grant Round 1 African Agricultural Foundation (AATF) In general, AATF has stayed aligned with the business plan outlined in the SMOG, though the progress toward milestones has been slower than expected in Kenya. AATF was able to work with P4I to adjust milestones and had received three SMOG modifications by May 31st, 2015, the most recent of which was in March 2015. Modifications included extending the sub-grant period of performance, increasing the number of milestones from 45 to 57, extending most sales tonnage milestones to “as completed, but prior to [the end of the sub-grant],” and changing the MoV for seed sales from extensive details about famers who made purchases to records of seed company sales to distributors, retailers, and sales agents. While production and sales have increased, to date sales in Kenya are slightly below the original targets. The later than expected start of the sub-grant, along with poor weather and crop disease outbreaks, meant that the seed companies in Kenya were not able to produce as much seed as was needed for the early milestones. Dependence on other organizations with less stake in the outcome of the sub-grant was cited as a challenge. An AATF informant speculated that the low price for the IR seed that was negotiated with one of the seed companies had led that company to produce less seed than was expected by AATF. Purdue University The work under this grant appears to have stayed well￾aligned with the business plan The private sector second-level sub￾grantee was able to substantively increase commercialization of their technology, well beyond SMOG targets. The sub-grant helped Bell Industries improve marketing. Getting information on all the details about purchasers required for the MoV was cited as a substantive challenge. There were some early challenges in securing a manufacturer to produce the bags. Netafim Netafim has had difficulties in meeting targets of financed sales of drip irrigation kits because no Kenyan financial service providers have as yet introduced a financing mechanism for small-scale irrigation purchase. While Netafim and Amarin have been able to take advantage of an existing loan instrument, it had limited scope, and relatively few farmers were able to access it Due to the absence of financial instruments, sales meeting sub-grant criteria are far below targeted levels. In May 2015, it was expected that the first milestone of “$500,000 in sales of smallholder drip irrigation kits with a portion of each sale financed through the new loan program” would likely be reached but that the remaining milestones totaling an additional $9.5million in financed sales would not be met. However, an Amarin informant expressed confidence that the protocol it had developed for tracking and supporting small-scale drip irrigation kits would help it increase sales in the future. The primary challenge faced by this sub-grant was that the banks that the partners thought were lined up to participate and thought had agreed to provide loans had not actually fully committed to doing so. Connexus provided a full-time staff person (based at Amarin) to support negotiations with banks, but no agreements have yet been accomplished. This appears to be one of the few sub-grants where the performance based milestone agreement was not used as a flow-down to all partners. P4I Midterm Performance Evaluation Report | November 2015 26 Sub￾grantee Notes on 2A Notes on 2B Notes on 2C Sub-grant Round 2 Surehatch Overall, Surehatch’s work has been in general alignment with the plan laid out in the SMOG. In part because an incubator represents a relatively large investment for a smallholder farmer, early Surehatch work focused on more on training people on how to improve their poultry raising practices and business skills than had been expected, but Surehatch staff indicated that this investment is paying off. Surehatch has sold approximately 200 small-scale incubators and trained over 1000 Kenyans, mostly women, on raising poultry as a business. Early sales were slightly lower than expected, but sales have increased.  During SMOG negotiations, P4I and Surehatch realized that the market that Surehatch targeted in its proposal was not the smallholder farmers that USAID had targeted for access to innovative technologies. Surehatch had planned its work based on a definition of smallholder farmer that was a considerably larger farmer than is covered by USAID’s definition, and Surehatch had planned to work in some areas outside the FTF zones of influence. Surehatch was agile in redesigning its work to focus on the required target market and FTF zones of influence. However, this meant that Surehatch (which had planned its work based on a definition of small-holder farmer that is targeted by P4I) is having to invest much more than originally anticipated in training small-scale chicken producers to help them earn money to invest the relatively large amount needed to purchase an incubator.  Training events are more popular than anticipated, which has increased Surehatch’s training costs.  Surehatch faces challenges in assuring that the customers who purchase incubators are smallholder farmers as defined by USAID. MEA Because of the lag time between proposal and SMOG finalization, MEA was unable to get a key piece of equipment from the supplier originally arranged. MEA has experienced substantive delays in scale-up due to seeking a new supplier and waiting for delivery and customs clearance. Two SMOG modifications had, among other things, each pushed back the milestone for the date of installation from early February 2015 to May 4 to May 29. As of the evaluation team’s last interview with MEA (June 5th) the equipment had not yet arrived, and the milestone had been missed. While installation and production are behind schedule, MEA staff expressed confidence that the delay would not affect their ability to produce enough BIOFIX to have the supply needed to meet their sales milestones for the next rainy season. Demonstration plots, trainings and promotion are proceeding as planned. As the needed equipment has not yet arrived or been installed, production of BIOFIX has not yet increased. However, work is moving ahead on the establishment of demonstration plots and farmer training and sales agents are continuing to promote BIOFIX to agrodealers and to farmers. Later than expected signature of the SMOG created delays because of the need to re-procure the equipment needed for scale-up. MEA staff expressed concerns about agrovet dealers’ willingness and ability to collect names and contact information from all people who purchase BIOFIX packets of 50gm or less. P4I Midterm Performance Evaluation Report | November 2015 27 Table 5: Component 2 Sub-grants in Mozambique: Milestone Progress and Modifications Sub￾grantee Sub￾grantee is a private sector firm Concept Second￾level sub￾partners Firm said needed non￾profit Milestones Notes on Evaluation Question 2C13 # originally expected to reach by 6/1/15 # reached by 6/1/15 % of expected reached # of modi fica￾tions # of mile stones modi￾fied % modi￾fied EMCL/ ETG Yes Storage; purchase; advice Agro Tractors & Technobrain N/A (sub￾grantee is a firm) 2 2 100% 0 0 0 ETG expressed confidence about overcoming the challenges of obtaining permissions for construction of hubs. iDE No Capacity building; access to inputs; Lusosem & HUB unknown 3 3 100% 1 2 14% While, as the FTF zones of influence where iDE is working are poorly served by formal agrodealers, it was a challenge to find 40 business people who sell agricultural inputs, iDE did meet this milestone. NCBA/ CLUSA No Seed systems, fertilizers, inoculants Oruwera & Phoenix Seeds No 3 2 66% 3 6 43%  One of the two second-level sub-partners names in the proposal had to be replaced in early 2015. Oruwera was brought in through SMOG Modification 3 in mid-March. However, no changes were made to the SMOG to reflect that Oruwera was a different company with capacities different from the original firm’s.  NCBA/CLUSA representatives indicated that it will be a challenge to collect MoV information on all smallholder purchasers.  A second-level sub-partner noted it would be a challenge to fulfill the total required tonnage of seed sales in part because of the high demand for sesame seed, which is expensive for farmers but which weighs very little and thus contributes little to the required milestone weights. Opportunity International (OI) No Sesame and soybean value chains Banco Oportuni￾dade de Mozambique (BOM) Yes 3 3 100% 0 0 0 A key challenge faced by BOM is that disclosure of some information on the identities of loan recipients required in the SMOG is prohibited by law (as well as by financial services best practices). A BOM representative indicted that OI and BOM had tried to negotiate MoV that did not include the protected information but that they had been unsuccessful. The representative expressed confidence that, when BOM submitted milestone verification information without the protected information, Fintrac still would accept the milestones as complete. 13 The sub-grants in Mozambique had not been in operation long enough for there to be substantive responses to 2A and 2B. P4I Midterm Performance Evaluation Report | November 2015 28 Conclusions: Evaluation Question 2 1. Overall Utility of the SMOG Model Funding Structure 1.1 SMOG Design: Overall, the SMOG funding structure has been an effective mechanism that has supported most of the interviewed sub-grantees to work towards their objectives for scaling up smallholder access to their technologies. However, there are some issues with SMOG negotiation process and content that should be addressed and some adjustments that could make the SMOGs more effective. The utility of a mechanism like the SMOG is greatest when the funded sub-activity meets the Fixed Amount Awards criteria, particularly when risks are low and the milestone achievements are predictable in terms of timing and level of achievements. 1.2 Appropriateness of the SMOG: A SMOG agreement may not be appropriate for all of the funded activities and sub-grantees. a) Non-profit and private sector cost/cost recovery structures: Private sector firm and NGO sub-grant recipients face different cost and incentive structures in the implementation of the funded activities. A substantive difference is that private sector sub-grantees have income from sales supported through the sub-grant to at least partially cover their expenses, while NGOs are fully reliant on the grant to cover their good faith expenditures and thus face a greater risk than do private sector firms. The SMOG does not take into account this important difference. b) Risk and Uncertainty: USAID guidance is that Fixed Amount Awards (of which the SMOG is functionally a sub-type) are appropriate when desired milestones can be priced with a reasonable degree of certainty. The level and type of external risks inherent in some sub-grant innovations that support FTF goals and some applications for P4I funding may mean that a SMOG (at least as currently designed) is not the most appropriate mechanism for funding the proposed work. 2. Room for Improvement in the SMOG Negotiation Process 2.1 Imperfect Understanding of the SMOG and Milestones: A central challenge for most sub-grantees was what they perceived as an unexpected imposition of specific milestones after the negotiation process was well underway. Milestones and associated budgets were essentially retrofitted into proposed activities. While P4I subsequent willingness to modify was seen as positive, as evidenced by the number and type of SMOG modifications, the design of the milestones was not always completely successful. 2.2 Appropriate Milestones: Stretch targets may be most appropriate for types of grants where the grantee does not face substantial financial risk if quantitative targets are not 100% achieved. 2.3 Length of Negotiation: The longer-than-anticipated time and the type and level of due diligence information needed for negotiation of budgets and other SMOGs terms contributed to frustration, particularly on the part of private sector sub-grantees. Some delays had implications for sub-grantee’s ability to meet milestones, particularly those aligned with phases of the agricultural calendar. 3. Means of Verification: MoV that include information such as name, contact information, size and location of land holdings, etc. are more feasible to collect when the item being sold is relatively costly and complex and is being sold to a relatively small number of purchasers. That is, the relative cost per sale of collecting MoV for low cost items is much higher than the cost of tracking fewer, higher cost items. For instance, the burden of collecting detailed information on the targeted 250 purchasers of poultry egg incubators is less than the burden of collecting similarly detailed information on 10,000 purchasers of agro P4I Midterm Performance Evaluation Report | November 2015 29 input kits. The utility of some MoV data is unclear. When private sector firms have the interest and capabilities to use some of the MoV data for subsequent marketing, collecting it becomes more attractive. The lower level of detail for MoV in Component 2 SMOGs may reflect MOV design lessons learned. 4. Milestone Renegotiation: The common need for modifications to SMOGs, particularly adjustments to milestone due dates – many of which have occurred relatively early in sub-grant implementation – and the number of modifications that some sub-grants have received is an indicator that the process of designing milestones needs to be improved. 4.1 Protocol for Modifications: It is not clear whether P4I followed a standard approach to modifications or whether each modification was done ad hoc with a specific focus on the sub-grant at hand alone. 4.2 “As Completed” Due Dates: The relatively recent frequency of modifying milestone target dates to “as completed but before [X date or the end of the grant period]” appears to be a reaction to sub￾grantee difficulties in meeting milestone due dates. While this type of due date may be expedient, it may not improve the prospects that the milestone is achieved or reduce risk to the sub-grantee (or to P4I). Shifting the preponderance of production and/or sales milestones to the end of the period of performance appears to undermine the purpose of a milestone-based grant. 5. Failure to Meet Milestones: Many sub-grantees (and their second-level sub-partner) are operating under a lack of clarity on the repercussions of failing to fully achieve milestones. This impedes their ability to effectively manage risk. 6. Clarity and Consistency in SMOG Documents: The SMOG is a legal document that governs the use of USAID funds (from just over $100,000 to over $6 million). The lack of clarity and consistency (which appear to be due to unclear writing or editing errors) can lead to different parties having different interpretations of how the contract must be fulfilled and can create more work for the parties. 7. Evaluation Questions 2a, 2b, and 2c: Most on-going sub-grants that were reviewed are aligned with their business plans and have been able to work towards commercializing their technologies. (The Netafim sub-grant is a notable exception, where lack of bank buy-in to provide financing for smallholders has limited Netafim’s ability to meet milestones.) All sub-grantees interviewed expressed appreciation for and understanding of the utility of a performance-based funding mechanism. However, a variety of issues with the P4I SMOGs were found. Despite the overall conclusion that the private sector firms were able to increase smallholder access to their technologies through SMOG-based sub-grants, these issues indicate that other types of grant mechanisms might be appropriate in some cases. Recommendations: Evaluation Question 2 1. Overall Utility of the SMOG Model Funding Structure 1.1 Improve Fintrac Grants Manual for SMOGs a) Improve SMOG Alignment with FAAs: P4I should review and realign the SMOG template and other guidance in the SMOG Grants Manual with specific reference to both lessons learned and USAID guidance on Fixed Amount Awards in ADS 303.3.25. This section of the ADS is supported by “Fixed P4I Midterm Performance Evaluation Report | November 2015 30 Amount Award Entity Eligibility Checklist: A Mandatory Reference for ADS Chapter 303”14 and “Fixed Amount Awards to Non-Governmental Organizations: An Additional Help Document for ADS Chapter 303.”15 Make adjustments to the template and Grants Manual as appropriate. b) Include Risk and Complexity Criteria: P4I’s protocols for review of applications and due diligence should include risk and complexity criteria that would allow for use of other contract types or rejection for sub-grants with high external risks. Alternatively, explore options to reduce risk and complexity. c) Prepare Guidance on and Improve Design and Modification of Milestones i. Standardize Milestones: P4I should prepare guidance on design and modification of milestones based on a review of the milestones used in all P4I grants to determine which types of milestones are most effective for incentivizing and tracking commercialization and/or Component 2 Mission objectives. It may be useful to review milestones used in similar grant mechanisms. Consider creating a template for use across awards to ensure correct formatting for later use in GIS applications and push marketing. ii. Work with Sub-grantees to Assure Milestones are Feasible: The IP should support sub-grantees to assess the types of milestones that might be most appropriate for their sub-activity, the risks and benefits of different milestones, and MoV, including providing examples of different types of milestones and MoV. Assess the utility of taking a “tough” stance in negotiations for what applicants view as stretch targets. Milestone setting should take into account 1) the level of change possible in the time and budget available and 2) the level of change needed to boost the technology’s potential for successful and sustainable commercialization as well as 3) the need to contribute to USAID FTF objectives. iii. Milestone Due Dates: The IP should consider the effect that milestone due dates that include the end of the grant period may have on a sub-grantee’s likelihood to achieve targets and the risk borne by the sub-grantee and the IP and should adjust them as appropriate. iv. Assure means of verification are appropriate for the activity measured: P4I should simplify the MoV, particularly for data on sales. MoV should be simple; focused on what is needed to verify achievement of the milestone and provide information to USAID; consistent across sub-grants as appropriate; and feasible and cost-effective for the sub-grantee to collect and report. Determine what level of detail is actually needed for verification of milestones related to sales of low-cost, high-sales volume products (see Question 3). In particular, assess under what circumstances the level of effort needed to collect detailed data on people who purchase small items is necessary and worthwhile. Where feasible, modify the SMOGs of sub-grantees from early rounds who are still held to unnecessarily burdensome MoV. 14 USAID, “Fixed Amount Award Entity Eligibility Checklist.” https://www.usaid.gov/sites/default/files/documents/1868/303mak.pdf 15 USAID. “Fixed Amount Awards to Non-Governmental Organizations” https://www.usaid.gov/sites/default/files/documents/1868/303saj.pdf P4I Midterm Performance Evaluation Report | November 2015 31 v. Milestone Modification: When making modifications to milestones, consider making more substantive changes to the milestones beyond postponing due dates. It may be more effective for P4I to work collaboratively with the sub-grantee to determine whether the target should be modified (e.g., downward adjustments to selected targets, potentially accompanied by a downward adjustment in the associated payment). This can be done with or without modifying final totals. 1.2 Alternatives to the SMOG a) Modified SMOG: P4I should explore whether it would be appropriate to design a modified version of the SMOG instrument for non-profit sub-grantees or whether cost-reimbursable grants would be more appropriate for funding some desired sub-grants. b) Other Grant Mechanisms: As part of the due diligence and the risk assessments for sub-grant awards, when levels of risks of failure to meet milestones are being assessed, high levels of risk should lead to consideration of whether a SMOG agreement is appropriate. Consider using a different type of agreement, such as a cost reimbursable grant, for undertakings with high-levels of inherent internal risk (such as the risks faced in seed production or high levels of dependence on actors outside of the control or influence of the sub-grantee). 2. Improve the SMOG negotiation process 2.1 Strengthen Transparency and Information: The IP should assure that the expectations for a milestone￾based fixed obligation grant mechanism should be transparent and fully understood by applicants from the beginning of the solicitation process. This will allow applicants to design their activities, budgets and targets holistically. The IP should provide 1) Detailed information on how milestone-based agreements work, with examples of the types of milestones typically used and 2) A range for the amount of time typically taken between submission of the full application and the signing of the agreement (particularly important for activities closely linked to the local agricultural calendar). 2.2 Align Expectations with Agricultural Calendar: The IP should assure that each sub-grant start date and commercialization milestones are aligned with the agricultural calendar such that work can be carried out as proposed and planned. If aligning the start date is not possible, then planned work, milestone content or dates, and/or period of performance should be adjusted as necessary to accommodate the start date. 2.3 Response to failure to meet milestones: The IP should make sure, during negotiation, that sub-grantees and their sub-partners fully understand what will happen if they do not fully achieve a milestone on time. EVALUATION QUESTION 3: Assessing Benefits to Smallholders: How can P4I improve its ability to assess whether P4I Component 1 and 2 grants and partner investments are increasing smallholder men and women farmers’ access to and use of targeted technologies? Overview of P4I’s Monitoring and Evaluation Plan and Indicator Tracking Fintrac has an approved overall Monitoring and Evaluation Plan for P4I (M&E Plan: February 2014) that sets out requirements for its performance. All indicators are tracked through P4I’s PMP Target Tracker (an Excel workbook) and are reported to USAID/BFS through periodic reports (see Table 6). According to Fintrac staff, the Cooperative Agreement for P4I requires data on commercialization and private investment and does not require gathering data on farm-level outcomes. P4I Midterm Performance Evaluation Report | November 2015 32 Table 6: P4I Monitoring and Evaluation Plan Indicators and their Data Indicator Notes 4.5.2-12: Number of public￾private partnerships formed as a result of FTF assistance These required indicators under P4I’s Cooperative Agreement measure the outputs and outcomes of the fact that grants with match requirements are made and sub-grantee co-funding is leveraged.  4.5.2-12: Information is generated by P4I  4.5.2-38: Information is collected through sub-grantee reporting on funds leveraged. 4.5.2-38: Value of new private sector investment in the agriculture sector or food chain 4.4.2-39: Number of technologies or management practices made available for transfer as a result of USG assistance. These three indicators track technologies made available and commercialized under Components 1 and 2 and P4I’s work under Component 3.  Information for all three of these indicators is compiled directly by P4I staff. Custom: Number of technologies commercialized in FTF countries Custom: Number of investment design/models developed as a result of USG assistance 4.5.2-2: Number of hectares under improved technologies or management practices as a result of USG assistance These two indicators address aspects of farmer access and use of supported technologies. The data on which the reported information is based is collected by the sub-grantees, primarily as required in the MoV for the milestones in their SMOG. Then P4I uses various means of calculating the numbers that are reported. Data for these indicators are subject to the challenges regarding identifying number of individuals, households and women that are described in the findings below.  4.5.2-2: For nearly all of the data collected to date for this indicator, P4I applies a multiplier to sales data reported. For instance, applying a multiplier for the number of hectares of land that are assumed to be planted by a kilogram of a given seed. The one exception to this is Netafim which reports actual land area under drip irrigation. The EMCL/ETG does have in its plans to report on actual land area as well.  4.5.2-5: This is calculated in two ways. In some cases P4I uses the number of farmers who have purchased a technology and in other cases P4I applies a multiplier. For instance, for the Purdue PICS bags, Purdue reported the number of individual purchasers and the number bags bought by bulk purchasers. P4I uses the number of individuals who purchased PICS bags and adds the number of bags bought by bulk purchasers divided by 4 (assuming an average of 4 bags per farmer). In other cases, the calculations are more abstracted. For instance for AATF, P4I calculates the acreage of farmland that could be planted by the number of kilos of seeds sold (reporting that under indicator 4.52-2) and then assumes that each acre of land is worked by two farmers. In all cases, purchasers are assumed to have fully applied the technology purchased. 4.4.2-5: Number of farmers and others who have applied new technologies or management practices as a result of USG assistance Each Mission that buys into Component 2 has the option of adding indicators to meet their internal monitoring and reporting needs and to track information on the Mission-specific sub-grants, The P4I M&E Plan for Mozambique contains a total of 14 indicators, six of which are also in the overall P4I M&E Plan (P4I’s custom indicator on design of investment models is not relevant for Mozambique) and eight of which were additions (see Table 7; see also the P4I results framework in Annex VII). P4I Midterm Performance Evaluation Report | November 2015 33 Table 7: Additional Indicators for Mozambique Component 2 P4I Indicator for Component 2 in Mozambique Notes 4.5.2-43: Number of firms (excluding farms) or Civil Society Organizations (CSOs) engaged in agricultural and manufacturing and food security related services now operating more profitably (at or above cost) because of USG assistance Information for this indicator can be collected directly from the sub-grantees on their and their sub-partners’ financial status and from Opportunity International’s second-level sub￾partner’s information on loan recipients. 4.5.2-29: Value of agricultural and rural loans Data for these three indicators will come primarily from the Opportunity International sub-grant and will be collected as part of the second-level sub￾partner’s routine operations. 4.5.2-30: Number of MSMEs, including farmers, receiving USG assistance to access loans CUSTOM: Number of people with a savings account or mobile money account as a result of USG assistance 4.5.2-13: Number of rural households benefitting directly from USG intervention Data for this indicator is relatively straightforward to measure by using training registration sheets. 4.5.2-7: Number of individuals who have received USG supported short-term agricultural sector productivity or food security training Data used as input for these three indicators will be collected as part of the MoV of milestones in the sub-grant SMOGs and are subject to the challenges regarding identifying number of individuals, households and women that are described in the findings below. 4.5.2-37: Number of MSMEs, including farmers, receiving business development services from USG assisted sources GNDR-2: Percentage of female participants in USG-assisted programs designed to increase access to productive economic resources Findings - Evaluation Question 3 1. Information on Smallholder Farmers: Some sub-grantees collect information on smallholder farmers buying their technologies. In other cases it cannot be clearly determined, based on the data reported to P4I, whether sales are actually being made to smallholders. Where agents are used to sell the P4I partner inputs, such as seeds or inoculants, there are not always specific sales records for small holder farmer purchases. Partners and agrodealers state that many or most sales are to smallholder farmers. The MoV in two SMOGs (OI and EMCL/ETG) require collection of information on farm size. Other sub-grants use the size/quantity of the product sold as a proxy for farm size. MEA’s MoV specify counting sales of small packages of inoculant: 10 gram, 20 gram and “half of total sales of 50 gram packs.” The latter is to take into account that this size is not purchased solely by small farmers(reports on the percentage purchased by small farmers differ among staff). Netafim’s SMOG (modification 2) specifies that sales of drip irrigation kits of “500m2, .5 hectare and .5 acre” qualify to be reported. While the requirement that customers access finance for their irrigation kit purchase as a condition of being counted towards the milestones does help narrow the target market, the kits are still a relatively large purchase for a smallholder farmer, even with financing. Drip irrigation kit size is a poor proxy for farm size or smallholder status; kits of this size could also be purchased by people interested in testing the technology or in growing vegetables on part of their land. The Surehatch SMOG uses incubator size as a proxy for the purchaser qualifying as a smallholder. This is problematic as the technology is designed to work well for (and has elsewhere been marketed for) semi-urban areas for non-farm and non-poor households, and some portion of the sales appear to be to farmers who are not smallholders. Four of the sub-grantee’s technologies are specifically developed to appeal to smallholders (as are OI’s financial services). For the Purdue’s PICS bags, a plausible assumption was made that the technology P4I Midterm Performance Evaluation Report | November 2015 34 would primarily be appealing and useful for smallholders. iDE has developed small input “kits” of basic inputs, and, similarly, NCBA/CLUSA has developed small input “packages,” both targeted to smallholders. Both sub-grants include training for local people to promote and support use of the inputs. The remaining sub-grant, AATF, does not include specifications specific to smallholder farmers. AATF reports on tonnage of seeds sold. No data on farmer use or impact is collected under P4I. 2. Information on Women Smallholder Farmers: The program description in Fintrac’s Cooperative Agreement for the P4I activity includes the following statements:  “In line with FTF’s commitment to gender equality, the [P4I] Activity will consider impacts of each partnership on women’s and men’s access to new technologies and opportunities in smallholder commercial agriculture.” [p. 11]  “In order to ensure that the [P4I] Activity supports gender equality, a priority under FTF, special consideration will be given to partnerships with women‘s organizations, or with organizations that have significant female participation … and analysis will be performed on the effects of different partnership models on the participation of women. Technologies supported under this project will also be evaluated for impact or access issues arising from differentiated gender roles.” [p. 12] The P4I M&E Plan and P4I periodic reports to USAID do not appear to include attention to these points. The agreement also says, “The activity will also contribute to FTF‘s priority of gender equity by ensuring that increased opportunities for women are integrated into each sub-activity.” [p. 12] Fintrac reported that during sub-grant solicitation, “applicants are required to comment on the potential impact of their technology on women and how they will target women as a potential market segment.” It is not clear how Fintrac monitors or reports on whether this is done or how successful it is in practice. Cooperative Agreement language under Component 1 includes, "In keeping with USAID’s commitment to gender equality, companies should have codes of conduct in place or be able to show substantive plans to implement and enforce good practices in this regard.” [p.14] The evaluation team did not receive materials from Fintrac that included information on this aspect of due diligence. Cooperative Agreement language under Component 2 includes, “All partners will be expected to ensure opportunities for women’s participation, and to demonstrate the effectiveness of those efforts. For example, multinational partners that source from smallholder producer groups should require that those groups actively recruit female members and have women in decision-making roles.” [p. 16 – emphasis added] While MoV for milestones in Component 2 sub-grants did not appear to include information to verify this, it is possible that P4I is requiring this information through other means. 2.1 Gender in Implementation: While P4I uses a gender criterion as part of ranking sub-grant applications, there appears to be little integration of gender in the implementation of P4I. 2.2 Data on Women; P4I currently has no mechanism that effectively tracks sex-disaggregated data on technology purchase, on use of the technology within the household, or on benefits derived from its use and no explicit means of tracking or demonstrating outcomes for women. a) Collecting Data on Women: Purchase vs. Use: Access to and use of the P4I supported technologies by women does not appear to be being tracked effectively. Almost all SMOGs have MoV that require collection of information on “gender” for sales and/or training (and some for demonstration plots). P4I Midterm Performance Evaluation Report | November 2015 35 The data on the sex of the purchaser collected at the point of sale is problematic. The sex of the person actually making the purchase may have little or no relation to the sex of the person who learned about the item, the sex of the person who made the decision to spend the money to purchase the item, or the sex of the person who will actually use the item. This is a particular issue in Mozambique where many informants reported traveling long distances (up to 80 kilometers) to get to an agricultural input dealer. In such situations, the household member with a reason or flexibility to travel might be the one making the purchase. Some informants reported that, the more expensive an item, the more likely that it would be a man who makes the purchase. A family or community group may also delegate one individual to make a purchase, which will then be shared. b) Reporting Data on Women: Sex-disaggregation should only be used where it is meaningful. Tracking and disaggregating data based on the sex of people trained through sub-grant activities provides meaningful data. Tracking and reporting data on FTF Indicator 4.5.2-5 “Number of farmers and others who have applied improved technologies or management practices as a results of USG assistance” based on the sex of the person who made a purchase is not meaningful. 3. P4I Monitoring and Evaluation Plan: Most of the monitoring information reported in P4I periodic reports appears sound, although there are substantive improvements that can be made. 3.1 Data on Use of Promoted Technologies: Some indicators being reported do not accurately represent the number of smallholder men and women farmers accessing and, more importantly, applying promoted technologies. In the overall P4I M&E Plan, the custom indicator “Number of technologies commercialized in FF countries,” includes the assumption “P4I will consider a technology commercialized when the subaward is signed.” This means that the technology is reported to USAID as “commercialized” well before any item supported by P4I is sold to a smallholder. 3.2 PIRS: The overall P4I M&E Plan (February 2014) does not include PIRS. In the P4I M&E Plan for Mozambique, there are minor irregularities in a few of the PIRS. For instance, the PIRS for FTF Indicator 4.5.2-2 states “P4I will use the assumption of ___ [sic] hectares per farmer to report on this indicator.” Text for two of the PIRS goes well beyond the page margins, so the full PIRS cannot be read. 3.3 Gender: In the overall P4I Monitoring and Evaluation Plan, gender is not mentioned and data disaggregation by sex is mentioned only once, when it is stated in the indicator table that for FTF Indicator 4.5.2-2 hectares under improved technologies or management will not be disaggregated by sex “unless by land ownership ratios in the country” (p. 11). While the PIRS for Indicator 4.5.2-5 for the P4I Monitoring and Evaluation Plan for Mozambique does include disaggregation by sex, the text where this is specified is beyond the limits of the printable and viewable page of the “Final” February 2015 version that was provided to the evaluation team. (This PIRS also appears to conflate sales of a technology and application of the technology.) In the PMP Target Tracker excel workbook (June 23, 2015) provided to the evaluation team by P4I, the spaces for disaggregation by data for the nine sub-grants covered by this evaluation were blank, and data did not appear to have been disaggregated by sex. 3.4 Calculated Data: Farm-level reporting on farmer application of new technologies and on hectares under new technology is not based on direct field observations or farm-collected data. Some reporting on smallholder men and women farmers’ access to and use of targeted technologies is based on data reported by sub-grantees (that in many cases is collected by third parties based on sometimes P4I Midterm Performance Evaluation Report | November 2015 36 problematic milestone MoV) and/or derived using indirect estimates based on proxies and assumptions whose validity has not always been fully tested. A few calculations described in the P4I M&E Plan and used in the PMP Target Tracker for 4.5.2-2 and 4.4.2-5 appear to provide incorrect estimates of outputs. See Table 6 above on calculations and proxies. As another example, the both P4I M&E Plans’ table on Post-Award Reporting and Data Collection (p. 7 in the overall M&E Plan, p.10 in the M&E Plan for Mozambique, and Table 8 below and described to the team by P4I staff) appears to be problematic. Table 8: Post-Award Reporting and Data Collection (from the P4I M&E Plan) 4.5.2-5 Number of Farmers and others who have applied new technologies or management practices as a result of USG assistance. Year 1 Year 2 Year 3 Year 4 Year 5 Total New 50 100 0 0 0 150 Continuing 0 25 75 75 75 250 Total 50 125 75 75 75 400 It is valid to assume that some proportion of beneficiary farmers will continue to apply the technologies made available through P4I. However, the table shows the farmers continuing to be counted towards P4I indicators across the five year P4I period of performance (in some cases well past the life of the sub￾grant) and illustrates a calculation of total “number of farmers and other who have applied new technologies or management practices as a result of USG assistance” that counts some farmers five times. Thus, the 150 farmers who were actually calculated to have applied a new technology during the life of a given sub-grant are shown as having become 400 farmers by the end of P4I. While individual farmers can be counted and reported again in subsequent years, they should not be multiplied into additional farmers by adding the totals of each year together to get a cumulative total for the life of P4I. This same set of calculations also appears to be applied in situations where it is not relevant, such as for the calculation of metric tons of seed sold or number of PICS bags sold. In this case, it does make sense to total up the tons of seed sold each year to a cumulative total of seeds sold. However, it is not valid to assume that farmers will continue to purchase the same amount of seeds or the same number of PICS bags in years 3, 4 and 5 that they did in Year 2. Note that this also assumes that the firms will continue to sustainably sell the technologies. The practice of estimating and adding up yearly totals to get cumulative totals described here appears to be being used in P4I reporting on indicators 4.5.2-2 (number of hectares) and 4.5.2-5 (number of farmers and others). 4. Means of Verification: Reporting Opportunities and Challenges: Some data for reporting on FTF indicators is provided by sub-grantees. For instance, data on hectares under improved technologies and number of farmers applying technologies is provided through sub-grantee reporting on milestone MoV. (Challenges inherent in sub-grantee collection of data on sales milestone MoV are also discussed under Evaluation Question 2 and under “Gender” above.) 4.1 Data Collected by Sub-grantees and Sub-partners: Data on end users is reasonably strong for the P4I awards where sales and distribution from the awardee to the end users are under the direct control of sub-grantees and their second-level partners (e.g., Netafim and Surehatch - and potentially OI and ETG). The two sub-grantees whose technology is both relatively expensive and complex (Netafim with its drip irrigation systems and Surehatch with its egg incubators) had systems that directly collected the types of information requested for milestone MoV and for the P4I M&E system. Because these firms actively use P4I Midterm Performance Evaluation Report | November 2015 37 purchaser information as part of their business plan, their data is likely to be reasonably accurate. Because it works with loans, OI’s second-level sub-partner Banco de Oportunidade de Mozambique will need to collect the data required for the MoV in its SMOG. Importantly (as noted elsewhere), by law Banco de Oportunidade cannot report to Fintrac all of the information required for its MoV. 4.2 Data Collected by Third Parties: Sales to farmers of technologies for three sub-grants in Kenya (AATF, MEA, and Purdue), as well for two sub-grants in Mozambique (NCBA/CLUSA and iDE), will be made by third- or fourth-party actors (e.g., agrodealers or entrepreneurial extensionists). As noted under Evaluation Question 2, collection of data on these types of sales is especially challenging, both because it covers low-cost/high-volume products and because those asked to conduct data collection have little experience in data collection or little stake in its use. 5. Data Validation: Because payment of grant funds to sub-grantees for many milestones is dependent on reaching sales targets, this type of grant carries built-in incentives for exaggerating sales. (Note that verification of data reported by sub-grantees was beyond the scope of this evaluation.) The MoV in the SMOG are designed, in part, to mitigate this risk. Conducting sales validation helps manage this risk. P4I staff initially told the evaluation team that P4I planned to undertake a sales validation survey (with a few simple questions on use) to verify reported sales data for selected sub-grants, P4I staff recently reported that P4I will do this for all sub-grants at or near completion. The only sales/end use validation survey completed at the time of evaluation field work is for the Purdue PICS bags sub-grant, through which an attempt was made to call and administer the survey to 131 buyers on a list provided by the sub-grantee. Through P4I’s survey, just 66% of buyers on that sales list were verified as having purchased PICS bags. Of the 131, there was no phone number or a wrong phone number for 24%, 2% said they did not receive a PICS bag, and 5% refused to be interviewed. Although this survey was small, data show a high level of farmer satisfaction with the PICS bags. Other data included  The range of number of bags sold to an individual or organization ranged from 1 to 18,000.  Many sales were to individuals and distributors outside of the FTF zones of influence (ZOI) (not a problem in this case as FTF ZOI were not required by the sub-grant solicitation or SMOG). 6. Reporting: Final reports are not required under the SMOG agreements reviewed. There are no comprehensive sub-grant summaries that put the milestones in context by setting out sub-partner achievements, challenges and how they were overcome, P4I impacts on the firm, and lessons learned. Final reports can help the partners identify USAID grant engagement that has made a difference for companies and their new small holder farmer clients. Several sub-grantees (primarily non-profits) indicated that they would like to assess and report on (and showcase for donors and others) what they accomplish beyond the milestones in the SMOGs. Several Mission staff mentioned that they would value a report on accomplishments and on challenges faced and overcome along with lessons learned. Such reports would also provide P4I and USAID/BFS with a means for focusing on process improvement. 7. Sub-grant recipients indicated internal interest in data on smallholder access and use 7.1 Data Collection by Private Sector Firms: Several Fintrac and USAID staff said that private businesses do not want to collect monitoring data and should not be required to do so. However, representatives of all four of the private sector firms that received sub-grants indicated that detailed information on the P4I Midterm Performance Evaluation Report | November 2015 38 end-users of their technology and on how useful the technology was to them would be helpful in growing their business. Two of the three firms in Kenya were already doing so and the firm in Mozambique was developing detailed plans on how it would collect and use the data. Two private sector firms sub￾partnering to non-profits indicated interest in collecting data on end-users to improve sales. 7.2 NGO Data Collection: Three of the four sub-grantees in Mozambique are NGOs who, for their own internal monitoring purposes, are planning to apply their organization’s results-based tracking system to measure outputs and results independent of P4I’s own requirements. 8. Anecdotal Evidence of Farmer Access and Use of P4I Technologies: The team interviewed a small sample of agrodealers and farmers. In Kenya, some agrodealers cited customers who returned to make additional purchases as evidence of farmer use. One agrodealer said they would have purchased a larger inventory had they had the liquidity to do so, and another said they would have purchased a larger inventory had more of the item been available. The few farmers interviewed who had used one of the technologies confirmed use and benefit. Mozambique sub-grantee sales are still incipient. Conclusions: Evaluation Question 3 1. Smallholder Farmers: Effective targeting of and reach to smallholder farmers is critical to P4I’s success. Targeting of smallholder farmers and tracking whether they are being reached is done in a wide variety of ways, which appear to vary in their effectiveness. Yet, sales did happen and the products moved from agro-dealer stores. 2. Women: It is difficult for P4I to monitor, make needed adjustments, and demonstrate outcomes for women when women’s access to and use of the promoted technologies is not being effectively monitored. Data on the sex of buyers of most reviewed technologies is not a valid measure of access to or use of technology by men or women and should not be submitted for disaggregated reporting under standard FTF indicators. Appropriately, P4I does not appear to be using data from most sub-grantees on the sex of the buyer as a proxy for the level at which P4I is reaching women smallholder farmers. 3. P4I Monitoring and Evaluation Plan 3.1 Data on Technology Use: Gaps in the set of standard FTF and custom indicators in the P4I M&E Plans limit P4I’s ability to report on central Component 1 and 2 outputs and achievements. The P4I M&E system is currently not well set up to assess whether P4I sub-grants and partner investments are increasing smallholder men and women farmers’ access to and use of targeted technologies. Adequate data is collected to determine program outcomes for private investment to support FTF. Distribution and sales data for agrodealers and farmer sales are reasonable measures of commercialization, however, the custom indicator on commercialization is not. In order to assess outcomes related to BFS and Missions’ FTF strategies, information on farmer-level outcomes and results is also necessary. 3.3 PIRS: The absence of PIRS in the overall P4I M&E Plan impedes P4I’s ability to consistently collect and report indicator data that has the expected integrity, reliability, precision and impedes USAID’s ability to interpret and appropriately aggregate the data. 3.3 Gender: It appears as though P4I is not reporting sex-disaggregated data to USAID for use in USAID PPRs, which is appropriate given the difficulties with that disaggregation. P4I Midterm Performance Evaluation Report | November 2015 39 3.4 Data Challenges: A few means of calculating data for indicators in the overall P4I M&E Plan and P4I M&E Plan for Mozambique appear to be problematic. There are challenges in the use of data provided by sub-grantees. There is not a perfect match between the data collected through milestone MoV (which, themselves, vary greatly among SMOGs) and the data needed for reporting on FTF indicators. 4. Utility and Challenges of Means of Verification Data 4.1 Data Collected by Sub-grantees: There are significant challenges to 1) collecting indicator data from multiple sub-grantees with varying levels of experience with indicator data collection and reporting and 2) accurately designing multipliers for estimating indicator data and aggregating across multiple sources. 4.2 Data collected by Third Parties; P4I sub-grants are designed to get innovations into the hands of smallholders with little access to technology. Several sub-grants are providing their innovative inputs to very small rural shopkeepers and/or training local people to act as promotors. Although some SMOGs require that these parties record detailed information on sales, shopkeepers in particular may have little incentive and limited means to do so. This is particularly of concern for the sub-grants reviewed here as the number of end-users expected to receive P4I inputs/technologies from direct distribution is very small compared to the number receiving the technology from third-parties. These factors can compromise the quality of the data reported to sub-grantees and ultimately to USAID. 5. Data Validation: Sales validation surveys, such as the one that P4I conducted for the Purdue PICS sub-grant, can be an important means of verifying MoV data and of bolstering and improving the assumptions and formulas used as part of Fintrac’s reporting to USAID. As the survey confirmed just 66% of sales on the list provided by the sub-grantee, there may be issues regarding accuracy of sales data recoding or reporting. 6. Reporting and Results Tracking: Overall, P4I is funding innovations that have potential for furthering progress toward FTF objectives and that have potential as learning mechanisms (different from what is already being done under P4I Components 3 and 4). 7. Utility of M&E Data to Sub-grantees: Data on smallholder men and women farmers’ access to and use of technologies would be useful to a variety of stakeholders. Private sector firms may be more interested in collecting this type of data than IP staff have asserted, as long as the data has clear utility and is cost-effective to collect. Recommendations: Evaluation Question 3 Overall, P4I should consider increasing the resources allocated to M&E both in terms of having a staff person with M&E expertise and sufficient time to dedicate to M&E and in terms of funds. 1. Smallholder Farmers: Given the centrality of the importance to P4I of reaching smallholder farmers Fintrac should verify the purchases that can be clearly attributed to smallholder farmers. This should be done for all awards. 2. Women Smallholder Farmers 2.1 Gender and Implementation: Fintrac should provide more explicit attention to gender in the implementation of P4I. P4I should contract an external consultant to support P4I to do a gender assessment and/or provide recommendations on how to better include gender in its work and its output P4I Midterm Performance Evaluation Report | November 2015 40 and outcome tracking (e.g., improving MoV and outcome measures related to women smallholder farmers on current and new sub-grants), and to work with P4I to address 2.2 and 2.3 below. 2.2 Gender and M&E: Gender, and how outputs and outcomes for women will be monitored, should be explicitly addressed in the next update of the P4I M&E Plan and in Component 2 Mission M&E Plans. 2.3 Monitoring outcomes by Sex: Fintrac should give greater attention to accurate tracking outputs and outcomes by sex. P4I should work with sub-grantees to assure that they are appropriately and accurately collecting and reporting sex disaggregations and to design new options where they are not. Where sub￾grant implementers are collecting data on actual users of the supported technology (e.g., Surehatch and Netafim), if this data is being recorded well, it can be disaggregated by sex for reporting to USAID. MoV should be revised to improve utility of data on sex. (See also Data Verification below.) 3. P4I Monitoring and Evaluation Plan 3.1 Data on Technology Use: Consider including additional FTF standard and other indicators to the overall P4I Monitoring and Evaluation Plan to capture farm-level outcomes not currently being captured. Some indicators used in the P4I M&E Plan for Mozambique would be relatively straightforward to track with few, if any, modifications to SMOGs, and could be useful for the overall P4I M&E Plan and for up￾coming M&E Plans for P4I programming with Missions. Possible options include number of individuals trained and the number of individuals receiving loans and the value of these loans. Also, change the definition of the custom indicator on commercialization so that it reports on actual commercialization. 3.2 DQA: A data quality assessment (DQA) should be done by an experienced DQA specialist who can provide training or guidance to P4I staff - and sub-grantees - following the DQA could help resolve any outstanding measurement issues and improve the accuracy of standard FTF indicator data provided to USAID. The DQA should go well beyond the minimum DQA requirements and should include examination of original data and interviews with those responsible for both collecting and for cleaning and aggregating the data under the sub-grants with the most complex or problematic data collection and reporting. The DQA should also include suggestions for improving data collection and analysis. 3.4 PIRS: PIRS should be added to the overall P4I M&E Plan, and the PIRS used for P4I Component 2 should be reviewed and revised as necessary. 3.4 Gender: See recommendation 2 above. 3.5 Data Challenges: P4I should make sure that its collection and reporting of indicator data corresponds to what is stated in its M&E Plans. P4I should revise its M&E Plans and/or its data collection and reporting and assure that future Mission M&E Plans and PIRS are accurately designed. 4 & 5. Means of Verification, Data Validation, Monitoring and Evaluation 4/5.1 Sub-grantee Data Collection a) Utility of Means of Verification Data: During SMOG negotiations or modifications, the IP should work with sub-grantees to adapt MoV so that data is as useful as possible 1) to the sub-grantee for commercialization; 2) to the IP’s sub-grant management; and 3) to USAID reporting on FTF indicators. To ensure alignment, the M&E Plan for each set of Mission awards should be developed in conjunction with milestone development. P4I Midterm Performance Evaluation Report | November 2015 41 b) MoV Data Accuracy: P4I should work with sub-grantees to assure accuracy of MoV data collection and reporting. c) Mission Support for Mission Data: Under Component 2, the Mission should work with P4I and the sub￾grantees on how to better align the information collected for MoV in SMOG requirements and how P4I systems can most efficiently feed into Mission FTF reporting. To the extent that there are missing M&E systems and resources, the Mission could consider adding support for FTF reporting. d) Incorporate Cost of Means of Verification into Sub-grants: During SMOG negotiations, the IP should build into the sub-grant budget the funds needed for collecting the data needed for reporting to USAID - if the data required for sales milestone MoV are beyond what the private firm (sub-grantee or second￾level sub-partner) would normally collect for its own business interests. e) High Volume/Low Cost Technologies: The IP should give extra attention to the formulation of MoV for technologies that are designed to have high volume sales of relatively low cost items. If data on sales and/or use of a technology are expected to be collected by a third party, then the IP should work with the sub-grantee to design a clear, simple, cost-effective, and reliable protocol for data collection and cross-checks for data accuracy. If no such protocol is feasible, then the MoV should be changed to something that is feasible. In some cases, third parties can be relied on to gather the requested information. In other cases (such as the iDE sub-grant in Mozambique as well as for Netafim and Surehatch in Kenya), information can be gathered and used to follow up with purchasers as part of the sub-grant’s implementation. In other cases, verification of a sample of sales (e.g., through point of sale observations) can be built into the sub-grant design. f) Feasibility of Means of Verification: Where sub-grantees can make a clear case that reporting some types of MoV data is not feasible, P4I should work with the sub-grantee to redesign the MoV. 4/5.2 P4I and/or External Data Collection: Of the following, P4I should do a) along with b) and/or c). a) Mid-term Mini-surveys: Mid-term mini-surveys for sales validation should be conducted with a (where possible, random) sample of people who accessed the supported technologies to assess who is being reached and to acquire basic information on outcomes. P4I should plan, develop and implement output and outcome surveys on 1) verification of sale; 2) farmer use; 3) satisfaction; 4) production impacts of the technology; and 4) impact on food security. Such surveys can be based on the model that P4I has already used for sales validation (and would replace that survey). However, the mid-term mini-surveys should be conducted at the sub-grant’s mid-point to allow time for adjustments to implementation if needed. The mid-term mini-surveys should verify and validate data reported on sales milestones and should include a more in-depth set of questions on use of the technology and on outcomes of its use along with information on the sex of the technology purchaser, the sex of the technology user(s), and information on farm size and a proxy for level of poverty. b) Final assessments: Final assessments (carried out by Fintrac staff or a neutral third party) should be conducted of selected sub-grants, preferably by an independent external party. These assessments could be carried out for sub-grants from Rounds 1, 2 and 3 that would be completed before the beginning of Year 4 of the activity so that results and lessons learned can be applied during the life of P4I. These assessments could focus on the following points: 1) extent to which the sub-grant catalyzed purchase - and use - of the supported technology by smallholder men and women farmers; 2) extent to P4I Midterm Performance Evaluation Report | November 2015 42 which the technology was successfully used by men and women smallholder farmers; 3) extent to which the supported technology has been viably commercialized; 4) extent to which the sub-grant contributed to FTF objectives in that country; and 5) extent to which USAID engagement with the private sector was enhanced through the grant. (See also the final report under Evaluation Question 4.) c) Endline mini-survey: For sub-grants not selected for an external assessment, P4I should allocate funding for a statistically strong endline mini-survey that builds on the mid-term mini-survey to assess smallholder men and women farmers’ access to and use of targeted technologies. 4/5.3 Reporting and Outcome Tracking: One of more of the following should be done: a. Sub-grantee Final Reports: In the future, a final report on each sub-grant should be included as a final deliverable in the SMOG. The final report is an opportunity for the sub-grantee to summarize the achievements, outline the problems encountered, describe how the grant affected each sub-grantee and second-level sub-partner, and document good practices and lessons learned. This is an important opportunity for sub-grant implementers to report on what outcomes and results they accomplished that are not fully captured by milestones. The IP should prepare an annotated outline report that details the information that should be included in the report and the suggested length for each section. b. Fintrac Reports on Sub-grants: Fintrac should submit a final report on each sub-grant to USAID/BFS. It can build in the sub-grantee report and should, at a minimum, include the following information: sub￾grant amount; period of performance; information on the supported technology/implementation activities; sub-grantee and second-level sub-partner roles and responsibilities; milestone targets (content/date); milestone achievements; challenges faced and how they were addressed; sub-grant modifications and reasons for them; information from the mini-survey(s); achievements not measured under milestones; extent of commercialization and its sustainability; summary of end-of-sub-grant business plan or next steps toward commercialization, sustainability or business growth; financials; and lessons learned through awarding and/or managing the sub-grant. Fintrac should prepare a report outline for USAID’s review. Where a final report is a SMOG deliverable, P4I should include information from the sub-grantee’s report. This would be useful for assessing progress towards FTF goals overall and in the implementation country. c. Evaluation: To fulfill P4I’s potential for testing out approaches, to be more rigorous and intentional in assessing implementation, challenges and ways they were overcome, and to document outcomes, impacts and lessons learned, a comprehensive external, independent evaluation could be carried out by a neutral third party for selected P4I sub-grants. This evaluation should go beyond the sub-grantee’s final report to assess outcomes and results at the smallholder farmer level and to assess sustainability. (See also Evaluation Question 3.) Evaluation questions and SOWs should be co-designed with USAID. d. Review of Similar Programming: A comprehensive review of all USAID FTF innovation and commercialization programs and approaches, if this is not accomplished through P4I Components 3 and 4, could lead to a better understanding of the process and potential for USAID to improve the outcomes of P4I and the design of future programs. 6. Data Utility and Sub-grantees P4I should assess the content of milestone MoV, in collaboration with sub-grantees, to make sure they feasibly generate reliable data for both showing that milestones have been reached and for meeting P4I Midterm Performance Evaluation Report | November 2015 43 USAID’s reporting needs. This should include assessing cost-effectiveness of data collection and utility of data collected to sub-grant implementers along with P4I’s management and reporting data needs. Where appropriate, adjustments should be made (see also recommendations under Evaluation Question 2). EVALUATION QUESTION 4: Implementation: How might management of P4I (by Fintrac, BFS, and/or Missions) be improved to maximize P4I’s outputs and outcomes? Findings: Evaluation Question 4 1. Management by Fintrac 1.1 Planned and Actual Implementation a) Delay in Component 2 Implementation: Both USAID and Fintrac staff reported that the Fintrac understanding of Component 2 during the first year of P4I was not fully in line with USAID/BFS objectives and plans. That substantive misunderstanding was not resolved until the beginning of the second year of P4I implementation. Through revisions to the Second Year Work Plan, USAID and Fintrac were able to establish a common approach and plan. The first solicitation for competitive sub￾grants under Component 2 was issued in May, 2014. The evaluation team estimates that the misunderstanding delayed the award of Component 2 sub-grants by approximately one year.16 Implementation of Component 2 is now progressing well. Though P4I reports that it expects to expend its grant ceiling, P4I staff noted that the activity will be serving fewer Missions and awarding fewer and shorter grants than would have been possible had the misunderstanding not occurred. b) Significant Staff Turn-over in the Early Stages of P4I: There was a high rate of staff turn-over and gaps in staff coverage during the first 18 months of P4I implementation, including the Team Leaders for all four P4I components, causing significant impacts on both Component 1 and Component 2. For instance, for Component 1, two individuals were hired before Fintrac settled on the current Team Leader. This also meant that Component 1 Round 1 sub-grantees sometimes worked with several different P4I staff; in some cases, different staff members were reported to have different expectations for the sub-grantees. c) Staff Learning and Grant Award Trade-Offs: P4I staff noted that, as a means to develop staff understanding of the technology awards process, the core activity of P4I, Fintrac management chose to have a large set of staff members carry out field work on sub-grants. Management sought to have senior staff gain experience negotiating SMOGs and strengthen the SMOG through sharing experiences and learning. Seven different P4I staff members (all but two of whom are no longer with P4I) participated in the negotiation of the five sub-grants in Kenya. A P4I staff person cited the 16 Award of sub-grants under Component 2 is necessarily preceded by P4I negotiations with a Mission on how much buy-in the Mission will invest in P4I and how those funds should be directed and crafting an agreement with the Mission. In December 2013, P4I staff first attended a regional USAID event to promote Component 2 to Missions and to seek Missions interested in buying in to the P4I Component 2 mechanism for funding FTF PPPs. The time lapse between this event and the May 2014 issuance of the Mozambique solicitation for sub-grant proposals was six months. This was 18 months after the first solicitation for Component 1 sub-grants. The 12 month delay was calculated by subtracting the six months for identifying and working with the Mission to design the Component 2 mechanism from the 18 months between the emission of the solicitations. P4I Midterm Performance Evaluation Report | November 2015 44 variety of staff member approaches to SMOG negotiation as a source of inconsistencies in text and content among SMOG documents and as a source of the variety of SMOG negotiation experiences reported by sub-grantees. This approach did not work well in terms of sub-grant consistency, and much of the learning gained was lost as staff left. Sub-grantees were faced with a variety of program managers and inconsistent processes as the Component 1 manual was being developed. P4I reports that its approach had since been codified and that staff are provided hands-on training. d) Component 1: Informants from both USAID/BFS and P4I indicated that implementation of Component 1 started when expected and moved forward at a pace that was acceptable. P4I launched in September, 2012 and the first solicitation for competitive sub-grants was issued in November, 2012. The first sub-grants were awarded in the fourth quarter of FY13. While issues described in the two points immediately above affected the quality of the early Component 1 work, P4I staff have made advances in the quality of due diligence and SMOG negotiation. e) Level of P4I collaboration with USAID: USAID/BFS staff expressed interest in having stronger communication and collaboration with P4I staff and in having P4I facilitate (rather than “control”) USAID engagement with sub-grant stakeholders. USAID staff in BFS and Mozambique expressed interest in having increased interactive engagement with the sub-grantees and their sub-partners. 1.2 Sub-grantee Perception of Fintrac Management of P4I Sub-grants: Representatives from most sub￾grantees indicated that P4I staff took a supportive and positive approach to monitoring and verification of the grant milestones. In most cases, Fintrac was seen as a partner working in collaboration with the firms and NGOs to achieve sub-grant objectives. Sub-grantee representatives characterized P4I staff’s work with them in ways that ranged from “very helpful” and “tough but supportive” to “it felt like we were in the crossfire 90% of the time.” In a few cases, Fintrac was characterized as “micromanaging,” “rigid,” or “more interested in milestones” than in outcomes for small farmers. 1.3 See also Evaluation Question 1 findings 3.2, 3.3, and 4; Evaluation Question 2 findings 1.2, 2, 3, 5 and 6; and Evaluation Question 3 findings 2 and 3. 2. Management by USAID/BFS and USAID Missions: See also Evaluation Question 1 findings 1.1b, 1.2, 2.4, and 4.1b. 3. Grant Selection Criteria: The Component 1 Round 1 sub-grant solicitation materials and selection criteria led to selection of at least some sub-grants that did not mesh well with the value chains and/or ZOI prioritized in the Mission FTF strategy in the target country. In Kenya, all three Round 1 sub￾grants were able to count implementation activities outside the Kenya Mission’s FTF ZOI toward achievement of their milestones. In Round 2, this issue still existed. For instance, in addition to allowing Surehatch to count sales to people living outside the FTF ZOI towards achievement of its milestones, the technology Surehatch is commercializing did not fit within a value chain prioritized by the Mission. P4I revised the grant application structure and selection criteria after both sub-grant Round 1 and Round 2. The most important changes were adding as criteria that the proposed technology must be 1) within the value chains prioritized under the target country Mission’s FTF strategy and 2) commericialized to smallholder farmers within FTF ZOI. The revisions improved P4I’s ability to identify strong applications with improved potential for meshing well with the target countries’ FTF strategy. The revised criteria and structure - along with the addition of detail on P4I’s performance-based approach to grant management - also facilitated SMOG negotiations by improving the clarity with which applicants understood USAID and P4I Midterm Performance Evaluation Report | November 2015 45 P4I expectations. This assisted P4I to help shape the sub-grant activities to better contribute toward FTF objectives (where needed) and facilitated reaching agreement on sub-grant terms and budget. Conclusions: Evaluation Question 4 1. Management by Fintrac 1.1 Planned vs. Actual Implementation a) Delay in Component 2 Implementation: Implementation of Component 2 was not initially carried out as planned. The delay reduced the number of Missions P4I could work with and reduced the maximum possible length of the Mission-supported sub-grants. This reduced the overall possible impact of P4I. b) Staff turnover: While staff turnover and gaps in coverage did inhibit some progress at the beginning of the activity, staff composition appears to have stabilized. c) Staff and Grant Development Decisions: SMOGs are legal documents that have a substantive impact on the sub-grantees and their sub-partners. While it can be of value to have more than one staff person able to carry out key tasks, assigning a variety of staff to negotiations can create inconsistencies in how sub-partners are treated, how they perceive the activity, and in the content of the SMOGs. 1.2 Level of P4I collaboration with USAID: USAID and P4I staff appear to have different expectations about the level and type of collaboration and communication (between each other and among USAID, Fintrac and sub-grant stakeholders) deemed appropriate and optimal. Some current means of communication, such as the periodic sub-grant phone calls, could be used to greater advantage. 1.3 See also Evaluation Question 1 conclusions 1.3 and 2.2a; Evaluation Question 2 conclusions 2, 3, 4, 5, and 6; and Evaluation Question 3 conclusion 3. 2. Management by USAID/BFS and USAID Missions See also Evaluation Question 1 conclusions 1.3, 2.2, and 4.2. 3. Grant Selection Process 3.1. Grant selection criteria a) Sub-grantee Understanding of Milestones: P4I staff indicated that grant application materials and communications with potential sub-grantees during vetting and negotiation processes make the concept of milestones (and definitions of some other key terms such as smallholder farmer) abundantly clear. However, many sub-grantees indicated that the import and impact of milestone reporting and payments was a surprise to them or not well understood not only during the application process, but also during the early parts of the negotiation process. b) Alignment with Mission Strategy: While some early sub-grants were not aligned with Mission FTF strategies, by Round 3, changes in grant application and selection criteria led to better alignment. 3.2 Risk of market distortion: In sectors where there are few players or players that are large relative to the size of the market, the award of funds to a private sector firm can sometimes constitute picking a winner, creating market distortion or exacerbating a potentially monopolistic situation. P4I Midterm Performance Evaluation Report | November 2015 46 Recommendations: Evaluation Question 4 1. Management by Fintrac 1.1 Level of P4I collaboration with USAID: P4I and the recently appointed USAID/BFS AOR should work together to arrive at a shared understanding of communication and collaboration expectations and protocols. This could include elements such as type and frequency of communication and other engagement between Mission staff and Component 1 and 2 sub-grantees and second-level sub-partners; frequency of informal updates provided to the USAID/BFS AOR; etc. 1.2 Communication: To enhance USAID/private sector engagement, on periodic phone calls with P4I, USAID, and sub-grantees, P4I should create more space for active USAID and private sector participation. 1.3 Engagement: USAID/BFS and P4I should agree on a common definition of USAID engagement with the private sector and a plan on how P4I implementation will facilitate and increase that engagement. 1.4. See also Evaluation Question 1 recommendation 1 and Question 3 recommendations 2, 3 and 4/5.2a. 2. Management by USAID/BFS and USAID Missions 1.1 Engagement: USAID/BFS and P4I should agree on a common definition of USAID engagement with the private sector and a plan on how P4I implementation will facilitate and increase that engagement 1.2 See also Evaluation Question 1 recommendation 1 and Evaluation Question 3 recommendation 3. 3. Grant Selection Process 3.1 Assure Sub-grantee Full Understanding of Milestones: P4I should review sub-grant solicitation and application materials and other communications materials and design ways to further clarify for potential applicants how the milestone-based grant agreements will operate in practice. Provide examples of milestones and MoV to help applicants and sub-grantees prepare for negotiations to develop appropriate milestones that meet the needs of USAD, P4I, and private sector and non-profit sub-grantees. Assure that these materials and early communications very clearly define key eligibility concepts, such as the definition of the target population and eligible implementation locations. 3.2 Align Sub-grant Selection Criteria with Mission Strategy: For future similar programs, sub-grant selection criteria should assure alignment with program objectives. The efficacy of selection criteria should be periodically reviewed, and adjustments should be made as needed. 3.2 Risk of market distortion: Selection of grant recipients should take into account balancing the need for market penetration into remote areas, working small holder farmers and supporting viable businesses with contributing to market distortion and monopoly or monopsony. P4I Midterm Performance Evaluation Report | November 2015 47 ANNEXES ANNEX I: EVALUATION STATEMENT OF WORK C.2 SCOPE OF WORK 1) Objectives The TAP BFS-PE TO will support four PEs of BFS centrally-funded/centrally-managed projects or activities. 2) Proposed Activities To fulfill Evaluation Policy and ADS 203 requirements, the TAP BFS-PE TO Contractor shall, in coordination with BFS, manage, design, and implement four PEs of BFS Feed the Future centrally funded and centrally-managed activities. The TAP BFS-PE TO Contractor shall utilize BFS guidance and templates for the management, design, and implementation of PEs. Given the range and diverse nature of the activities implemented by the targeted BFS mechanisms, expertise across a range of research, M&E and capacity-strengthening food security-related areas will be required, including organizational and staff/consultant expertise in the use of both quantitative and qualitative methodologies in conducting PEs. The BFS centrally-managed mechanisms to be evaluated are listed below. The parameters of each PE are contained in Attachment J.2. Activity Final Performance Evaluation African Cocoa Initiative: The project develops the cocoa sector by fostering public-private cooperative investments in cocoa, improving the genetic quality and productivity of the cocoa, expanding farmer education and training programs, and improving cocoa input supply chains. It works in Cameroon, Côte d’Ivoire, Ghana, Nigeria, and Togo. (http://worldcocoafoundation.org/wcf-african-cocoa-initiative/) RESAKSS Analysis and Capacity Building: This is a consortium of CGIAR and regional organizations being led by IFPRI to deepen and broaden available food policy information and analysis, as well as networks of food security and policy researchers and practitioners both in country and regionally. (http://www.resakss.org/) Solutions for African Food Enterprises (SAFE): SAFE provides customized technical assistance to select food processors with the aim of increasing the competitiveness of the African food processing sector and to expand availability of affordable and nutritious foods. SAFE is carriedout through an alliance between Partners in Food Solutions (PFS) and TechnoServe. PFS is a non-profit organization formed by General Mills and includes technical expertise from General Mills, Cargill, Royal DMS, and most recently Buhler. PFS provides remote-based, demand driven customized technical assistance to selected food processors which is carried out by TechnoServe, as well as broader trainings on key industry knowledge gaps delivered to larger sector wide groups through seminars. TechnoServe provides the on-the-ground food technology specialists and business advisors to facilitate the on-site customized technical assistance recommended by PFS. SAFE builds on the structures put in place by a two-year predecessor program, the African Alliance for Improved Food Processing (AAIFP) which ended November 30, 2012. SAFE operates in Kenya, Malawi, and Zambia (http://www.partnersinfoodsolutions.com/) P4I Midterm Performance Evaluation Report | November 2015 48 Activity Midterm Performance Evaluation FTF Partnering for Innovation: This cooperative agreement is focused on finding and commercializing agricultural technologies that have the potential to impact smallholders and supporting private sector engagement within Feed the Future. The program works to engage the private sector to commercialize technologies to the smallholder farmer market and builds and procures public-private partnerships that support Feed the Future efforts. (www.partneringforinnovation.org) C.3 TASKS For each Performance Evaluation, the Contractor shall:  Design, plan, and implement independent rigorous PEs of BFS centrally managed activities todetermine project or activity performance and/or impacts, focusing on progress, successes, obstacles, constraints, and areas of need and for improvement.  Conduct data collection and analysis, and produce reports and briefings on PE findings that follow USAID guidance and format.  Contribute to building the capacities of host country institutions (government, NGO, and private sector) to conduct PEs by including local institutions and individuals in all phases of evaluation design and implementation, to the maximum extent feasible. In addition to the requirements listed above, the Contractor shall: 1. Confer with USAID and with the Implementing Partner to create the statement of work for the PE, including identifying the PE questions and evaluation methods to be used (within two weeks of PE start date) 2. Identify the evaluation team members and obtain BFS approval (within three weeks of PE start date) 3. Develop the PE Plan and obtain BFS approval (within three weeks of PE team’s start date) 4. Conduct desk review of required documents (timing of remaining steps determined in PE Plan) 5. Compile performance reporting data 6. Conduct stakeholder, key informant and/or focus group interviews in person or via phone or email 7. Conduct country visits to identified country(ies), as applicable 8. Conduct field visits while in country, as applicable 9. Provide exit briefing(s) to USAID mission(s) 10. Produce up to two versions of draft PE report, incorporating USAID feedback 11. Provide in-person or telephone briefing on the draft report to BFS and to the implementing partner 12. Produce the final PE report 13. Present the final PE report to USAID and partners in Washington DC 14. Provide data required for Open Data posting. P4I Midterm Performance Evaluation Report | November 2015 49 ANNEX II: EVALUATION METHODS AND LIMITATIONS METHODOLOGY FOR QUANTITATIVE AND QUALITATIVE DATA COLLECTION This mid-term performance evaluation used a process evaluation approach based in qualitative data. A process evaluation is designed to analyze how an activity operates and what factors (internal and external) appear to be most and least successful in forwarding the expected outcomes. As requested by BFS, this evaluation was designed to focus on P4I Components 1 and 2 as they have been implemented in Kenya and Mozambique respectively. Rigorously and systematically analyzed qualitative data was at the core of the evaluation design. Qualitative data was used to provide robust, objective evidence to answer the evaluation questions. Qualitative questions addressed the experiences, perceptions, opinions, motivations, and knowledge of the stakeholders interviewed, the context within which the activity is operating; and the factors (causal mechanisms) leading to observed results (or non-results). In-depth conversations guided by structured and semi-structured key informant interview instruments provided the opportunity for a flow of information that is not constrained by pre-determined response categories. This allowed for exploration of subjects that surfaced during the interviews and elicited information that provided nuance and insight for addressing the evaluation questions. We applied qualitative techniques to elicit information from a variety of P4I stakeholders, including BFS, Missions, the implementing partner, sub-grant recipients (partners) and their sub-partners, and retailers selling the targeted technologies. Interviewing this wide variety of stakeholders allowed for triangulation of information on P4I processes and its progress from multiple perspectives and from people with a variety of different types of objectives for their involvement in P4I. As an example of data integration, where appropriate, secondary data from sub-partner monitoring records was reviewed during interviews with the sub-partners who collected the data to allow for real￾time discussion of monitoring methods and results. This was done both in light of sub-grant process and provided information for findings on monitoring systems. a. Data Collection Timing Data collection was carried out between April 16 and June 12, 2015. Much of the primary qualitative data and some of the secondary data was collected during field work in Mozambique (May 18 – May 26) and Kenya (May 27 – June 5). b. Data Collection Methods Qualitative data was collected primarily through key informant interviews and observation. Secondary data sources were also reviewed. This section briefly describes the methods and sampling plan that was used. Refer to Table 4 for an illustration of the data collection methods that was used for each evaluation question. Before departing for data collection in Mozambique and Kenya, the evaluation team acquired an understanding of P4I’s goals, objectives, results framework, theory of change, inter-relationship among the components, implementation plans, and a basic understanding of the implementation history. This was accomplished through document review and interviews with BFS and P4I staff. In addition, the evaluation team observed part of the process of selecting candidates for the fourth round of sub-grants (see below) P4I Midterm Performance Evaluation Report | November 2015 50 and conducted key informant interviews with U.S.-based sub-grant partners for the implementation of Component 2 in Mozambique. There was just one round of data collection for this midterm performance evaluation. The evaluation team members were responsible for data collection. While no local data collectors were hired, an interpreter was brought in to assist with a sub-set of observations and opportunistic conversations in Mozambique. Key Informant Interviews: Key informant interviews (KIIs) were the primary data collection technique for this evaluation. Key Informant Interview Instruments: The KIIs captured information from the perspectives of the key stakeholders for P4I Component 1 and Component 2: P4I staff, sub-grant recipients and their sub￾partners, retailers, USAID/Mozambique and USAID/Kenya staff and BFS staff. The evaluation team designed a separate key informant interview guide (found in Appendix III) for each type of stakeholder. Each guide included a core set of questions that were asked of all key stakeholders. Each set of stakeholders were also asked a set of questions that was specific to their role in P4I. The questions in the interview guides were designed to elicit information to address the evaluation questions. Sampling Plan for Key Informant Interviews: The evaluation team conducted KIIs with the stakeholders listed below. The majority of interviews were conducted with individuals. In a few cases, two or three members of the same organization were interviewed together in a joint KII. In order to reduce potential contributors to bias in responses, these joint key informant interviews did not mix stakeholder groups. For example, interviews conducted with sub-partner institutions did not include partner institutions, and interviews with retailers did not include partner or sub-partner institution staff. 1. P4I staff: The evaluation team conducted KIIs with the P4I Chief of Party and with team members responsible for Components 1 and 2. 2. Partners (Sub-grant recipients): The evaluation team interviewed staff of each partner institution that received a sub-grant through P4I for Component 1 in Kenya and through Component 2 in Mozambique (see Table 1). Where it was not possible to conduct face-to-face meetings, interviews were conducted by phone. Table 1: Partners Sub-grant recipients a. Component 1 in Kenya b. Component 2 in Mozambique i. Perdue ii. Netafim iii. African Agricultural Technology Foundation (AATF) iv. Surehatch v. MEA Fertilizers i. Export Marketing Company Limited ii. International Development Enterprises (iDE) iii. National Cooperative Business Association/ Cooperative League of the United States of America International (NCBA/CLUSA) iv. Opportunity International 3. Sub-partners (organizations that work directly with sub-grant recipients): The evaluation team interviewed staff involved with P4I and with almost every sub-partner institutions that received funding through PFI for Component 1 in Kenya or through Component 2 in Mozambique (see Table 2). While the intention was to conduct interviews with as many sub-partners as possible, limited time in each P4I Midterm Performance Evaluation Report | November 2015 51 country and the distance between locations where different sub-partners operate resulted in the evaluation team using opportunistic sampling to prioritize meetings with sub-partners that are located close to each other geographically. Where it was not possible to conduct face-to-face meetings, other methods of collecting information (such as phone interviews) were used. Table 2: Sub-partners Component 1 in Kenya Component 2 in Mozambique i. Bell Industries (sub-partner of Perdue) ii. Amiran and Connexus (sub￾partners of Netafim) iii. One of the two seed companies with which AATF works i. Agro Tractors Limited and Technobrain Ltd. (sub￾partners of Export Marketing Company Limited) ii. Lososem Mocambique, Lda and HUB Asistencia Tecnica e Formacao (sub-partners of iDE) iii. Oruwera (sub-partner of NCBA CLUSA) (The team was unable to meet with Phoenix Seeds) iv. Banco Oportunidade de Mozambique (sub-partner of Opportunity International) 4. Retailers who sell technologies supported through P4I grants: Retailers were selected based on opportunistic sampling while the evaluation team was in the field. 5. Mission staff: Mission staff were identified based on level of involvement with P4I.  In Mozambique, this included Activity Managers for the Component 2 sub-grants in Mozambique and an M&E officer.  In Kenya, the team met with the director of the ABEO office and an Agriculture Team Leader and FTF Coordinator. The team also met with former USAID/Kenya staff who worked with FTF at the time the P4I sub-grants were awarded. 6. BFS staff: The evaluation team interviewed the current Agreement Officer’s Representative. The interim Agreement Officer’s Representative was not available to be interviewed, and the in-coming interim Agreement Officer’s Representative was not yet in place during the period when the evaluation team was collecting data. Observation Site Visits: The evaluation team traveled to selected sites where activities were taking place. To get a better understanding of how technologies and partners are selected as sub-grant recipients, the evaluation team observed parts of the meeting during which the P4I Technical Review Committee assesses and rates the applicants (as part of the fourth round of sub-grants) to arrive at selection of a short list of potential recipients. The team also conducted observations at selected retail outlets where supported technologies are sold. The site visits in the field allowed the team to confirm data found in project reports and/or KIIs as well as to make observations of market dynamics. During observation visits to retailers, the team selected farmers (male and female) to ask a brief set of questions about interest and an accessibility of technologies supported by P4I. The evaluation team also met with two groups of farmers and eight individual farmers who lived near or were involved in demonstration plots. P4I Midterm Performance Evaluation Report | November 2015 52 Secondary Data Document Review: The bulk of the secondary data for this evaluation was gathered from existing documents from BFS, the Missions, P4I and from sub-grant partners and sub-partners. These sources included both quantitative and qualitative data. Examples of types of documents included P4I award documents, and implementation documents and progress reports; documents on Mission public private partnerships; documents related to sub-grant solicitations and awards for the four rounds of Component 1; documents related to investment sub-grant solicitations and awards for Component 2; and documents on and from sub-grant recipients and their sub-partners. See the list of documents in Annex V for more detail. Monitoring Data: Data from the P4I performance monitoring system was reviewed in addition to data on sales that sub-grant recipients and their sub-partner had available. This was particularly important for addressing evaluation question 4. c. Evaluation Question Matrix Table 3 provides a summary of the data sources, data collection methods and data analysis methods that were used for each evaluation question. More detail on the data sources and collection and analysis methods, as well as on the limitations of the methodology and design described here, can be found in the report sub-sections following this table. P4I Midterm Performance Evaluation Report | November 2015 53 Table 3: Evaluation Question Matrix Evaluation Questions Data Sources17 Data Collection Methods Data Analysis Methods 1. P4I as USAID interface with the private sector: What are the strengths and weaknesses of having the centrally funded P4I activity support Missions to expand private sector engagement in FTF in Mozambique and Kenya? a. BFS staff b. USAID/Mozambique staff c. USAID/Kenya staff d. Funded partner staff in Mozambique and Kenya e. Funded partner sub￾partners in Mozambique and Kenya f. Fintrac staff g. Fintrac documents h. Partner documents a.- f. Key informant interviews g.- i. Document review a. – f. Qualitative data analysis g. – h. Compilation of information from documents 2. Effects from the perspective of partners: What are effects of the structure of the SMOG funding model on private sector firms’ ability to scale up smallholder access to their technologies? a. Funded partner staff in Mozambique and Kenya b. Funded partner sub￾partners in Mozambique and Kenya c. USAID/Mozambique staff d. USAID/Kenya staff e. Fintrac staff f. Fintrac documents g. Partner documents h. Farmers a.- e. Key informant interviews f. – g. Document review h. Opportunistic interviews (very small number) a. – e. Qualitative data analysis f. – g. Compilation of information from documents 3. Implementation: How might management of P4I (by Fintrac, BFS, and/or Missions) be improved to maximize the results? a. P4I staff b. Mission staff c. BFS staff d. Partners e. Sub-Partners a. – e. Key informant interviews a. – e. Qualitative data analysis 4. Assessing Benefits to Smallholders: How can P4I improve its ability to assess whether P4I Component 1 and 2 grants and partner investments are increasing smallholder men and women farmers’ access to and use of targeted technologies? a. P4I activity monitoring and evaluation plan and indicators b. Indicator data reported by partners in periodic reports c. Partners d. Sub-Partners e. Retailers who sell partners’ technology f. P4I staff g. Mission staff f. BFS staff a. - b. Document review c. - d. Review of partner and sub-partner records (as available) c. – d. Review of partners’, sub-partners’ and retailers records/data on purchases and purchasers (as possible and available) e. Review of retailers’ records/data on purchases and purchasers (as possible/available) c. - h. KIIs a -b. Compilation of information from documents b. – e. Review of data on purchases and purchasers c. - h. Qualitative data analysis 17 Note: Activities and data sources are subject to change based on contextual factors during work in Kenya and Mozambique. P4I Midterm Performance Evaluation Report | November 2015 54 METHODOLOGY FOR QUANTITATIVE AND QUALITATIVE DATA ANALYSIS a. Qualitative Data Analysis Qualitative data analysis involves the identification, examination, and interpretation of patterns and themes in textual data. The evaluation questions are the linchpins of the analysis. The analysis was focused on determining how these patterns and themes help answer the evaluation questions, and focused on emergent themes as they related to each evaluation question. The analysis approaches for each evaluation question are summarized in the Evaluation Question Matrix in Table 3 above. Where possible, data was disaggregated by gender. Opportunities were sought to interview women staff members of the sub-grant partners and sub-partners. Possible opportunities to improve gender disaggregation of data collected by partners and sub-partners were explored. The qualitative data for this evaluation was analyzed through a five-step process, as summarized below. Note that preliminary findings (see Step 3 below) for Mozambique were presented at the Mission outbriefing meeting. The Kenya Mission was unable to schedule a formal outbriefing meeting, so, while the evaluation team did discuss preliminary findings with two Mission staff, no formal outbrieinf meeting was provided. The team spent approximately eight days in each country being visited, so the preliminary findings presented at the outbriefing for the first country visited – Mozambique – were relatively limited. Five qualitative data analysis steps: In addressing the four evaluation questions, the evaluation team applied the steps below to analyze the qualitative data from the key informant interviews of the various stakeholders in the US, Kenya and Mozambique. Step 1: Process and Record Data Immediately: Upon completing an interview, we summarized our interview/discussion notes. This included notation of pertinent quotes and highlights related to each evaluation question covered during the interview. We also recorded other observations and made note of issues to follow-up on in subsequent interviews and other research work. Step 2: Begin Analyzing as Data is Being Collected: As data was being collected, we began reviewing the data and mentally processed it to identify themes or patterns. To facilitate this process, we held routine team “check-ins” during the fieldwork. This was done daily when the two team members were in the same location. During these check-ins, the team members discussed findings and interrogated the data. Step 3: Data Reduction: After data was collected, we undertook a data reduction process in order to identify and focus in on what is most meaningful. In this step, the team combed through the raw data to determine what was significant. The team continued the data reduction and analysis after their return to the US as well. Step 4: Identifying Meaningful Patterns and Themes: In order for qualitative data to be analyzable, it must first be grouped into the meaningful patterns and/or themes. This process is the core of qualitative data analysis, and it involves two analytical techniques: content analysis and thematic analysis. Content analysis is carried out by coding the data for certain words or content, identifying their patterns and interpreting their meanings. Coding is done by going through all of the text and labeling words, phrases, and sections of text (either using words or symbols) that relate to the evaluation questions. After the data was coded, the team sorted and examined the data by code to look for patterns, for example in sub-partner experiences with the milestone protocol for funding disbursements. Thematic P4I Midterm Performance Evaluation Report | November 2015 55 analysis involves taking the coded data and organizing it into themes that will help answer the evaluation questions. Given the wide heterogeneity of key informants and the relatively small number of interviews, coding and thematic analysis was done by hand. No data analysis software was used. Step 5: Conclusion Drawing and Verification: In this step, the team conducted a thorough review of the analysis and findings. This step is centered on an iterative process of review, reflection, and discussion to interpret what the findings mean, to determine how the findings help answer the evaluation questions, and to draw conclusions solely from the findings from the data collection. Based on the findings and conclusions, the team developed recommendations related to the evaluation questions for USAID and Fintrac to make improvements as needed. b. Quantitative Data Analysis While the bulk of the data for responding to the evaluation questions was qualitative, quantitative secondary data, such as data from the P4I monitoring and evaluation system and partner and sub-partner records, was also collected and analyzed. For instance, the evaluation team examined data collected by the partners and sub-partners and compared it for consistency with the data that is reported to P4I. OUTCOME MEASURES Because the evaluation is designed to address how P4I can be improved, the evaluation questions led to open-ended responses rather than to yes/no responses, numerical responses, or categorical responses. Necessarily, the findings based on this type of outcome measure were largely open-ended as well. Table 4 provides the evaluation questions, selected outcomes of interest, illustrative outcome measures, and the type of data that was used to respond to each P4I Midterm Performance Evaluation Report | November 2015 56 Table 4: Evaluation Questions, Outcomes and Outcome Measures Evaluation Questions Outcomes Illustrative Outcome Measures (to be disaggregated by component where appropriate) Data Collection Methods and Principal Data Sources for these outcome measures in Mozambique and Kenya 1. P4I as USAID interface with the private sector: What are the strengths and weaknesses of having the centrally funded P4I activity support Missions to expand private sector engagement in FTF in Mozambique and Kenya?  P4I is an effective mechanism for Mission engagement of the private sector, including alignment of FTF strategies of USAID/BFS, USAID/Kenya and USAID/Mozambique  Extent to which USAID staff hold a positive opinion of the P4I work under Components 1 & 2  Themes of pros, cons, and comments about the P4I model of engagement Key information interviews:  BFS staff  USAID/Mozambique staff  USAID/Kenya staff  Funded partner staff  Funded partner sub-partners \  Fintrac staff 2. Effects from the perspective of partners: What are effects of the structure of the SMOG funding model on private sector firms’ ability to scale up smallholder access to their technologies?  Sub-grant commercialization milestones are met/ technologies are commercialized and made available in the market/ technologies have potential to reach small holder farmers?  Extent to which sub-grant recipients in Kenya are meeting their commercialization milestones and staying aligned with the plan specified in their contract Key informant interviews  Funded partner staff  Funded partner sub-partners  USAID/Mozambique staff  USAID/Kenya staff  Fintrac staff Document review  Fintrac documents  Partner documents  Sub-grant awardees perceive the SMOG funding model as positive  Themes of pros, cons, and comments from sub-grant recipients on the content of their SMOG Key informant interviews  Funded partner staff  Funded partner sub-partners P4I Midterm Performance Evaluation Report | November 2015 57 Evaluation Questions Outcomes Illustrative Outcome Measures (to be disaggregated by component where appropriate) Data Collection Methods and Principal Data Sources for these outcome measures in Mozambique and Kenya 3. Implementation: How might management of P4I (by Fintrac, BFS, and/or Missions) be improved to maximize the results?  Management of P4I by Fintrac, BFS and missions is perceived as effective at maximizing P4I results  Extent of Mission staff and sub-grant awardee satisfaction with management of P4I  Themes of pros, cons, and comments expressed by sub-grant recipients and USAID staff on the management of P4I Key informant interviews  P4I staff  Mission staff  BFS staff  Funded partner staff  Funded partner sub-partners  P4I is implemented as planned  Types of issues found with implementation as planned (if any)  Extent to which implementation by Fintrac has gone as planned Key informant interviews  P4I staff  Mission staff  BFS staff  Funded partner staff  Funded partner sub-partners Document review  Fintrac documents  Partner documents 4. Assessing Benefits to Smallholders: How can P4I improve its ability to assess whether P4I Component 1 and 2 grants and partner investments are increasing smallholder men and women farmers’ access to and use of targeted technologies?  Sub-grantees effectively collect and report information needed to assess whether their work increased smallholder men and women farmers’ access to technology  Extent to which sub-grantees in Kenya are collecting data needed for milestone verification.  Extent to which sub-grantees express confidence in being able to collect different types of data that will be needed for milestone verification Key informant interviews  P4I staff  Mission staff  BFS staff  Partners  Sub-Partners  Retailers who sell partners’ technology Document review  Fintrac documents  Partner documents and monitoring records P4I Midterm Performance Evaluation Report | November 2015 58 METHODOLOGICAL STRENGTHS AND LIMITATIONS The evaluation methodology had significant strengths along with two main limitations (and a few minor limitations that should also be noted) that affect the ability to generalize based on findings. The evaluation design took these limitations into account. Site visits in two countries: As per the request for task order proposals and the contract, this evaluation work was centered on the implementation of two components of P4I in two countries: Component 1 in Kenya and Component 2 in Mozambique. This is both a strength and a limitation. Both countries were sound choices. Kenya had the highest concentration of Component I sub-grants. Five of the sixteen sub-grants that were awarded in 10 FTF countries were awarded in Kenya, including the only sub-grant to have been completed, and there has been a high concentration of proposals from organizations in Kenya. Kenya also offered the opportunity for comparison with other grant-making models. Mozambique was a clear choice for Component 2 as the Mission in Mozambique was the first to participate in P4I’s Component 2 work and as there are four public private partnerships supported by PFI that are currently being implemented. While work under Component 2 was progressing with other Missions, as the evaluation was designed, no other Missions had partnerships that were up and running. At the same time, because the opportunities and challenges were different in different country contexts, carrying out the evaluation work in just one country per component limited the ability to generalize conclusions. Distance and time: The evaluation covered nine grants that are operating in a variety of locations in two countries. The large distances between sites and the short amount of time in each country (approximately seven days in each) limited the evaluation team’s ability to meet with all of the stakeholders of interest, though the team as able to meet in person or by phone with nearly all of the sub-grantees and their second-level sub-partners. This mean that the findings, while robust, do not provide a fully comprehensive representation of P4I’s work and preliminary results. In the case of Mozambique, three of the four partners that received sub-grants were based in the U.S. (Washington DC, Colorado and Illinois), with one is in Maputo. While a few of their sub-partners are based in Maputo, most are in the northern and central parts of the country (Nampula, Zambezia, and Manica). While the evaluation team’s time in Mozambique was extended beyond the time originally envisioned (from one week to one-and-a-half weeks) to allow for some travel to the field, it was still not be possible to visit all of the sub-partner firms in the time available. The time in-country allowed for only one site visit, and Nampula was chosen because the largest concentration of partners and sub-partners were there. Extending the time in Mozambique required reducing the time allocated for Kenya. All of the partner sub-grant recipients in Kenya have some presence in Nairobi, but their field operations are scattered in outlying counties. Some of these are within driving distance of Nairobi, but others are in the more distant areas of Kisumu and Kakamega. The evaluation team chose to fly to Kisumu, where several sub￾grantees had field operations, and they were able to visit several outlying areas during that trip. It would have been optimal to have had enough time in the field to allow for setting up formal focus groups with farmers (both users and non-users of the technologies supported by P4I. As part of the change in schedule to allow for increased time in Mozambique, the evaluation team and USAID agreed that formal focus groups would not be feasible. P4I Midterm Performance Evaluation Report | November 2015 59 To address this limitation, the evaluation design combined opportunistic sampling and maximum variation sampling. Through this approach, the evaluation team traveled to a variety of sites within a 2.5 hour radius of Kisumu with staff of sub-grantees and/or their second-level sub-partners. Sites visited included agrodealer shops (both those that sold P4I-supported technologies and those that did not, demonstration sites, and farmer fields. The team also created opportunities to speak with purposefully selected farmers and purchasers at agrodealer and other shops. In addition, the evaluation team met with one opportunistically organize group of women, and with three opportunistically assembles small groups (two groups of men and one mixed group). Other limitations Partial coverage of P4I Components: As directed by the request for task order proposals, this performance evaluation addresses Components 1 and 2 of a four-component activity. While it was clear that these two components were priorities for learning for USAID, it must be taken into account that this midterm performance evaluation was an evaluation of two components of P4I and not of P4I in its entirety. Subjective Measurements: In qualitative research, measurement of results consists to a large degree of subjective perceptions by key informants. The evaluation did not have the scope to conduct quantitative measurements of certain activities or preliminary results. The evaluation questions were designed taking this into account, and the evaluation team used in-depth interviews to gather rich qualitative data and apply rigorous analysis to address those questions. Language: Neither of the members of the evaluation team spoke Portuguese. While many of the interviews in Mozambique could have be conducted in English, an interpreter was hired to support the team for opportunistic interviews in the Nampula area. P4I Midterm Performance Evaluation Report | November 2015 60 ANNEX III: P4I SUB-GRANTS P4I Component 1 and 2 Sub-grants: March 2013 – April 2015 Country Name of Implementer/ Sub￾Grantee Sub-grantee is Private Firm or Not-for-Profit Period of Performance Grant Description Component 1 Piloting Rwanda Purdue University (with EcoPlastics) Not-for-Profit March 2013- March 2014 PICS post-harvest grain storage bags Cambodia WorldFish Not-for-Profit August 2013- June 2014 Model fish ponds Zambia Catholic Relief Services (CRS) Not-for-Profit August 2013- February 2014 Groundnut postharvest processing Mexico Buen Manejo del Campo Private Firm August 2013- February 2014 Biodigester demonstration systems Round 1 Cameroon, Côte d’Ivoire, Ghana, Nigeria World Cocoa Foundation Not-for-Profit May 2013- May 2014 Mobile extension program India Driptech, Inc. Private Firm July 2013 – November 2014 Drip irrigation systems Rwanda PortaScience Private Firm August 2013 – November 2014 Mastitis detection and udder hygiene kits for cattle Senegal Compatible Technology International (CTI) Not-for-Profit July 2013 – July 2014 Pear millet thresher Kenya Netafim (with Amarin and Conexus) Private Firm September 2013 – September 2015 Drip irrigation kits Kenya Purdue University (with Bell Industries) Not-for-Profit September 2013 – September 2014 PICS post-harvest grain storage bags Kenya, Tanzania, Uganda African Agricultural Technology Foundation (AATF) (with seed companies) Not-for-Profit January 2014 – March 2017 Imazpyr-resistant seeds Zambia iDE (with Toro) Not-for-Profit September 2013- October 2014 Drip irrigation kits Round 2 Bangladesh Moana Private Firm March 2014 – August 2015 Black tiger shrimp Zambia AGCO Private Firm May 2014 – November 2015 Grain storage and training Guatemala Mercy Corps (with Syngenta Foundation) Not-for-Profit July 2014 – July 2015 Electronic traceability technology Kenya Surehatch Private Firm July 2014 – July 2015 Poultry egg incubators Kenya MEA Limited Private Firm October 2015 – December 2016 Rhizobial inoculant for legumes Round 3 Ethiopia EthioChicken Private Firm March 2015 – June 2016 Improved poultry breeds, inputs and training P4I Midterm Performance Evaluation Report | November 2015 61 Country Name of Implementer/ Sub￾Grantee Sub-grantee is Private Firm or Not-for-Profit Period of Performance Grant Description Honduras Zamorano (Pan American Agricultural University) (with distributors) Not-for-Profit April 2015 – January 2017 Biological pest control Component 2 Mozambique Mozambique International Development Enterprises (iDE) (with Lusosem Mocambique, Lda and HUB Assistancia Technica e Formacao) Not-for-Profit November 2014 – July 2017 Input packages and extension services Mozambique Export Marketing Company Limited (EMCL) Private Firm January 2014- July 2017 Retail hubs, seeds, agrochemicals, equipment, and storage Mozambique National Cooperative Business Association CLUSA International (NCBA/CLUSA) (with Phoenix Seeds and Oruwera) Not-for-Profit November 2014- July 2017 Input packages and extension services Mozambique Opportunity International (OI) (with Banco de Opportunidade de Mozambique) Not-for-Profit November 2014- July 2017 Sesame and soybean services Malawi Malawi Opportunity International (OI) (with Opportunity International Bank Malawi) Not-for-Profit February 2015 – July 2017 Financial services, training and value chain integration Malawi Universal Industries, Ltd. Private Sector February 2015 – July 2017 Orange-fleshed sweet potato production Malawi Rab Processors Ltd. Private Sector March 2015 – July 2017 Marketing and storage facilities P4I Midterm Performance Evaluation Report | November 2015 62 ANNEX IV: SUMMARY OF EVALUATION QUESTION 1 STRENGTHS AND WEAKNESSES: What are the strengths and weaknesses of the ways in which P4I works with the private sector (through Components 1 and 2) as a means to increase USAID’s private sector engagement in Mozambique and Kenya? EVALUATION QUESTION 1: P4I As USAID Interface with Private Sector: What are the strengths and weaknesses of the ways in which p4i works with the private sector (through Components 1 & 2) as a means to increase USAID’s private sector engagement in Mozambique and Kenya? 1a) What worked well and what could be improved to maximize both the engagement of the private sector and the strength of those partners’ relationship with USAID? 1b) How well do the funded sub-grants align with the FTF strategies of USAID and of the Kenya and Mozambique missions? 1. Mission Needs and P4I Component 1 and 2 Sub-grants: Kenya and Mozambique Strengths Component 2 Agility: According to USAID/Mozambique staff, the P4I Component 2 sub-grant development and award process moved more quickly than an award through their Mission could have. USAID/Mozambique staff indicated appreciation for the opportunity for the Mission to take advantage of P4I as a transitional FTF funding mechanism for the Mission that awarded sub-grants as a program bridge to the next generation of FTF programming in Mozambique. USAID/Mozambique staff involved with the P4I work expressed appreciation for the timeliness of the availability of P4I’s Component 2 and of P4I’s ability to quickly put FTF awards and programming in place. Component 2 Low Cost and Low Level of Effort for Mission: USAID/Mozambique staff expressed appreciation that the P4I Component 2 allows USAID/Mozambique to fund FTF programming and report on FTF indicators while also alleviating the Mission of the need to allocate substantive staff time or budget for management of the FTF activities it funds. Fintrac performs most of the administration and management of the sub-grants. The Mission found it adequate for management to depend on Fintrac and on the USAID/BFS-based AOR and to simply assign a staff member to each of the four P4I sub-grants as an Activity Manager. Activity Managers indicated appreciation that little technical or management staff time was needed for sub-grant award and management. Component 1 and Smallholder Farmers: USAID/staff in Kenya noted that there were benefits to expanding smallholder farmer access to promising technologies. Weaknesses Component 1 Missing Opportunities for Coordination to Improve Sub-grant Results: For Component 1 in Kenya, there has been little active coordination or cooperation between P4I-supported sub￾grants and activities and USAID/Kenya staff after the sub-grants were awarded. While, in general, P4I is in line with the USAID/Kenya FTF strategy and supports USAID/Washington engagement with the private sector, there have been some disconnects with the Mission FTF strategy. For instance, in the case of the P4I Surehatch grant, while its interventions and development of small scale chicken and P4I Midterm Performance Evaluation Report | November 2015 63 egg production are moving forward well, poultry is not a priority value chain in the Kenya FTF strategy. As noted above, USAID/Kenya has a large FTF program and an FTF grant-making activity for innovations. Mission staff indicated that the independent operation of another FTF grant-making activity in Kenya that was not associated with the Mission introduced some complications into their relationships with private sector actors who conflated the two FTF grant mechanisms. This was seen as undermining the goal of a single, coherent FTF strategic approach. 2. Private Sector Engagement Strengths Support for private sector engagement: P4I supports private sector engagement under FTF. Acceptance of private sector or non-profit sub-grant leads: While P4I was designed to support commercialization by private sector firms, P4I has created a flexible structure that accepts consortia of private sector firms and non-profits. Substantive leverage: In Mozambique Component 2 sub-grants, private sector partners bring a match of 51-68%. In Kenya Component 1 sub-grants, the match is between 8% and 49%. Weaknesses High Percentage of Non-Profit Sub-grantees: Of the nine sub-grants covered in this evaluation of P4I, an activity which is aimed at partnering with the private sector, five sub-grants were led by non-profit organizations. While all P4I sub-grants must include private sectors firms at some level, overall, half of the 24 sub-grants have been awarded to non-profit organizations and half to private sector firms. Six of the nine sub-grantees were based outside of the country in which the sub-grant was being implemented. Both of these situations limit Missions’ ability to engage effectively with the private sector firms being funded in their country. Lack of Awareness of USAID Role: Some key staff of entities awarded sub-grants through Component 2 that were interviewed were not aware of USAID/Mozambique’s role in funding their work. Some NGO sub-grantee staff confidently stated that they had a standard sub-partner arrangement with Fintrac, as P4I’s prime. While they could name the AOR in Washington, they indicated that they were not aware of any USAID/Mozambique involvement in their work. Low Levels of USAID Engagement: While P4I grants in Kenya are technically USAID/(BFS) private sector engagement, there is almost no USAID/Kenya engagement with the sub-grantees and little involvement with P4I beyond review of the short-listed application and concurrence for the sub￾award. The P4I sub-grants operate independently from the Mission. At the time the interviews were conducted, some people in leadership roles in private firms that were second-level sub-grantees (including some who were subbing to NGOs) had never met anyone from USAID or had only met USAID staff through in an informal opportunity with no substantive discussion about the firm’s FTF work. Some second-level sub-grantees were said to be included in the periodic phone calls with P4I (reportedly largely as listeners, and none appear to be listed as participants in the call notes for three P4I Midterm Performance Evaluation Report | November 2015 64 sub-grantees in Mozambique). Mission staff indicated that they would like more formal opportunities to interact with P4I private sector sub-grantees and second-level sub-grantees. 3. Management Complexity Strengths P4I Strong and Responsive Communications: P4I staff’s on-going communications with the sub￾grantees, USAID Missions, and USAID/BFS were strong and important in keeping all on board in this multi-layered program structure. Weaknesses Complex Lines of Communication: Management and communications links are sometimes cumbersome in Component 2. Two USAID/Mozambique staff pointed out that this creates distance between the Mission and some of the private sector implementers of the Mission-funded work. 4. Roles and Responsibilities Strengths P4I Communication: P4I staff are strong communicators during on-going sub-grant management. AOR Role: Stakeholders spoke highly of the USAID/BFS AOR. Mission staff in Mozambique accorded her particular recognition for the amount of work done and the difficulty of the assignment. Weaknesses Washington, DC-based Management: The Washington, DC locus of management support and decision-making is far away from where the sub-grants operate. There is limited ability for P4I staff to observe sub-grant operations on-the-ground or to provide hands-on, face-to-face support. The USAID AOR for P4I is based at USAID/Washington. This is a difficult assignment, similar to that faced by P4I staff, but all falling under the responsibility of one person. Many, if not most, of the decisions on the cooperative agreement and particularly on the management of the SMOGs come from technical and business challenges in a diverse set of P4I countries and circumstances far removed from USAID/Washington. The lack of hands-on management of awards by people with a rich understanding of the context in which the awards operate was pointed out as a limitation by informants from Missions, sub-grantees, and second-level sub-grantees. Very Wide Array of Expertise Needs: The Fintrac team does not have knowledge and expertise in all local development contexts, political and economic structures, country business operating environments, country regulations and protocols, local markets, and product applications in all the FTF countries. The range of technical topics and types of business programs that P4I staff must manage is similarly broad. Staff does not have technical capacity in all of the sub-grantee’s technologies and innovations. In terms of both country and operating context, the range of proposals that P4I is required to review, assess, and prioritize is extremely wide. Component 2 Mission Role: Mozambique: While indicating appreciation that Mission staff did not need to spend substantive time managing the FTF awards, most Mission staff interviewed said that they would prefer to have a greater role in oversight and support for the awards. P4I Midterm Performance Evaluation Report | November 2015 65 Other Strengths Macro-level Action: P4I provides USAID/BFS with a vehicle for action at a macro scale. This gives BFS the ability to match opportunities to needs and to support innovations that have the potential for scaling up in ways that support FTF’s global objectives and that have potential for replication in a variety of contexts. Sub-grant Negotiations: Some (though not all) sub-grantee firms expressed satisfaction with the pace of the process. All private sector sub-grantees interviewed indicated that the negotiation process was not as fast and straight forward as contract negotiation with a private firm. (See Evaluation Question 2.) The USG does have many grant requirements beyond what is found in commercial contracts: there are approximately 40 pages of standard and other provisions in the SMOGs that cover up to 20 different requirements for certification, agreement, or operational procedures. Many private sector sub-grantees noted that they were not clear on the need for and relevance of these requirements. P4I Midterm Performance Evaluation Report | November 2015 66 ANNEX V: DATA COLLECTION INSTRUMENTS Key Informant Semi-Structured Interview Guide: USAID/Partners (Sub-Grantees) Interviewers provide introduction that covers the following points:  Brief introduction of the evaluation team members  Purpose of the evaluation and of the interview  Main topics of the of the interview  Confidentiality of responses  Request for and receipt of permission to interview Interview Code #: Sub-grant Name/Topic: Firm/Organization: Position: Male or Female: Country: Date & time of interview: Name of primary interviewer: Name of primary note taker: Background 1. Can you please give us an overview of your role with (sub-grant name/topic)? a. How long have you been working with P4I b. What is your role? Evaluation Question 2: What are the effects of the structure of the SMOG funding model on private sector firms’ ability to scale up smallholder access to their technologies? 2. How has this grant affected your business? (How do you value the grant? What changes have you made because of the grant? What has it enabled you to do that you could not do before? Any negative effects?) 3. How would you assess the P4I selection process for sub-grantee awards? (LOE to complete meet SMOG, Length of time to decision and award? Steps? Clarity? Fairness? Adequacy of grant amount? Other?) a. How does this compare to other donor grant awards // loans from banks? b. What changes would you suggest in the process? P4I Midterm Performance Evaluation Report | November 2015 67 4. If P4I did not exist, would you have undertaken the outreach to smallholders? a. Would you have borrowed money to undertake the P4I activity as planned? Or undertake it in some modified way with a loan? 5. A key first step the design of the SMOGs is a business plan for the company or technology. a. How closely does your implementation match your business plan in the proposal and the SMOG? b. Did you find the SMOG to a good guide for reaching the business outcome sought? 6. What has been your experience with the milestones on this grant? a. What challenges are you facing meeting milestones? b. Do the requirements/content, timing and funds associated with each milestone work well to incentivize performance and align your internal processes and funding needs? 7. Kenya: a. Have the required milestones helped the firm grow its business with smallholder farmers? b. How have the grants affected the way you and your sub-partners market and sell the innovation /technology to smallholder farmers? c. Does (your technology) appear to be selling to smallholders in line with the business plan and SMOG milestones? How do you know? (To what extent have you been able to commercialize the technologies that are the subject of the grant?) 8. What are the challenges you facing in successfully and sustainability commercializing (your technology)? Evaluation Question 4: How can P4I improve its ability to assess whether P4I Component 1 and 2 grants and partner investments are increasing smallholder men and women farmers’ to access and use innovative technologies? 9. What kind of information do you currently collect on the farmers you that your technology is sold to? (Probe for the types of data in the verification milestones for the grant.) 10. How do you collect and manage this information? (Ask to see examples) 11. How easy or how challenging is it to collect and manage the information that is required? If challenging, probe for why and for how to make it less challenging.) 12. What are the challenges of gender data for the technology that is being promoted? (How do you define how to record whether your technology is being used by men or by women?) 13. Is there any data or planned collection of information that would show if the farmers are benefitting from using the new technology? How would you know? (Explain outcome measures and ask if there are ways that they could help to collect output or outcome data.) 14. Thinking about the information that you collect for milestone verification - how are you suing that information in your business? P4I Midterm Performance Evaluation Report | November 2015 68 Evaluation Question 1: What are the strengths and weaknesses of the ways in which P4I works with the private sector (through Component 1 or 2) as a means to increase USAID’s private sector engagement in Mozambique and Kenya? 15. Where is the money for this grant coming from? 16. Please describe your relationship with Fintrac. (Probe for supervisory, support role, etc.) How often are you in contact with Fintrac? 17. Please describe your relationship with USAID. (What kinds of interactions do you have with USAID staff? How often or you in contact with USAID?) Evaluation Question 3: How might management of P4I (by Fintrac, BFS, and/or Missions) be improved to maximize P4I’s outputs and outcomes? What changes in P4I management, administration and reporting could be carried out to maximize P4I’s results? 18. What kinds of support has Fintrac provided to you? // to your sub-grantee(s)? a. Have you received technical support from Fintrac on the technical implementation of the work under your grant? b. Have you received administrative support from Fintrac on the technical implementation of the work under your grant? c. Have you received other support from Fintrac to reach the objectives set out in your sub award to help sell the technology/service to the small holder? 19. How would you describe the support? (Poor? Adequate? Strong? Intrusive? Not understanding our business? Other?) 20. What happens if you are not able to meet a milestone? (What do you do? Who makes the decision? What are the ramifications?) 21. What are the main strengths and weaknesses of Fintrac’s interactions with you? 22. How could the grants and Fintrac support be improved? Are there any changes that you would recommend in the technical and business support that you did receive? 23. If you were to design the solicitation and SMOG process from scratch, is there anything you might do differently? (If so, what?) Wrap Up 24. Would you write a proposal for something like P4I again in the future if it were available? 25. Any other recommendations or advice for improving current or future programming? 26. Is there anything else that would be useful to the evaluation and that you think we should know? Thank you very much for your time. P4I Midterm Performance Evaluation Report | November 2015 69 Key Informant Semi-Structured Interview Guide: USAID/Partners (Sub-Grantees) Interviewers provide introduction that covers the following points:  Brief introduction of the evaluation team members  Purpose of the evaluation and of the interview  Main topics of the of the interview  Confidentiality of responses  Request for and receipt of permission to interview Interview Code #: Sub-grant Name/Topic: Firm/Organization: Sub –Partner to: Position: Male or Female: Country: Date of interview: Time of interview: Name of primary interviewer: Name of primary note taker: Note: Some of the sub-partners are not very involved in the management issues for the project, communication with Fintrac, collection of verification information etc., so many of the questions below will not be appropriate or needed for some sub-partners. Background 1. Can you please give us an overview of your role with (sub-grant name/topic)? a. How long have you been working with this project? b. What is your role? Evaluation Question 2: What are the effects of the structure of the SMOG funding model on private sector firms’ ability to scale up smallholder access to their technologies? 2. How has this grant affected your business? (How do you value the grant? What changes have you made because of the grant? What has it enabled you to do that you could not do before? Any negative effects?) 3. How would you assess the P4I selection process for sub-grantee awards? (LOE to complete meet SMOG, Length of time to decision and award? Steps? Clarity? Fairness? Adequacy of grant amount? Other?) a. How does this compare to other donor grant awards // loans from banks? b. What changes would you suggest in the process? P4I Midterm Performance Evaluation Report | November 2015 70 4. If P4I did not exist, would you have undertaken the outreach to smallholders? a. Would you have borrowed money to undertake the P4I activity as planned? Or undertake it in some modified way with a loan? 5. A key first step the design of the SMOGs is a business plan for the company or technology. a. How closely does your implementation match your business plan in the proposal and the SMOG? b. Did you find the SMOG to a good guide for reaching the business outcome sought? 6. What has been your experience with the milestones on this grant? a. What challenges are you facing meeting milestones? b. Do the requirements/content, timing and funds associated with each milestone work well to both incentivize performance and align your internal processes and funding needs? 7. Kenya: a. Have the required milestones helped the firm grow its business with small holder farmers? b. How have the grants affected the way you and your sub-partners market and sell the innovation /technology to smallholder farmers? c. Does (your technology) appear to be selling to smallholders in line with the business plan and SMOG milestones? How do you know? (To what extent have you been able to commercialize the technologies that are the subject of the grant?) 8. What are the challenges you facing in successfully and sustainability commercializing (your technology)? Evaluation Question 4: How can P4I improve its ability to assess whether P4I Component 1 and 2 grants and partner investments are increasing smallholder men and women farmers’ to access and use innovative technologies? 9. What kind of information do you currently collect on the farmers you that your technology is sold to? (Probe for the types of data in the verification milestones for the grant.) 10. How do you collect and manage this information? (Ask to see examples) 11. How easy or how challenging is it to collect and manage the information that is required? a. If challenging, probe for why and for how to make it less challenging 12. What are the challenges of gender data related to the technology that is being promoted? (How do you define how to record whether your technology is being used by men or by women?) 13. Is there any data or planned collection of information that would show if the farmers are benefitting from using the new technology? How would you know? (Explain outcome measures to partners and ask if there are ways that they could help to collect output and outcome data.) 14. Thinking about the information that you collect for milestone verification - how are you suing that information in your business? P4I Midterm Performance Evaluation Report | November 2015 71 Evaluation Question 1: What are the strengths and weaknesses of the ways in which P4I works with the private sector (through Component 1 or 2) as a means to increase USAID’s private sector engagement in Mozambique and Kenya? 15. Where is the money for this grant coming from? 16. Please describe your relationship with the grant recipient. 17. Please describe your relationship with Fintrac. (Probe for supervisory, support role, etc.) How often are you in contact with Fintrac? 18. Please describe your relationship with USAID. (What kinds of interactions do you have with USAID staff? How often or you in contact with USAID?) Evaluation Question 3: How might management of P4I (by Fintrac, BFS, and/or Missions) be improved to maximize P4I’s outputs and outcomes? What changes in P4I management, administration and reporting could be carried out to maximize P4I’s results? 19. Could you have written a winning proposal or managed this grant without your partner (iDE/NCBA/CLUSA/Opportunity International/ Purdue?) 20. What kinds of support has Fintrac provided to you? a. Have you received technical support from Fintrac on the technical implementation of the work under your grant? b. Have you received administrative support from Fintrac on the technical implementation of the work under your grant? c. Have you received other support from Fintrac to reach the objectives set out in your sub award to help sell the technology/service to the small holder? 21. How would you describe the support? (Poor? Adequate? Strong? Intrusive? Not understanding our business? Other?) 22. What happens if you are not able to meet a milestone? (What do you do? Who makes the decision? What are the ramifications?) 23. What are the main strengths and weaknesses of Fintrac’s interactions with you? 24. How could the grants and Fintrac support be improved? Are there any changes that you would recommend in the technical and business support that you did receive? 25. If you were to design the solicitation and SMOG process from scratch, is there anything you might do differently? (If so, what?) Wrap Up 26. Would you write a proposal for something like P4I again in the future if it were available? 27. Any other recommendations or advice for improving current or future programming? 28. Is there anything else that would be useful to the evaluation and that you think we should know? Thank you very much for your time. P4I Midterm Performance Evaluation Report | November 2015 72 Key Informant Semi-Structured Interview Guide: USAID/Kenya Staff Interviewers provide introduction that covers the following points:  Brief introduction of the evaluation team members  Brief background on Partnering to Innovation (P41) and its different components  Purpose of the evaluation and of the interview  Main topics of the of the interview  Confidentiality of responses  Request for and receipt of permission to interview Interview Code #: Position: Bureau/Office Male or Female: Country: Date of interview: Time of interview: Name of primary interviewer: Name of primary note taker: Background 1. Can you please give us an overview of your role with P4I? a. How long have you been working with P4I b. What other roles do you have in your position at USAID? Evaluation Question 1: What are the strengths and weaknesses of the ways in which P4I works with the private sector (through Components 1 and 2) as a means to increase USAID’s private sector engagement in Mozambique and Kenya? 2. What are the primary benefits of P4I sub-grants so far? 3. What are the primary challenges? 4. How have the P4I awards been important to the Mission in engaging the local private sector? Explain the experience. 5. What worked well to maximize both the engagement of the private sector and the strength of those partners’ relationship with FTF and USAID? //What has not worked well? //Why? 6. How well do the Component 1 funded sub-grants align with the Mission’s FTF strategies of the Mission? P4I Midterm Performance Evaluation Report | November 2015 73 7. How involved has the Mission been in the P4I selection process for potential sub-grants for innovative technologies in Kenya? a. Does the process “work” in terms of Mission involvement, sub-grantee/technology selection and outcomes? 8. Are there synergies and complementarities with USAID/Kenya programs? a. Are there competitions and duplications? b. Please tell us about the programmatic and operational relationships between P4I awards and the Kenya FTF Innovation Engine Project (KIFIE). 9. What Mission programs other than KFIE support the FTF New Alliance Public/Private Partnerships? a. Has the Kenya Mission considered a P4I Component 2 private partnership award program? 10. Are there other USAID Kenya programs besides KIFIE that engage the private sector in innovation for small holder farmers? 11. How would you compare P4I’s Component 1 work in Kenya with other similar initiatives in Kenya? 12. At the time that the New Alliance for Food Security and Nutrition began (May 2012) what mechanisms did the Kenya Mission have and use to support this initiative. (Note: BFS statement that after the New Alliance was announced, P4I was one of the first mechanisms to promote PPP for FTF.) 13. Thinking about your level of involvement in P4I work in Kenya, do you think it is too much, about right, or too little? (Please explain.) 14. Thinking about other mission staff’s level of involvement in P4I work in Kenya, do you think it is too much, about right, or too little? (Please explain) 15. Does the mission report on the P4I outputs and outcomes as part of its FTF reporting on standard indicators? Evaluation Question 3: How might management of P4I (by Fintrac, BFS, and/or Missions) be improved to maximize P4I’s outputs and outcomes? What changes in P4I management, administration and reporting could be carried out to maximize P4I’s results 16. How crucial is faster than normal turn-around for soliciting and awarding to the Missions FTF objectives? a. How long did P4I take from solicitation to contract signing? b. How long would it have taken had the Mission done the procurement? c. Do you have any current or expected FTF programming gaps that might make a fast turn￾around attractive? 17. Can you please tell us about your impressions of/experience with the P4I SMOGs? a. Do the requirements/content, timing and funds associated with each milestone appear to be well-matched to both incentivize performance and align with sub-grant internal processes and funding needs? b. Have you worked with performance-based contracts before? i. If so, please tell us your thoughts in the strengths and weaknesses of the P4I SMOGs and other performance-based contracts you are familiar with. P4I Midterm Performance Evaluation Report | November 2015 74 18. How well does/do the sub-grantee(s) appear to be doing in meeting milestones? a. What kinds of challenges do you foresee them facing? b. Do any of those pose critical risks for completing the sub-grant requirements on by the end of the grant period? c. What will happen if they don’t meet a milestone? 19. What % of their time did key staff people spend on P4I process from solicitation through the present? // How much time do you anticipate P4I taking in the future? (more? - or less?) 20. If you were to design the solicitation and SMOG process from scratch, is there anything you might do differently? 21. What kinds of support has Fintrac provided to you? // to the subgrantee(s)? // On administrative issues? // On technical issues? 22. What are the main strengths and weaknesses of Fintrac as the implementer of this work? Evaluation Question 4: How can P4I improve its ability to assess whether P4I Component 1 and 2 grants and partner investments are increasing smallholder men and women farmers’ to access and use innovative technologies? 23. Do the means of verification in the sub-grant milestones provide the information you need for your indicator reporting/PPRs? 24. Do you think that sub-grantees will find it easy or challenging to meet the requirements in the means of verification for the milestones? a. If challenging, probe for why and for how to make it less challenging Wrap Up 25. Would you sign up for something like P4I again in the future if it were available? 26. Any other recommendations or advice for improving current or future programming? 27. Is there anything else that would be useful to the evaluation and that you think we should know? Thank you very much for your time. P4I Midterm Performance Evaluation Report | November 2015 75 Key Informant Semi-Structured Interview Guide: USAID/Mozambique Staff Interviewers provide introduction that covers the following points:  Brief introduction of the evaluation team members  Brief background on Partnering to Innovation (P41) and its different components  Purpose of the evaluation and of the interview  Main topics of the of the interview  Confidentiality of responses  Request for and receipt of permission to interview Interview Code #: Position: Bureau/Office Male or Female: Country: Date of interview: Time of interview: Name of primary interviewer: Name of primary note taker: Background 1. Can you please give us an overview of your role with P4I? a. How long have you been working with P4I b. What other roles do you have in your position at USAID? Evaluation Question 1: What are the strengths and weaknesses of the ways in which P4I works with the private sector (through Components 1 and 2) as a means to increase USAID’s private sector engagement in Mozambique and Kenya? 2. What are the primary benefits of P4I so far? 3. What are the primary challenges? 4. How have the P4I awards been important to the Mission in engaging the local private sector? Explain the experience. 5. What worked well to maximize both the engagement of the private sector and the strength of those partners’ relationship with FTF and USAID? //What has not worked well? //Why? 6. Has the P4I Solicitation for New Agricultural Partnerships (SNAP) positively engaged the private sector? P4I Midterm Performance Evaluation Report | November 2015 76 7. What mechanisms were used prior to the P4I SNAP? GDAs? Grant awards to private sector under existing FTF programs in the country? Other? 8. Has the P4I public private partnerships (PPPs) model influenced/affected a. USAID/Mozambique’s other private public partnerships? b. FTF approaches to the private sector? 9. What are the workload implications of the P4I Component 2 on Mission staff/offices? 10. How would you assess the P4I negotiation and agreement to establish a partnership program with the USAID/Mozambique? 11. What are the pros and cons of going through a mechanism like P4I to manage Mission-funded activities? 12. Any recommendations for improvement? 13. Are there plans to add funding to P4I Component 2? Evaluation Question 3: How might management of P4I (by Fintrac, BFS, and/or Missions) be improved to maximize P4I’s outputs and outcomes? What changes in P4I management, administration and reporting could be carried out to maximize P4I’s results 14. How crucial is faster than normal turn-around for soliciting and awarding to the Missions FTF objectives? a. How long did P4I take from solicitation to contract signing? b. How long would it have taken had the Mission done the procurement? c. Do you have any current or expected FTF programming gaps that might make a fast turn￾around attractive? 15. Can you please tell us about your impressions of/experience with the P4I SMOGs? a. Do the requirements/content, timing and funds associated with each milestone appear to be well-matched to both incentivize performance and align with sub-grant internal processes and funding needs? b. Have you worked with performance-based contracts before? i. If so, please tell us your thoughts in the strengths and weaknesses of the P4I SMOGs and other performance-based contracts you are familiar with. 16. How well does/do the sub-grantee(s) appear to be doing in meeting milestones? a. What kinds of challenges do you foresee them facing? b. Do any of those pose critical risks for completing the sub-grant requirements on by the end of the grant period? c. What will happen if they don’t meet a milestone? 17. What % of their time did key staff people spend on P4I process from solicitation through the present? // How much time do you anticipate P4I taking in the future? (more? - or less?) 18. If you were to design the solicitation and SMOG process from scratch, is there anything you might do differently? P4I Midterm Performance Evaluation Report | November 2015 77 19. What kinds of support has Fintrac provided to you? // to the subgrantee(s)? // On administrative issues? // On technical issues? 20. What are the main strengths and weaknesses of Fintrac as the implementer of this work? Evaluation Question 4: How can P4I improve its ability to assess whether P4I Component 1 and 2 grants and partner investments are increasing smallholder men and women farmers’ to access and use innovative technologies? 21. Do the means of verification in the sub-grant milestones provide the information you need for your indicator reporting/PPRs? 22. Do you think that sub-grantees will find it easy or challenging to meet the requirements in the means of verification for the milestones? a. If challenging, probe for why and for how to make it less challenging Wrap Up 23. Would you sign up for something like P4I again in the future if it were available? 24. Any other recommendations or advice for improving current or future programming? 25. Is there anything else that would be useful to the evaluation and that you think we should know? Thank you very much for your time. P4I Midterm Performance Evaluation Report | November 2015 78 ANNEX VI: SOURCES OF INFORMATION List of persons interviewed No. Organization/Entity Location # of Key Informant Interviews (KIIs) Fieldwork-Based Key Informant Interviews (KIIs) 1 iDE Mozambique Maputo, Mozambique 2 2 Lusosem Maputo/Nampula, Mozambique 2 3 HUB Nampula, Mozambique 2 4 USAID/Mozambique Maputo, Mozambique 5 5 ETG Mozambique Maputo, Mozambique 3 6 Techno Brain Maputo, Mozambique 1 7 Connexus Maputo, Mozambique 1 8 NCBA/CLUSA Nampula, Mozambique 2 9 Oruwera Nampula, Mozambique 3 10 Agrodealers Nampula, Mozambique 2 11 Banco Oportunidade de Mozambique Maputo, Mozambique 2 12 Purdue University Nairobi, Kenya 1 13 Surehatch RSA South Africa 1 14 Surehatch Kenya Nairobi, Kenya 3 15 Netafim Israel Israel 1 16 Amiran/Connexus Nairobi, Kenya 1 17 African Agricultural Technology Foundation (AATF) Nairobi, Kenya 5 18 Bell Industries Nairobi, Kenya 2 19 MEA Nairobi/Bugoma, Kenya 4 20 WeRate Bugoma, Nairobi 1 21 Agricultural Input Dealers Kakmega/Kisumu/Bugoma, Kenya 10 22 FreshCo Seeds Nairobi, Kenya 1 23 USAID/Kenya Nairobi, Kenya 2 24 IFDC Nairobi, Kenya 1 25 Farmers Bugoma/Kisumu, Kenya 7 US-Based Key Informant Interviews (KIIs) 26 NCBA/CLUSA Washington, DC 3 27 iDE Washington, DC 2 28 Opportunity International Washington, DC 1 29 Connexus Washington, DC 1 Meetings 30 USAID BFS Washington, DC 6 31 Fintrac Washington, DC 6 Opportunistic Interviews 32 Farmers Mozambique/Kenya 23 P4I Midterm Performance Evaluation Report | November 2015 79 Documents reviewed For this evaluation, the evaluation team will review a wide variety of documents. Many have already been provided by BFS and by P4I. Other documents will be requested from P4I and from sub-grant partners during the fieldwork preparation phase, and others will be requested of partners and sub￾partners while the evaluation team is in the field. This desk review contributed to understanding P4I activities, and documents on the grant application and award processes will be particularly important for understanding expected resource mobilization, descriptions of the proposed technologies and their relevance to smallholder farmers, explanations of proposed commercialization strategies, descriptions of target markets, applicant’s experience with organizational capacity and partnerships, and overall details of market sustainability and comparing these to results to date. The primary documents and types of documents are listed below.  P4I award documents, including BFS Request for Applications, contract with Fintrac, and performance monitoring plans and P4I innovation indicators  Documents from USAID/Kenya and USAID/Mozambique on public private partnerships and other innovation grants programming  Documents on FTF objectives and implementation in Kenya and Mozambique, such as FTF factsheets, etc.  P4I progress reports, including annual reports for 2013 and 2014 and semi-annual reports  P4I implementation documents, including work plans (FY13 and FY14), P4I Partnership Portfolio (current partners, completed partnerships, current partnerships in Mozambique), FTF P4I Factsheet, P4I Organizational Chart  Documents related to sub-grant solicitations and awards for Component 1 Rounds 1, 2, 3 and 4, including calls for proposals, proposals/application submission, selection criteria and TEC Review, and grant award documents (including modifications), SMOGs, information from assessment of Purdue/Bell Industries award, etc.  Documents related to investment sub-grant solicitations and awards for Component 2, including the Mozambique SOW, the Solicitation for New Agricultural Partnerships (SNAP) in Mozambique, selection criteria, successful and unsuccessful investment applications, selection memos, award documents (including modifications), etc.  Documents on and from and on sub-grant recipients, including award contracts, periodic progress reports, data for reporting on indicators, meeting notes, updates, memos, notifications, etc. P4I Midterm Performance Evaluation Report | September 2015 80 ANNEX VII: P4I LOGICAL FRAMEWORK (WITH ADDITIONAL INDICATORS FOR COMPONENT 2 WORK WITH USAID/MOZAMBIQUE)18 The expected results of P4I’s interventions are presented below in Table 1: P4I’s logical framework with Mozambique-specific Component 2 indicators. This logical framework contains all of the information from P4I’s overall logical framework. It also includes (in blue font) additional standard F and FTF indicators that P4I added to incorporate P4I’s Component 2 work with USAID/Mozambique. Objectives Indicators for PFI* Indicators specific to P4I Component 2 work with USAID/Mozambique Means of Verification Assumptions Goal: Find new approaches to food security through new and innovated partnerships, tools, and methodologies that improve market access for food insecure households in Feed the Future countries. Sub-Goal (specific to Mozambique Partnership): Increased Agricultural Sector Growth & Food Security in Focus Provinces with Emphasis on Women19 Outcome 1: Commercial availability of improved agriculture technology to small farmers increased Output 1.1: Companies introduce appropriate technologies to smallholder markets with reduced risk. 4.5.2-12 – number of PPPs formed (tech commercialized); 4.5.2-37 – number of MSMEs, including farmers, receiving business development services from USG assisted sources; 4.5.2-43 – number of firms (excluding farms) or Civil Society Organizations (CSOs) engaged in agricultural and manufacturing and food security related services now operating more profitably (at or above cost) because of USG assistance Number of technology commercialization subgrants awarded; Sales/loan/training data from subrecipients; po\st-award surveys when able. Effective local partnerships to introduce and promote technologies and activities; FTF P4I funding continues; Private sector is able to scale technologies to be appropriate to smallholder farmers. Output 1.2: Smallholder farmers use new technologies to improve yields, increase incomes, reduce disease/pests, or make other improvements to their farming practices. 4.5.2-2 – number of hectares of land under improved technologies or management practices as a result of USG assistance; 4.5.2-5 – number of farmers and others 4.5.2-7 – number of individuals who have received USG supported short-term agricultural sector productivity or food security training; 4.5.2-13 – number of rural households benefitting directly from USG interventions; 4.5.2-29 – value of agricultural and rural loans; CUSTOM – number of people with a savings account or mobile money account as a result of USG assistance; Sales/loan/training data from subrecipients; post-award surveys when able. Market prices are sufficient to support technology investments; adequate sales and support services. 18 Adapted from Fintrac, “Monitoring and Evaluation Plan: Feed the Future Partnering for Innovation Mozambique Mission Partnership,” February 26, 2015. 19 This goal and its associated indicators are specific to P4I’s work with the Mission in Mozambique. P4I Midterm Performance Evaluation Report | September 2015 81 who have applied improved technologies or management practices as a result of USG assistance; 4.5.2-30 – Number of MSMEs, including farmers, receiving USG assistance to access loans; GNDR-2 - Percentage of female participants in USG-assisted programs designed to increase access to productive economic resources; 4.5.2- 39 – number of technologies/mgmt. practices; TBD – number of technologies commercialized. Outcome 2: USAID leverages support from private companies to meet individual program goals Output 2.1: Agreements with private sector companies show private sector investment essential to the success of the activity. 4.5.2-38 – private sector investment (tech commercialized & partnerships developed) Leveraging reports from subrecipients Subrecipient has resources to leverage; accounting and other data maintained to verify leverage. Output 2.2: Missions and private sector partner for activities aligned with Mission/FTF priorities. 4.5.2-12 – number of PPPs formed (partnerships developed) Number of mission partnership subawards developed Technologies/activities vetted during pre-award phase to ensure alignment; mission and partners’ programs supportive Outcome 3: USAID is able to partner with the private sector to reach smallholder farmers effectively and within the parameters of private sector needs. Output 3.1: Partnership models and toolkit developed and shared with USAID missions. TBD – number of models developed. Number of models developed Data made available (by private and/or public sector). Output 3.2: Investment models developed and shared with USAID missions and private sector. Same as above Same as above Data made available (by private and/or public sector). Outcome 4: Private sector, Missions, stakeholders, and members of the development community have a central source of information and analysis on innovations reaching smallholder farmers. Output 4.1: lessons learned and best practices from literature review of agricultural commercialization, partnership seeding, and modeling of PS engagement continuously updated TBD – number of models developed and added to the total list. Number of models developed Agreement on best practices examples P4I Midterm Performance Evaluation Report | September 2015 82 ANNEX VIII: P4I INDICATORS AND PERFORMANCE Numbers are NOT Cumulative Indicator 4.5.2-12 Number of public-private partnerships formed as a result of FTF assistance Component # Year 1 Target Year 1 Achieved Year 2 Target Year 2 Achieved Year 3 Target Year 3 Achieved LOP Target Achieved to Date LOP % of Target 1. Agriculture Technologies Commercialized* 3 8 6 5 8 3 22 16 73% 2. Partnership Development 0 0 4 0 6 0 20 0 0% Indicator 4.5.2-38 Value of new private sector investment in the agriculture sector or food chain leveraged by FTF implementation (US$,000) Component # Year 1 Target Year 1 Achieved Year 2 Target Year 2 Achieved Year 3 Target Year 3 Achieved LOP Target Achieved to Date LOP % of Target 1. Agriculture Technologies Commercialized 533 - 1,043 772 1,539 437 3,920 1,208 31% 2. Partnership Development 562 - - - - - 562 0 0% Additional FTF Indicators for Subgrants Indicat or # Indicator Description Year 1 Target Year 1 Achieved Year 2 Target Year 2 Achieved Year 3 Target Year 3 Achieved LOP Target Achieved to Date LOP % of Target 4.5.2-2 Number of hectares under improved technologies or management practices as a result of USG assistance 30 30 1,962 605 17,983 487 45,749 1,121 2% 4.5.2-5 Number of Farmers and others who have applied new technologies or management practices as a result of USG assistance. 7,483 7,485 22,201 27,589 60,643 8,399 135,136 43,472 32% 4.5.2-39 Number of technologies or management practices made available for transfer as a result of USG assistance. 15 15 10 9 10 2 35 26 74% TBD Number of technologies commercialized in FTF countries 4 6 8 7 8 3 22 16 73% Number of program-assisted technologies with initial market sales 2 2 14 6 1 - 17 8 47% 3. Investment Design/Models 2 0 3 11 5 18 11 61% Note: This table is adapted from the “Summary” tab of the “P4I 06.23.15 PMP Target Tracker Comp 1 Only” Microsoft Excel file. It does not contain data from Component 2 (under which activities were still getting started). P4I Midterm Performance Evaluation Report | September 2015 83 ANNEX IX: P4I THEORY OF CHANGE P4I Midterm Performance Evaluation Report | September 2015 84 ANNEX X: DISCLOSURE OF ANY CONFLICTS OF INTEREST P4I Midterm Performance Evaluation Report | September 2015 85 P4I Midterm Performance Evaluation Report | September 2015 86 P4I Midterm Performance Evaluation Report | September 2015 87 88 ANNEX XI: FINTRAC STATEMENT OF DIFFERENCES 89 90 ANNEX XII: SOCIAL IMPACT RESPONSE TO FINTRAC STATEMENT OF DIFFERENCES Memorandum To: Salik Farooqi, Partnering for Innovation Performance Evaluation COR, USAID/BFS From: Rees Warne, Partnering for Innovation Performance Evaluation Team Leader, Social Impact Date: December 18, 2015 RE: Response to Fintrac’s Statement of Differences Social Impact is very glad to hear that Fintrac has found some of the recommendations in the Feed the Future Partnering for Innovation Midterm Evaluation Report to be useful. USAID staff have also informed us that recommendations from the evaluation report are being applied. Social Impact is focused on evaluation utilization, and we are delighted that items from the report have already been put to use. We appreciate Fintrac’s response to the evaluation and are happy to respond to the Statement of Differences provided by Fintrac as some of the content appears to not fully represent scope of the evaluation or the content of the evaluation report. Scope of the Evaluation: We do want to respond to Fintrac’s contention about the representativeness of the sites and grants chosen for the evaluation field work by USAID/BFS. In the P4I Midterm Program Evaluation Statement of Differences (also attached here), Fintrac points out that the evaluation covered nine grants in two countries while “P4I currently has 30 completed or ongoing grants in 16 countries.” This implies that USAID/BFS could have chosen among 30 grants in 16 countries as potential subjects for the evaluation. However, as Fintrac is aware, at the time the fieldwork was conducted (May-June, 2015), Fintrac had made 22 sub-grants in 13 countries.2021 (See Table 1.) The evaluation covered 41% of the sub-grants that had been made (33% of the Component 1 sub-grants and 57% of the Component 2 sub- 20 As agreed with Fintrac and USAID, the evaluation did not cover the four small pilot sub-grants that Fintrac made early in the activity cycle. 21 Note that 2 sub-grants covered multiple countries; if those outreach countries are included, the total was 18 countries. See Table 1. 91 grants). As for country representation, USAID BFS assigned the evaluation team to conduct the evaluation in the countries that were the sites of the largest number of sub-grants:  Of the 15 Component 1 sub-grants, five (one-third) were in Kenya. Only one other country had more than one sub-grant – Zambia had two sub-grants.  Of the seven Component 2 sub-grants, four were in Mozambique, and three were in Malawi. Because the sub-grants in Malawi were awarded in February and March of 2015 (two to three months prior to the time of the field work), it was judged that they did not have enough implementation time to be better sources of information for the evaluation than the sub-grants in Mozambique. Table 1: P4I Component 1 and 2 Sub-grants: March 2013 – April 2015 Country Name of Implementer/ Sub￾Grantee Sub-grantee is Private Firm or Not-for-Profit Period of Performance Grant Description Component 1 Round 1 Kenya Netafim (with Amarin and Conexus) Private Firm September 2013 – September 2015 Drip irrigation kits Kenya Purdue University (with Bell Industries) Not-for-Profit September 2013 – September 2014 PICS post-harvest grain storage bags Kenya, Tanzania, Uganda African Agricultural Technology Foundation (AATF) (with seed companies) Not-for-Profit January 2014 – March 2017 Imazpyr-resistant seeds Cameroon, Côte d’Ivoire, Ghana, Nigeria World Cocoa Foundation Not-for-Profit May 2013- May 2014 Mobile extension program India Driptech, Inc. Private Firm July 2013 – November 2014 Drip irrigation systems Rwanda PortaScience Private Firm August 2013 – November 2014 Mastitis detection and udder hygiene kits for cattle Senegal Compatible Technology International (CTI) Not-for-Profit July 2013 – July 2014 Pear millet thresher Zambia iDE (with Toro) Not-for-Profit September 2013- October 2014 Drip irrigation kits Round 2 Kenya Surehatch Private Firm July 2014 – July 2015 Poultry egg incubators Kenya MEA Limited Private Firm October 2015 – December 2016 Rhizobial inoculant for legumes Bangladesh Moana Private Firm March 2014 – August 2015 Black tiger shrimp Zambia AGCO Private Firm May 2014 – November 2015 Grain storage and training Guatemala Mercy Corps (with Syngenta Foundation) Not-for-Profit July 2014 – July 2015 Electronic traceability technology Round 3 92 Country Name of Implementer/ Sub￾Grantee Sub-grantee is Private Firm or Not-for-Profit Period of Performance Grant Description Ethiopia EthioChicken Private Firm March 2015 – June 2016 Improved poultry breeds, inputs and training Honduras Zamorano (Pan American Agricultural University) Not-for-Profit April 2015 – January 2017 Biological pest control Component 2 Mozambique Mozambique International Development Enterprises (iDE) (with Lusosem Mocambique, Lda and HUB Assistancia Technica e Formacao) Not-for-Profit November 2014 – July 2017 Input packages and extension services Mozambique Export Marketing Company Limited (EMCL) Private Firm January 2015- July 2017 Retail hubs, seeds, agrochemicals, equipment, and storage Mozambique National Cooperative Business Association CLUSA International (NCBA/CLUSA) (with Phoenix Seeds and Oruwera) Not-for-Profit November 2014- July 2017 Input packages and extension services Mozambique Opportunity International (OI) (with Banco de Opportunidade de Mozambique) Not-for-Profit November 2014- July 2017 Sesame and soybean services Malawi Malawi Opportunity International (OI) (with Opportunity International Bank Malawi) Not-for-Profit February 2015 – July 2017 Financial services, training and value chain integration Malawi Universal Industries, Ltd. Private Sector February 2015 – July 2017 Orange-fleshed sweet potato production Malawi Rab Processors Ltd. Private Sector March 2015 – July 2017 Marketing and storage facilities Social Impact cannot speak to the reasons that USAID/BFS may have chosen to scope and budget the evaluation to cover two countries. However, we can opine that, if the evaluation were to be limited to two countries, then Kenya and Mozambique were the obvious, reasonable and logical countries to choose. 93 Further, Fintrac states that the evaluation “largely represented the first rounds of both centrally funded and mission buy-in awards, and it therefore did not sufficiently reflect the program’s learning in subsequent rounds of grant funding.” At the time the data collection was conducted, P4I had made three rounds of sub-grants under Component 1. There were eight sub-grants in Round 1 (three of which were in Kenya), five sub-grants in Round 2 (two of which were in Kenya), and just two sub-grants in Round 3. At the time the evaluation was conducted, the initial stages of assessing candidates for Round 4 sub-grants were underway. Therefore, it was inescapable that the data collection focus on sub￾grants made in the “first rounds” of the three rounds of sub-grants. The Social Impact evaluation team observed a Round 4 meeting during which applicants were short￾listed and interviewed Fintrac staff about the lessons learned and changes made between each Round in the process of emitting sub-grant solicitations, assessing applicants and choosing sub-grantees. Findings on this are set out in the evaluation report. The SMOG was used for sub-grants in all rounds, so the evaluation did cover that “key” program innovation. Social Impact states clearly in several places in the report that focusing the evaluation on two countries is a limitation. However, we do not believe that this limitation invalidates the findings, conclusions and recommendations. We were told by Fintrac staff that the design of the SMOG mechanism was the same in each country (with adjustments made for each individual sub-grantee), that the negotiation process was the same, that the information provided to sub-grantees was the same and that the process and practices for following up with sub-grantees and assessing progress against milestones was the same for sub-grants in each country. Fintrac’s internal protocols were also the same across countries. While any sample short of a census is “incomplete,” we do not think that there are substantive barriers to drawing conclusions and making recommendations based on findings on the SMOG (evaluation question 2), measuring increases in smallholder men and women farmers’ access to technologies (evaluation question 3), or P4I management by Fintrac and BFS (evaluation question 4) based on interviews and observations in Kenya and Mozambique and interviews conducted with Fintrac, USAID, and sub-grantee staff. That said, without having interviewed Mission staff in other countries where P4I operates, it is not possible to know how representative the Mission levels of engagement with the private sector in Kenya and Mozambique were (evaluation question 1) or how representative Mission participation in P4I (a portion of evaluation question 4) in Kenya and Mozambique were. We pointed out this limitation in the report, and, importantly, we took this into account when we designed the recommendations. SMOG: The evaluation team agrees that the SMOG does, as Fintrac asserts, show “great potential as a tool for working with the private sector.” The evaluation team did not question the utility of SMOGs. The report does, however, note that there are circumstances where SMOGs appear to be more and less useful and appropriate. We stand by those findings and conclusions. P4I staff informed the evaluation team that the design, implementation and management of the SMOGs were the same across the board, with no difference in design or management by Fintrac in different countries. While, given the small number of sub-grants, no sample could provide a perfect representation, given Fintrac’s assertion that they did not use different SMOG mechanism or different SMOG management in different countries, we are not clear why the sample of 41% of the extant sub￾grants is not reasonable for this evaluation. We do not agree that there are only two possibilities for extending sub-grants, as Fintrac’s statement appears to assert. 94 Improvements across Grant Rounds: As noted above, the evaluation team did describe improvements over time in the grant making process, and we commend P4I for identifying challenges and making the improvements. The evaluation team also commends P4I on reducing the time from sub￾grant application to award. The table in the report clearly indicates that it shows the time for the nine sub-grants covered by the evaluation. The team was not provided information on the timing for other sub-grant awards, and Round 4 grants had not yet been awarded. It is true that the evaluation would have been “more instructive” had it been conducted after Component 1 Round 4 and had it included more countries. However, Social Impact was assigned by USAID to do the evaluation in early 2015 and in the countries selected by USAID. The scope and budget of the evaluation did not include work in other countries. Evaluation Structure: We agree that the evaluation report structure is not perfect. The evaluation team worked with USAID BFS staff to explore a variety of options. It was determined that is was not practical to completely separate findings, conclusions, and recommendations for Component 1 and Component 2 as they had so many elements in common in relation to the evaluation questions. Likewise, it would have been too repetitive to completely separate out recommendations for current and future programming. Many recommendations serve for both and context would have needed to have been repeated as well. In the interests of keeping the length of the report manageable, to make clear that an recommendation was specifically for P4I rather than a recommendation for programming in general, the evaluation team decided to preface each recommendation for P4I with “P4I should” or the equivalent. Recommendations that are not specifically for P4I are prefaced with “USAID should” or “The implementing partner should” or the equivalent. 95 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523