TOPS Summative Evaluation FinalReport March 2019 Technical and Operational Performance Support (TOPS) Program Summative Evaluation Final Report March 14, 2019 Kent Glenzer Thabo Mubukwanu Catharine Struss 1 TOPS Summative Evaluation FinalReport, March 2019 2 Acknowledgements The evaluation team thanks TOPS Bridge program staff for administration of the evaluation process. Julia Crowley and Dasha Kosmynina provided all the support we could ask for, while Mark Fritzler and Laurie Starr gave strategic guidance that focused and sharpened our inquiry. Many thanks to reviewers of the draft report, too; your comments led to a much improved analysis. TOPS’ project documentation – records of performance, learning, and adaptive processes – was excellent. This enhanced our ability to focus our interviews: hats off to all TOPS staff over the life of the grant that contributed to the management of project performance information and data. We were impressed with the energy that respondents brought to interviews. Stakeholders were keen to help us understand TOPS’ past contributions and interested and vested in that future. Kent, Thabo, Catharine TOPS Summative Evaluation FinalReport, March 2019 2 Table of Contents Executive Summary ................................................................................................................................ 5 I. Overview of the Technical and Operational Performance Support (TOPS) Program ......................... 8 II. Evaluation Questions, Design and Methods.................................................................................... 10 III. Limitations.................................................................................................................................... 13 IV. Evaluation Questions: Data Analysis.............................................................................................. 14 1. To what extent did knowledge sharing meetings contribute to knowledge capture, knowledge generation, knowledge application, and knowledge sharing? .......................................................... 14 2. What were the strengths and weaknesses of the topic-specific, sectoral task forces, inrelation to knowledge capture, generation, application and sharing? ............................................................ 17 3. What were the most valuable components or uses of the Food Security and Nutrition (FSN) Network and why?......................................................................................................................... 20 4. To what extent did TOPS trainings translate to changed practices for food security practitioners? ... 23 5. How effective were the stakeholder consultations? ......................................................................... 32 6. To what extent did the products of Small Grants Program-funded projects result in food security IPs changing practices or methods?................................................................................... 34 7. How effective was the management of the TOPS Program?............................................................. 36 8. How sustainable are TOPS initiatives? ............................................................................................ 40 V. Conclusion ....................................................................................................................................... 41 ANNEXES Annex 1 Evaluation Scope of Work ...................................................................................................... 45 Annex 2 Emblematic “Positive Deviant” Learning CaseData Tables .................................................... 52 Commodity Management Data Table................................................................................................ 52 Social and Behavioral Change Data Table......................................................................................... 54 Permagardens Ethiopia Data Table.................................................................................................... 57 Resilience Data Table ....................................................................................................................... 59 Theory of Change Data Table ........................................................................................................... 62 Annex 3 Formal Stakeholder Consultations........................................................................................... 68 TOPS Summative Evaluation FinalReport, March 2019 2 Acronym List ANRM Agriculture and Natural Resource Management AOR Agreement Officer Representative CG Care Group CM Commodity Management COP Chief of Party CP Community of Practice CRS Catholic Relief Services CY Calendar Year DFSA Development Food Security Activity FANTA Food and Nutrition Technical Assistance Project FFP Food for Peace FH Food for the Hungry FSN Network Food Security and Nutrition Network ICB Institutional Capacity Building IDEAL Implementer-led Design, Evidence, Analysis and Learning IP Implementing partner KM Knowledge Management KS Knowledge Sharing KSM Knowledge sharing meeting LWA Leader with Associates M&E Monitoring and Evaluation MC Mercy Corps MG Micro Grant NGO Non-Governmental Organization OAA Office of Acquisition and Assistance PAC Program Advisory Committee PIA Program Improvement Award PMT Program Management Team PR Program Results PVO Private Voluntary Organization PY Program Year REAL Resilience Evaluation, Analysis and Learning RFA Request for Assistance SBC Social and Behavior Change SC Save the Children SCALE Strengthening Capacity in Agriculture, Livelihoods, and Environment SGP Small Grants Program SOW Scope of Work TANGO Technical Assistance to Non-Governmental Organizations TF Task Force TIPS Trials of Improved Practices TOPS Technical and Operational Performance Support Program USAID United States Agency for International Development TOPS Summative Evaluation FinalReport, March 2019 5 Executive Summary The Technical and Operational Performance Support (TOPS) program, which ran from August 2010 until January 2018, met or surpassed nearly all agreed performance measures. It consistently did so within or under budget and not infrequently ahead of schedule. The program did this while taking on a large number of unplanned activities and deliverables at the request of the donor. More, the program accomplished such results while creating a new way of doing capacity-building and inter-organizational learning across the global community of Food for Peace (FFP) development programs and doing so through an experimental structure comprising five equal PVO partners in a novel consortium. The degree of difficulty of this – delivering products and services of value while inventing the very mode and manner of delivery – should not be overlooked. By nearly any performance measure derived from the program’s formal accountabilities, logical framework, and theory of change, TOPS was an unqualified success. Relying on a variety of qualitative and interpretive methods, the evaluation team pursued eight questions: 1. To what extent did knowledge sharing meetings contribute to knowledge capture,knowledge generation, knowledge application, and knowledge sharing? 2. What were the strengths and weaknesses of the topic-specific, sectoral task forces in relation to knowledge capture, generation, application and sharing? 3. What were the most valuable components or uses of the food security and nutrition network and why? 4. To what extent did TOPS trainings translate to changed practices for food security practitioners? 5. How effective were the stakeholder consultations? 6. To what extent did the products of small grants program-funded projects result in food security implementing partners changing practices or methods? 7. How effective was the management of the TOPS Program? 8. How sustainable are TOPS initiatives? Inquiry into these questions surfaced five important contributions of TOPS to the FFP food security development program community.1 Also emerging were four challenges to be overcome in future initiatives similar to TOPS, challenges which are structural in nature rather than rooted in the performance of any single individual or organization. The table below summarizes contributions/challenges and identifies lessons related to each. Contributions Lessons 1. TOPS’ role in facilitating dialogue between Food for Peace (FFP) and implementing partners (IP) at the global level – particularly but not limited to the formal stakeholder consultations – changed the nature of the relationship between the donor and its grantees. It led to global FFP policies, practices, and procedures that otherwise might not have arisen. In conjunction with a number of venues for informal exchange it created a space for dialogue and debate that minimized the power dynamic between grant makers and grant receivers. 1.1 Preparing for a complicated and possibly conflictual dialogue such as TOPS’ formal stakeholder consultations is time consuming. It requires several weeks of full-time work by a skilled qualitative researcher. 1.2 Preparation in advance of questions/issues, and different opinions on those questions/issues, and distribution of these opinions prior to the consultation aids in constructive dialogue. 1.3 FFP’s internal organizational culture shifted during the life of TOPS, greatly contributing to the establishment of new kinds of collaborative relations. 1.4 The FFP Agreement Officer Representative (AOR) exhibited extensive leadership and boundary management to give TOPS room for maneuver, space to operate, and license to experiment. 1.5 IP expertise and ability to be precise/actionable in their feedback to FFP aided influencing. 1 More granular strengths, successes, and contributions are found in the body of this report. TOPS Summative Evaluation FinalReport, March 2019 6 2. The TOPS program forged greater collaboration and mutual trust between IPs at the global level. 2.1 Creation of a powerful TOPS identity – separate from the identities of the five consortium members – helped create convening power, convening authority, and trust. 2.2 Performance measures/mechanisms – the Program Advisory Committee, % of small grants going to non￾consortium members, % of non-consortium member presenting at knowledge sharing meetings, Task Force demographics – kept TOPS management focused on inclusiveness. 3. The Food Security and Nutrition (FSN) Network recommended tools on the FSN website were mostly seen as highly practical by the intended IP audience. 3.1 Production of tools by multi-agency task forces/groups defused some territoriality, but not all. 3.2 Practitioners producing for practitioners – rather than academics, or consultants doing it – helped with uptake. 3.3 Tensions existed between simple/practical and comprehensive/academically or technically rigorous. Some level of tension is unavoidable and both forms of tools are likely needed. 4. Regional KSMs forged relationships, connections, and empathy among the FFP food security development program community and were highly valued by participants. 4.1 Active, adult learning techniques made the events uniquely useful, according to participants. 4.2 Agenda-building was time consuming and the work load needed should not be underestimated in the future. 4.3 Front line staff in DFSAs needed to receive extra guidance and assistance to successfully present and facilitate sessions. 5. The small grants program allowed smaller Private Voluntary Organizations (PVOs) to produce tools and research they otherwise would not have had the resources to do. 5.1 Grant sizes of $50,000 and $100,000 were substantial enough to incentivize most IPs. 5.2 Mutual reviewing of all grants – no matter the size – helped TOPS and FFP headquarters staff be on the same page about grant making. 5.3 The potential of small grants to develop and scale promising practices required careful coordination with grantees, technical areas, and FFP. An open transom approach did not align well with the objective of scale. Challenges Lessons 1. Stakeholders were not fully aware of the influence that the IP community had on FFP thinking, practice, and policy through TOPS’ facilitation role. 1.1 Follow up communication to the IP community needed to be more consistent. 1.2 Similar communications were needed within the FFP global organization. 2. Degree and quality of adoption within FFP development programs of skill trainings offered by TOPS was not any stakeholder’sresponsibility. 2.1 An explicit theory of change needed to guide this work. 2.2 Follow up to trainings should be part of DFSA annual operating plans and be a required topic in mid-term and summative evaluations. This could take the form of incorporation into DFSA M&E plans. 2.3 Some agent needed to be formally tasked with follow￾up communications with trainees. 2.4 Training and skill-building agendas coordinated with IP senior staff and FFP field staff would help with follow-through and continuity. 3. FFP field staff struggled to understand TOPS’ purpose, role, and ability to 3.1 Relying greatly on TOPS to promote itself to FFP field staff made purpose/role clarity harder. More active and continuous promotion of TOPS within the TOPS Summative Evaluation FinalReport, March 2019 7 add value throughout the life of the program. FFP global network – by FFP staff – would have helped. 3.2 The connection between FFP field staff priorities and TOPS’ capabilities needed to be made more explicit. 3.3 TOPS’ philosophy and strategy of being field-driven and field responsive – avoiding top-down initiatives -- contributed to problems of role clarity. 3.4 A delegative approach to managing consortium partners – giving partners space to act independently – contributed to challenges of role clarity and value added of TOPS. 3.5 A number of other United States Agency for International Development (USAID) funded entities existed that bore similarities to TOPS and field staff did not necessarily understand who was who. 4. Emergent, adaptive work, work that required learning-by-doing and iterative procedures was a challenge for IP field staff and TOPS’ role in and for the field was sometimes ambiguous for them. 4.1 Clarity about practices that are required vs. practices that are optional but strongly recommended vs. practices that are optional would have been useful. 4.2 When capacity-building on adaptive challenges is shared among actors, the role/responsibility of each actor needs careful articulation and extensive and repeated communication to stakeholders 4.3 Adaptive work that has implications across the core business processes of DFSA planning, review, oversight, implementation, and evaluation require orientation of all staff who engage with IPs on those processes. TOPS Summative Evaluation FinalReport, March 2019 8 I. Overview of the Technical and Operational Performance Support (TOPS) Program The TOPS Program, which ran from August 13, 2010 to January 31, 2018 was conceived to: facilitate knowledge exchange among practitioners in the food security community for two primary purposes: to improve the efficiency and effectiveness of U.S. Government-funded development food security activities; and to strengthen the knowledge, skills, and practices of the implementing partners (IPs) in delivering those activities to improve the lives of vulnerable families around the world.2 Conceptualized by Food for Peace (FFP) after realizing that decades of Institutional Capacity Building (ICB) grants to individual Private Voluntary Organizations (PVOs) had created little shared learning or widely scaled good practices, the program was unconventional in purpose, creative in approach, and courageous in its focus on adaptive learning. It was a calculated leap of faith for FFP to base a program on knowledge management and knowledge sharing activities, and an exciting challenge for Save the Children – the prime on the grant -- and the TOPS consortium members, despite having few precedents to guide their actions. The TOPS Program began with two linked principles that guided all actors as the journey progressed:  Openness and inclusiveness: From the start, TOPS adopted a “big tent” approach―the entire implementing partner community was invited to join, share, set priorities and pursue jointly identified interests.  Dialogue among equals: TOPS and FFP both recognized the need for collaboration, mutual respect and constructive consultation between donor and implementing partners to jointly advance food security activities. TOPS Strategies The program adopted five strategies to guide its work: 1. A directed program of capacity strengthening activities for FFP implementing partners and related actors, led by TOPS technical staff. 2. An inclusive community of practice developed through the Food Security and Nutrition (FSN) Network and its task forces and interest groups; membership was open to all FFP implementing partners and other food security and nutrition stakeholders. 3. A Program Advisory Committee made up of food security and nutrition stakeholder representatives to provide strategic guidance on TOPS activities on a regular basis. 4. A food security and nutrition website (www.fsnnetwork.org), an important component foundation of the TOPS knowledge-sharing system. 5. A Small Grants Program that promoted the creation, use, and adaptation of capacity strengthening tools, by implementing partners.3 TOPS Strategic Objective and Theory of Change The TOPS Strategic Objective is “highest quality information, knowledge, and best practices for improved methodologies in Title II food assistance commodity program performance identified, established, shared, and adapted.” 4 The program’s theory of change led to specification of four program results (PR) areas: PR1: Knowledge capture: Knowledge and skill needs of audiences identified. PR2: Knowledge generation: Reliable, high-quality information synthesized and produced in user-friendly, appropriate formats. 2 The TOPS Program. 2018. The Technical and Operational Performance Support (TOPS) Program Final Report: August, 2010 to January, 2018. Washington, DC: The TOPS Program, p. 1. 3 Another component of TOPS was Leader With Associate (LWA) awards. These were not an object of inquiry in this evaluation. 4 The TOPS Program, Ibid. TOPS Summative Evaluation FinalReport, March 2019 9 PR3: Knowledge application: Effective and appropriate traditional and non-traditional skill delivery approaches and systems / applications used. PR4: Knowledge sharing: Information, skill and knowledge exchange supported and expanded. While each of the four PRs produced stand-alone outputs, they were interdependent components of the TOPS Program as a whole. Activities undertaken in one PR were integral to achieving results in others. Figure One below summarizes the TOPS Strategic Objective and intermediary Program Results. Figure One: TOPS Program Framework and Theory of Change TOPS Consortium Save the Children led a consortium of five organizations, each bringing different technical expertise to the effort. Together, the TOPS consortium focused on thematic and technical areas deemed important for food security and TOPS Summative Evaluation FinalReport, March 2019 10 nutrition programming. Each consortium member led activities and bore responsibility for attaining program results in one (or more) specialties, summarized in Figure Two. Figure Two: Consortium Member & Thematic Area and Type TOPS Timeline The TOPS program’s implementation calendar is shown in Figure Three. This evaluation does not assess or analyze the TOPS Bridge Associate Award period. Figure Three: TOPS Timeline II. Evaluation Questions, Design and Methods Annex 1 contains the original scope of work for the evaluation. The evaluation team, consisting of a lead, deputy, and a junior evaluator, worked with TOPS leadership in the pre-award and start-up phase to fine tune/modify a number of evaluation sub-questions leading to prioritization of effort – which questions to spend more and less time on – and to agree on the overall approach and methods. The evaluation permitted no field travel. Inquiry methods were: A. Key Informant Interviews 1. Interviewees were selected through stratified snowball sampling. First, in collaboration with TOPS leadership, important types of respondents were identified. These types were:  Field staff implementing FFP development programs  FFP field staff  FFP HQ staff  Implementing Partner (IP) HQ staff  TOPS staff TOPS Summative Evaluation FinalReport, March 2019 11  TOPS consortium member staff (not working inside of TOPS)  Program Advisory Committee (PAC) members Eighty-two people were interviewed, some more than once. Figure Four below gives several different views on interviewee characteristics. Because some interviewees fall into more than one category, the numbers below add up to more than 82 Figure 4: Numbers of Interviews by Respondent Category PAC Members TOPS Staff FFP Staff HQ/Field USAID HQ Staff IP Staff (HQ/Field) HQ/Field Staff (Total, all) Other 10 19 9/6 4 19/10 58/22 2 2. TOPS staff generated an initial list of interviewees in these categories. The interview list was expanded during the evaluation process through snowball sampling. 3. Interview protocols were developed for each category of stakeholder. They served as guides, but not rigid protocols, for interviews. 4. Interviews were done via Skype, Zoom, UberConference or telephone. The interviews lasted between 25-75 minutes, although it was rare to go longer than 60 minutes. Interviewees were informed that their responses were anonymous. 5. Interview documentation was available only to the three members of the evaluation team. 6. Whenever possible, two members of the evaluation team participated in each interview. Both evaluation team members took notes – near-transcripts due to typing speed – during the interviews, compared notes afterward, and produced a consensus summary of the interviews. 7. All texts were uploaded into Dedoose – a qualitative data analysis software – for coding and analysis. B. Secondary Documentation 1. Project documentation was provided by TOPS. 2. Documentation was divided among the three members of the team. 3. Each document was read and coded by two members of the team. 4. Coding discrepancies were discussed and resolved. A code book was developed at the start of the evaluation using deductive coding based on the evaluation questions. New codes emerged as the evaluation move forward, and recoding of previous work was an ongoing process. Inter￾coder reliability was fostered through a start-up process in which all three members of the team coded texts independently, compared results, and refined/redefined code definitions and application protocols. C. Emblematic “Positive Deviance” Case Studies The evaluation Scope of Work called for inquiry into adoption of new practices by IPs. TOPS had tried this previously – three times – without satisfying results. Hurdles included a) low response rates to written surveys, b) difficulty in tracking down past participants, and c) verification of subjective reports. Rather than repeat previous approaches, the evaluation team proposed developing a small number of emblematic “positive deviance” case studies to surface probable successes and lessons for improving training transfer in future initiatives. Positive Deviance is based on the observation that in every community there are certain individuals or groups whose uncommon behaviors and strategies enable them to find better TOPS Summative Evaluation FinalReport, March 2019 13 solutions to problems than their peers, while having access to the same resources and facing similar or worse challenges.5 We followed an inductive process to identify cases, triangulating between TOPS staff opinions, FFP HQ staff input, field staff reports, and project documentation. We also considered the overall strategic significance of the subject matter of the training. The selection of cases occurred after two months of study and once identified – and agreed with TOPS Bridge leadership – led to deeper dives which entailed further reading, gathering additional written matter from participants/organizations, and additional interviewing. More detail on the methodology related to emblematic, positive deviance cases can be found on pages 23-26 below. Figure 5 below synthesizes the three principle methods and their relationship to evaluation questions. Figure 5: Methods x Evaluation Questions Evaluation Questions Methods Key informant interviews (N= 82) Document/ Database analysis Positive Deviance Case Studies 1. To what extent did Knowledge Sharing Meetings contribute to Knowledge Capture, Knowledge Generation, Knowledge Application, and Knowledge Sharing? X X 2. What were the strengths and weaknesses of the topic-specific, sectoral task forces (e.g., Ag/NRM), in relation to knowledge capture, generation, application and sharing? X X 3. What were the most valuable components or uses of the FSN Network and why? X X X 4. To what extent did TOPS trainings translate to changed practices for food security practitioners? X X X 5. How effective were the stakeholder consultations? X X 6. To what extent did the products of Small Grants Program-funded projects result in food security IPs changing practices or methods? X X X 7. How effective was the management of the TOPS Program? X X 8. How sustainable are TOPS initiatives? X X Figure 6 below shows the sequencing of methods. Capital “X” indicates high effort, lowercase “x” equals more moderate effort. Note that the evaluation was originally planned to end by December 31, 2018; the timeline was extended by one month. 5 https://positivedeviance.org/, accessed 2-28-19. See also, Richard T. Pascale, Jerry Sternin, and Monique Sternin, Positive Deviance: How Unlikely Innovators Solve the World’s Toughest Problems, Boston, MA: Harvard Business Press, 2010. TOPS Summative Evaluation FinalReport, March 2019 13 Figure 6: Evaluation Calendar Activities/Foci Sep Oct Nov Dec Jan Feb￾March 24- 28 1-5 8- 12 15- 19 22- 26 29- 2 5-9 12- 16 19- 23 26- 30 Review of Key Documents X X x x x X x x Key Informant interviews: TOPS consortium leaders and TOPS technical leads X X X Key Informant Interviews: PAC members X X X X Key Informant Interviews: IP X x X x x x x Key Informant Interviews: FFP (field) X x Key Informant Interviews: FFP HQ X X X Mid-Process Check-in in DC X Positive Deviance Case Study Development x x x x X X X Draft Report Submitted and discussed in DC meeting X Revision of report and finalization X X X III. Limitations The evaluation was qualitative in nature and so does not offer findings that are statistically valid. Methods that might have permitted generalization – such as surveys with representative sampling – had proven highly problematic in TOPS’ past. Rather than possibly expend resources on another attempt to secure a sufficient sample, TOPS Bridge leadership and the evaluation team agreed to focus more strongly on interviewing, theory-driven interpretation of qualitative data, and case studies. Interviews are subject to a number of forms of bias. First, they can suffer from instrumentation effects, in which the interviewer’s set of questions skew responses. To minimize this, interview protocols were shared for analysis of bias with third party evaluators at the lead evaluator’s academic institution. Second, evaluators can exercise selective listening. We combatted this by the norm of two evaluators attending each interview. Third, interviews in a summative evaluation rely on respondents’ memory. We had available to us a comprehensive participant list in all of TOPS’ KM and training activities which allowed us both to check the accuracy of memory and remind respondents of their physical involvements in TOPS. Having read all TOPS’ annual reports while coding them, evaluators were also able to catch memory errors on the spot, in a conversational manner. Finally, where respondent claims were important, we sought follow up documentation to affirm pragmatic details of their reports. While 82 interviews are substantial, this still represents a small number of TOPS program participants. As is well known, the answer to the question, “how many interviews is enough in qualitative inquiry,” is debated, as Harry Wolcott, a pioneer among qualitative researchers, once commented: That is, of course, a perennial question if not a great one. The answer, as with all things qualitative, is “it depends.” It depends on your resources, how important the question is to the research, and even to how many respondents are enough to satisfy [reviewers]. For many qualitative studies one respondent is all you need – your person of interest. But in general the old rule seems to hold that you keep asking as long as you are getting different answers… 6 The number of interviews conducted in this evaluation is related to Wolcott’s insight. We endeavored to seek additional interviews as long as we kept hearing different answers. Anthropologists call this reaching saturation and we felt this was achieved. 6 Sarah Elsie Baker and Rosalind Edwards, “How many qualitative interviews is enough? Expert voices and early career reflections on sampling and cases in qualitative research,” National Centre for Research Methods Review Paper, Southampton, UK: NCRM, 2012, pp 3-4. TOPS Summative Evaluation FinalReport, March 2019 14 The evaluation had no field travel. Respondents were generous with their time – nearly every respondent we reached out to reached back, and reached back in a timely manner – but richness and precision would have been greatly enhanced had we the ability, particularly related to the emblematic learning cases, to spend time in the field. IV. Evaluation Questions: Data Analysis 1. To what extent did knowledge sharing meetings contribute to knowledge capture, knowledge generation, knowledge application, and knowledge sharing? Knowledge sharing meetings (KSMs) were, as one informant put it, TOPS’ “bread and butter.” Another captured the overall tone of many other respondents, saying that KSMs “were a tremendous gift to the field.” The ten KSMs collectively brought together 1,639 participants, representing between 37-99 organizations. Post-event evaluations were positive, and TOPS knowledge management staff clearly learned from them, improving their utility over time.7 Respondents were nearly unanimous in their praise – the extent and depth of this is one of the strongest findings of this evaluation. They were “an amazing source of information and opportunity for organizations to share their learning and failures,” said one informant. One field staffer from an implementing partner underscored the social and professional capital the meetings built: A very important forum for sharing experiences across the PVO community….it allowed national staff in [NGOs] to participate and learn from staff around the world – not just [us, internally] …. It boosted confidence and learning both. We asked nearly all interviewees the question, “Broadly speaking, what was TOPS’ most important added value to the implementing partner community?” and the first response in most cases was KSMs. One FFP field officer, noting that such officers rarely have time to lift their head and connect with peers in other countries, called TOPS’ KSMs “connective tissues across Food for Peace.” A Chief of Party (COP) called attention to the spirt of collaboration created among formerly wary parties: The learning and sharing events are remarkable. It is a safe space; we are all there. FFP takes the spirit of it, nobody swaggers. I don’t see that in any other donor area, not even in PEPFAR. FFP uniquely with TOPS has been able to do this, and push some cutting edge learning out there. Many participants underscored the quality of meeting organization and the engaged and active knowledge sharing and learning methodologies that were deployed. Yet while the overall value of KSMs, from the perspective of IPs, is clear, the extent to which KSMs contributed to the four different PRs varied greatly. A. Knowledge Capture TOPS operationally defined “knowledge capture” as the process of identifying what food security and nutrition practitioners needed to know and the tools/information existing to help ensure they could meet those needs. Establishing needs was by design a bottom-up, participatory process that prioritized the stated needs of implementing partners. TOPS’s records show that all KSM agendas were built through a consultative process in which field and HQ staff of IPs, consortium technical staff, and FFP employees provided input on priority content. TOPS made a concerted effort to respond to these inputs, establishing in each KSM an array of 7 Quite a large number of respondents spoke to us within days/weeks of attending the TOPS Bridge program’s KSM in Thailand in September 2018. Those with multiple KSMs under their belt noted that the Thailand meeting was the best in the history of TOPS. Formalistically, this falls outside the boundaries of our scope of work however as TOPS ended in January 2018. Component Extent Knowledge Capture Medium TOPS Summative Evaluation FinalReport, March 2019 15 concurrent sessions where specific needs were addressed. At the same time, KSMs often had top-level, overarching themes – resilience, FFP’s strategy, KM/capacity building itself– that were driven by a desire to proactively push ideas out into the IP community. This latter activity was important for the goal of knowledge sharing (see below), yet it did conflict with the ethos and values of bottom-up knowledge capture that is embedded in the TOPS strategy. Combining agenda setting in this way led to challenges for IPs in selecting the “right” participants to send to a KSM, right in terms of experience, seniority, and ability to influence back-home practices and priorities. This, in turn, meant that the TOPS goal of being a connector to “the tools/information existing to help ensure they could meet” the stated needs of IPs in the context of KSMs was inconsistent. A large number of respondents confirmed that as much as they appreciated the active, adult-learning methods they encountered in KSMs, they also found the content hard to connect to their grassroots realities back in their programs. A small number of interviewees mentioned, too, that it was difficult to fully tap into the knowledge and experience of national staff at KSM events. B. Knowledge Generation TOPS operationally defined knowledge generation as creating tools and information for the food security and nutrition community. The program did this in many ways, only one of which was KSMs. As with knowledge capture, the program prioritized bottom-up processes of identifying tool/competency gaps while incorporating a small number of top-down foci. Interviewees converged strongly on a particular kind of knowledge generation that occurred at KSMs: relational and interpersonal. One IP Chief of Party (COP) was incisive and articulate on this front: the meetings, he said, loosened up the dialogue between IPs, broke down walls between organizations, and created a more open dialogue about failures and successes. Another COP noted that the core value of KSMs was “the connections you made…. you would go into a meeting and meet somebody from [another region or country] …and those connections are enduring and useful.” A FFP field officer went further: These KSMs were about sitting shoulder to shoulder with people and not them presenting to us. I do think that was something that previous to TOPS didn’t happen. Everybody’s on pins and needles [in those pre-TOPS meetings]. With TOPS…you’re talking about the technical subject and you’re all wrestling with it. In presenting the subject matter as a problem to be solved together helped to sort of break that down. Instead of donors and grantees, you were graduate students in the same class again. A COP affirmed the creation of a safe space, and noted the deliberateness with which it was created, calling KSMs “A terrific platform…in a very safe and open environment.” The COP noted that “everybody checks their agency hats at the door” at the events “and that’s in no small part due to the philosophy that TOPS brings to it.” Connecting the comments of many respondents who participated in KSMs, the evaluation team believes that KSMs created social, cultural, and professional bonds and connections. It would appear that such interpersonal and informal connections facilitated the emergence of tacit into quasi-explicit knowledge. This is an act of knowledge creation. It fosters knowledge of each other, it transforms distant names and email addresses into sentient human beings, and it generates knowledge of “the other” – whether that other is a COP, a technical lead officer in a DFSA, or an AOR/CBO in DC – that makes future communication more empathetic. The value of the informal sharing of experiences, thoughts, and opinions that permeated KSMs may best be framed by the 70/20/10 learning concept: Seventy percent of learning takes place from real-life and on-the￾job experiences; 20 percent from other people through feedback, mentoring, or coaching; and 10 percent Component Extent Knowledge Generation Medium￾High TOPS Summative Evaluation FinalReport, March 2019 16 from formal training.8 KSMs tapped into the 70 and 20 components of the model while at the same time facilitating the remaining 10 percent via two to three formal training sessions in every event. C. Knowledge Application TOPS operationally defined knowledge application as the development of “effective and appropriate traditional and non-traditional skill delivery approaches” that permitted IPs in “widely different contexts and environments” to acquire and/or strengthen skills for addressing food security risks.9 Knowledge application was addressed in a wide variety of ways, with KSMs being just one of them. KSM agendas always contained two or three formal training sessions intended to result in knowledge application. Many sessions, however, were meant as spaces of sharing experiences, engaging in generative discussions, orientating to new ideas or strategies, and building mutual understanding. For the majority of sessions, knowledge application, strictly defined, was not intended. In Section IV.4 below we set out a conceptual model – a theory of change – documenting the conditions necessary for knowledge application to occur. The model is rooted in widely understood, well-documented, and well￾researched “best practices” to achieve training transfer. Depending on the context, between 8-11 preconditions need to be in place. KSMs ensured only a subset of these 8-11 preconditions. By force of logic, the theory of change would suggest that knowledge application was unlikely to be strong. We found substantial corroboration for this interpretation among all categories of interviewees. “What’s lacking is, after the excitement of the [knowledge sharing] events…and then what?” said one COP, stating that application back home was rare. A number of informants noted that TOPS could have performed more follow-up after KSMs; this was not a contractual accountability for TOPS, but points nonetheless to a weakness in the overall system if application was an important goal. The KSMs, according to another COP who participated in several, were: for higher level people, managers, upwards. Below a manager…. The person who has a real interface with the activity in the field…. I mean, honestly would you expect me to have that time to give a training [after attending a KSM]?... [I]f lower levels go…and give me the action plan, then I can hold that person accountable for implementation. The same COP continued: sharing in a knowledge platform, summit, training, or sharing in a network…it’s not enough. You need to be pushing people, checking on people, and seeing if they’re doing it. Most TOPS technical staff interviewed said that they either concurred with this insight or stated they simply had no way of actually knowing whether or not application occurred. Some pointed to the reality that TOPS was not structured to conduct detailed follow-up, nor could TOPS influence organizational dynamics within implementing partner organizations. 8 Lombardo, M. M. & Eichinger, R. W. (1996). The Career Architect Development Planner. Minneapolis: Lominger. Note that it is an unwise use of this model to be rigid about the actual numbers in the ratio. 9 TOPS, “Annual Progress Report 2011,” p. 13. Component Extent Knowledge Application Low TOPS Summative Evaluation FinalReport, March 2019 17 D. Knowledge Sharing TOPS defined its objective regarding knowledge sharing as supporting and expanding information, skills, and knowledge exchange by creating new physical and virtual community exchange forums, and knowledge sharing mechanisms. The purpose was to strengthen global, regional, and local knowledge sharing and networking.10 End-of-event evaluations confirm the perceived quality of these events. The events were consistently adapted and improved based on such feedback, with several interviewees – who had attended multiple KSMs over the years – stating that they became better and better with each passing opportunity. Documentation of the events was excellent, and materials from KSMs were made available on the FSN online database. The KSMs deployed active methods for adult learners, something that many respondents praised. One HQ IP senior manager summarized the value in two ways: sharing among IPs, and sharing between IPs and donors. It’s been phenomenal to go to the knowledge sharing meetings both here in DC but also regional ones for our staff in the field…. [I]t’s a great way to get ideas for ourselves [about] what’s going on and also to share with others what we’re doing and there’s such a great collaborative spirit to those conferences and meetings… [A]nother aspect of those knowledge sharing meetings is being able to have face to face with the donor representatives and to hear their perspectives and hear their views and everything. Evidence converges on the conclusion that KSMs likely accomplished knowledge sharing to a high degree. The evaluation team concurs with one senior NGO staffer who noted the value of knowledge sharing to help move strategic agendas. KSMs “bring together people, it is at these forums where ideas can be shared,” the official said. “Over time, new ideas gain understanding and traction and eventually can be transformed into policy and resources.” 2. What were the strengths and weaknesses of the topic-specific, sectoral task forces, in relation to knowledge capture, generation, application and sharing? The TOPS FSN Network task forces (TFs) formed in the second quarter of the first year of the project. The TFs were developed using the model of CORE Group’s working groups, and were rooted in the strengths of consortium members. The initial slate of TFs was agriculture (later renamed agriculture and natural resource management), gender, monitoring and evaluation, nutrition and food technology, and social and behavioral change. Commodity management (year two), knowledge management (year two), and resilience (year six) task forces were added subsequently. TOPS original plans were to have TFs gradually become self-sustaining. The TFs were: One of the central mechanisms through which the TOPS Program identifies areas of common need and interest, gaps in capacity, and priorities for action to support capacity-building efforts. The task forces are also instrumental in helping TOPS identify information, tools, and approaches (ITA) that contribute to effective food security and nutrition programs.11 10 Ibid. 11 The TOPS Program, “Technical and Operational Support Program (TOPS) Year 1 Annual Progress Report.” Washington, DC: TOPS, 2011, p. 3. Component Extent Knowledge Sharing High TOPS Summative Evaluation FinalReport, March 2019 18 Each TF lead was given flexibility and authority to organize and run the TF in his/her own manner. “We each had a very different concept of our task force,” said one TOPS technical lead. TFs therefore took on a variety of forms over the life of the project and this diversity was consciously supported by TOPS leadership. While there were differences, all TFs a) sought and secured members from outside the consortium, and b) developed consensus forms of decision making. Knowledge Capture/Generation One of TOPS first tasks, early in the project, was to seek, review, and warehouse best practices that the PVO community had developed during decades of the ICB approach. As documented in the program’s early annual reports, the amount of this material was less than expected. Knowledge capture continued throughout the life of the program, taking the form of webinars, sessions at KSMs, and TF meetings in which guest speakers participated, among other activities. TFs produced TOPS recommended tools. The work could be “really super energizing for the task force…and people loved having a project that they were excited about,” commented one TOPS technical lead. These recommended tools were frequently mentioned by IP interviewees as a useful contribution of the TOPS program. Nearly all such tools were referenced as useful resources by FFP in RFA guidance. Several interviewees praised TOPS’ tools as practical and user friendly, and suggested this was due to the fact of practitioners producing with and for other practitioners. TFs also produced sets of core competencies for their technical area and these were mentioned as helpful by several IP interviewees. TF work took different shape across TOPS. They might be used by technical leads to seek input on policies and practices of FFP (such as developing consensus language for FFP to include in the FY13 RFA for Development Food Assistance Programs to promote an enabling environment among PVOs and donor support for knowledge sharing, or inclusion of funding for gender analysis in RFAs), as seedbeds for the emergence of small grant proposals (and subsequently homes for knowledge products coming from those proposals), to conduct surveys, or to gather and summarize feedback on a document (such as the Tufts Food Aid Quality Review). It bears mentioning that a handful of interviewees – technical experts themselves – questioned what they saw as an over-reliance on bottom-up approaches for knowledge capture/generation. They worried that TOPS might not be paying enough attention to innovations in practice happening outside the IP community. TOPS staff counter this opinion by pointing to the role of KSMs in the overall program, where sessions on innovative new ideas coming from outside the IP community were discussed. Knowledge Application TF members sometimes – depending on the TF, depending on the specific training – reviewed materials intended for use in knowledge application events/processes by a TOPS technical lead, helped conduct a particular field-based training, or helped conceive/design a webinar with knowledge application aims. TFs however were not designed, per se, to be responsible for knowledge application. TOPS’ overall strengths and challenges regarding knowledge application are covered later in this report. Knowledge Sharing Knowledge sharing was, for most TF members interviewed, of important value: There was a lot of passion and commitment…It stood out as one of those where there was more commitment to engage and follow-up. People were very networked. We knew each other and we share a lot. So I think it was one of the bright spot areas. (TOPS Technical and TF lead) TFs served for some as a knowledge and information sharing hub where new research might be distributed, updates on issues of concern to the technical area might be distributed, and members could even seek personal coaching on issues they were dealing with on the technical front: TOPS Summative Evaluation FinalReport, March 2019 19 I loved the task forces. We used them a lot as private support. Many task forces members would email me and say, I’m embarrassed but I don’t know what is going on with . (TOPS technical lead) For one TOPS technical lead, the TFs were: very helpful because people had a chance to meet their counterparts from different organization and different orgs used different approaches. They would exchange ideas. They would have a chance also to work together. For example: small grants paper writing, teams coming from different orgs would work together on the same project. They were learning. Another TOPS technical lead concurred with this saying that TFs were useful mechanisms to get people out of their “ruts” – their comfortable and known spaces of practice – and offer pathways towards leadership opportunities and personal development. At the same time, nearly all TF members interviewed noted how difficult it was to fully, equally, and equitably include field staff. Simple challenges around time zones were at play, as were issues of connectivity and language. We also heard from several technical leads that maintaining energy and interest in the TFs was hard: I would say on the flip side the activities that I found were less successful for me were the task forces. Trying to get engagement and meaningful participation in those were difficult because you’re dealing with people who are busy have a lot going on have a lot of demands from their own organizations so no one could fully give their attention to the task forces. Another TOPS technical lead echoed this: The TOPS person was always the one at the top of the task force, making things happen. Certain task forces tried to create shared leadership, and to transfer from TOPS to others, but it didn’t happen. Yet a third technical lead – one who entered TOPS towards the middle of its life span -- stated that his/her TF’s purpose was never clear: we never had a defined terms of reference, or scope of work, nothing that the members had to do, or engage. We had maybe a monthly meeting [a telephone conference, generally] and you could join if you wanted to, don’t if you don’t, maybe we would bring in a guest speaker, maybe we would have a discussion but there was no [purpose]…. I don’t know what the original vision of them was, but they all took their own shape. As can be seen, there is conflicting and contradictory evidence regarding the performance of the TFs with regard to the four PRs. For some the TFs were rich sites of engagement, sharing, tool production, and even policy influence. For others, the TFs seemed flat and without firm purpose. For some, TFs were quite vibrant nodes of engagement and sharing; for others it was difficult to get people to engage consistently and regularly. The evaluation team believes that all TFs contributed important value to the IP community during TOPS. Of particular value appears to be the work done on knowledge capture/generation. In interviewing TF members, it was striking that many of the positive evaluations of the TFs by members related to moments when the TF was working together on a specific product (core competencies, a tool, etc.). It is significant that the TFs experienced significant declines in participation and activity in the last year of TOPS, and that only the gender TF has extended – via a Facebook group – beyond the end of the project. Is it necessarily a bad thing, or a failure, that the TFs seem not to have outlived TOPS itself? Not necessarily. The evaluation team believes, however, that if such sustainability was desired, the TFs were likely under-resourced. TOPS Summative Evaluation FinalReport, March 2019 20 3. What were the most valuable components or uses of the Food Security and Nutrition (FSN) Network and why? TOPS intended the FSN Network to live on after TOPS funding ended. This is apparent in how TOPS defined the network: an open community of practice for food security and nutrition implementers, researchers, and donors. The FSN Network provided venues (website, newsletter, meetings, events, interest groups and task forces) for its members to collaborate and learn, while shaping agendas, influencing donor priorities, building consensus on promising practices, and sharing technical knowledge to strengthen programming.12 As per this definition, KSMs and task forces – covered above – were linked tothe FSN network, as were stakeholder consultations (covered below). Both KSMs and stakeholder consultations13 were perceived by most interviewees as TOPS’ highest strategic value to the FFP food security community. Within that broad generalization, sub-patterns are apparent:  IP HQ senior staff tend to identify stakeholder consultations as higher value than field staff  FFP field staff tend to identify KSMs as higher value than stakeholder consultations  IP field staff tend to identify KSMs as higher value  FFP DC and TOPS staff see the two as of equal value, just serving different objectives After KSMs and stakeholder consultations, interviewees most mentioned the value of the TOPS-vetted tools found on the website. TOPS treated a handful of tools in a special manner, branding them with a “FSN Network recommended” label. Ninety-three percent of these specially branded tools found their way into office FFP RFA guidance, an indicator that the tools were both strategically salient and of acceptable quality. The tools were perceived by many as distinct from similar products available elsewhere. The quote below from an IP partner, while focused on social and behavioral change (SBC) materials, is a good representation of what IP staff found most valuable about these items: What TOPS has very nicely done is to provide a good clear set of tools and approaches, that almost codify SBC practice. The people working with TOPS have really nicely pooled together SBC theory, these more theoretical things, and put them into practical actions. There was an awful lot of academic literature before TOPS….my impression is that previous resources were more theoretical. So, in those 3-4 key resources, TOPS pulls all this together and lays out a clear approach…. Their materials are very action-oriented. They are…directed towards action. I think that was missing from the field of SBC. The utility of these tools and the branding was mentioned many times by informants. “A very helpful toolbox,” said one COP. An IP HQ senior staff called the tool database “an incredible gift to the field” and “a clearinghouse and a disseminator of really top notch work.” It bears mentioning that not all respondents were as approving of TOPS products. Three technical experts interviewed critiqued certain tools for being simplistic. More on this issue is found below in Section IV.4 on emblematic cases, where this dynamic is unpacked with regard to SBC. Another respondent praised not just the reliability of the tools, but the web site interface: It’s quite good, it’s specialized, you can get a lot of guidance and tools. It’s got information that you can access quickly…it’s a site you can rely on for getting proper quality documents. The challenge, as you are navigating a web site, it depends on your own agility, it’s up to me to see something…. 12 The TOPS Program. 2018. The Technical and Operational Performance Support (TOPS) Program Final Report: August, 2010 to January, 2018. Washington, DC: The TOPS Program, p. 6. 13 What, precisely, constituted a “stakeholder consultation” was not clear to most respondents. The ambiguity around this is unpacked in the section below which addresses the evaluation question about stakeholder consultations (see pp 33-34). TOPS Summative Evaluation FinalReport, March 2019 21 Indeed, opinions on user-friendliness spanned a wide range, from one FFP field staff who essentially found it impenetrable to – a more modal opinion among our interviewees – that the site was acceptably navigable. The most oft-mentioned shortcoming was the site’s search function. The quote above points us towards a pattern that can be seen in the data: Seekers who knew what they were looking for, who had a specific, concrete need found the site relatively easy to use. More senior leaders and managers who accessed the site out of general curiosity, to get a feel for what was available, said they found the site less accessible. The site was designed for the former kind of seeker rather than the latter, and its utility and ease of use aligns with that strategy. Reviewing product download data -- there were 144 products (some with sub-component parts) on the website as of mid-November 2018 -- reveals enacted priorities by users. Figure 7 below summarizes the top 20 downloads as of mid-November 2018. Figure 7: Product Downloads 2011-2018 It is worth noting that the number of downloads does not correlate – in theory or in practice – with either use or application. It is even impossible to know whether, once downloaded, the resource was opened. Nor does download data tell us anything about the quality of the documents or ease of use. In other words, that a document was downloaded much more than another does not mean that one document was better, or more important, than one with fewer downloads. Nor, due to privacy laws, do we know who downloaded the files. Finally, it is possible that a very valuable resource was only meant for a small number of users: large numbers of downloads do not necessarily equate with utility. TOPS PRODUCTS DOWNLOADS DATA 2011-2018 Resource Name # of Downloads Monitoring and Evaluation Facilitator's Guide 5744 Designing for Behavior Change: For Agriculture, Natural Resource Management, Health and Nutrition 1173 Care Groups: A Training Manual for Program Design and Implementation 1073 Participatory Facilitation Techniques Workshop Curriculum 1019 Essental Nutrition Actions and Essential Hygiene Actions Framework 996 Warehouse Staff Safety Guide 897 Make Me A Change Agent: A Multisectoral SBC Resource for Community Workers and Field Staff: 535 Barrier Analysis Summary DOC 465 A Guide to Developing a Knowledge Management Strategy for a Food Security and Nutrition Program 461 Reading and Responding to Your Baby Lesson Plan 431 IYCF-E Toolkit: Rapid Start-up Resources for Emergency Nutrition Personnel 424 TOPS Permagarden Toolkit 409 TOPS Review of Promising Practices in Food for Peace Development Food Assistance Projects 398 REALIZE: Social and Behavioral Change for Gender Equity and Diversity 368 Resilience Design in Smallholder Farming Systems Approach 366 Designing for Behavior Change: A Practical Field Guide 330 Nurturing Connections - Adapted for Homestead Food Production and Nutrition 323 Care Groups in Emergencies: Evidence on the Use of Care Groups and Peer Support Groups in Emergency Settings 315 TOPS Summative Evaluation FinalReport, March 2019 22 While all these caveats are true, and great care must be taken with the above table, a counterfactual question reveals its importance despite these caveats: What if the materials had been downloaded in the single digits? Were that the case, we would rightly conclude that this aspect of TOPS failed. From a qualitative perspective, we look for triangulation for our insights, and the download figures corroborate what we heard from many interviewees about the utility of TOPS’ tools. We know, too, that visits to the FSN Network web site were robust. Figure 8 below summarizes the geography and scope of distinct visits over the life of the program:14 Figure 8: Product Downloads by Region These data also raise many questions that the evaluation scope of work did not include. The pattern of visits – highest in North America, followed by Africa and Asia – corresponds to TOPS’ overall level of effort, possibly indicating that the program successfully drove stakeholders to the site. While somewhat beyond the scope of the evaluation question regarding FSN Network strengths and weaknesses, looking at how the FSN website fits within the larger “search world” – i.e., Google – reveals that the site is positioned admirably. Figure 9 below reveals results as of November 28, 2018, of 13 search strings covering topics of importance in TOPS: 14 Ibid., p. 28. Figure 9: Google Search Results for Top Downloads TOPS Summative Evaluation FinalReport, March 2019 23 This indicates that if an IP field or HQ staffer uses Google to find knowledge about the areas covered in the search strings, they get pointed to the FSN Network web site most often on page one of Google’s recommendations. A smaller number of interviewees mentioned the FSN Newsletter as a strength, and not a single interviewee critiqued it. The distribution of the newsletter grew throughout the life of TOPS – see Figure 10 to the right15 – indicating that it had intrinsic value. The TOPS team conducted regular reviews of the newsletter’s contents and format throughout the life of the program, as well as readership surveys. No conclusive patterns about aspects of the newsletter that were particularly valuable arose during our interviews, although news/announcements of events of possible interest to IPs and promotion of IP-generated tools/knowledge were mentioned. 4. To what extent did TOPS trainings translate to changed practices for foodsecurity practitioners? It is important to reiterate that TOPS had no contractual obligations, no formal accountability, for implementing partner behavior change. Implicit, however, in the philosophy or theory underpinning the program’s conceptualization was that practices of IPs would change for the better, and so FFP development programs would achieve greater outcomes and impacts on food and nutrition security. TOPS performed three internal reviews16 which attempted to answer the question of changing practices. All returned anecdotal stories of success, while underscoring methodological limitations which prevented firm conclusions. We, too, gathered many anecdotes of adoption of new practices by IPs after participating in TOPS trainings. We also gathered anecdotes of difficulties in adopting new practices. Given that our evaluation had many of the same methodological limitations as previous efforts to understand the extent to which TOPS trainings translated into changed practices for food security practitioners, a different approach to answering this evaluation question was pursued. 1. We identified a small number of successful cases of transfer of training and dug deeper on each to better understand how and why they were successful. (Please see above Section II.C. “Emblematic ‘Positive Deviance’ Case Studies” on page 11 for a fuller description.). We did not pre-determine these cases from the start of the evaluation. Rather, we identified them from our reading of project documentation, and from interviews. 2. Leveraging long-standing research on training in the workplace, we articulated two closely related theories of change for training transfer, then used them to analyze TOPS successes from a systems standpoint. Our hope was that analyzing successes through a systems lens – a theory of change – would provide decision-makers/planners more useful insights – more useful and actionable than TOPS’ previous efforts to answer this question - for future efforts. Figures 11 and 12 below shows these theories of change. 15 Ibid., p. 29. 16 These were: Lisa Woodson, Laurie Starr and Nancy Mock, “TOPS Training Review,” February 2015; Patrick Connors, “Technical and Operational Performance Support Program Small Grants Review,” December 2016; and Darren Headley and Edith Mutalya, “Technical and Operational Performance Support (TOPS) Program Internal Review Report,” December 2016 TOPS Summative Evaluation FinalReport, March 2019 24 Figure 11: Theory of Change for Required Practices TOPS Summative Evaluation FinalReport, March 2019 25 Trainee Preconditions  Attendees are the right ones for determining/ deciding back-home adoption Figure 12: Theory of Change for Voluntary Practices Why two different theories of change? As we interviewed field staff of IPs, it became evident that from their perspective TOPS trainings could be put into two broad categories:  Training on practices IPs MUST adopt. A simple example of a “required practice” would be doing baselines and endlines.  Training on practices they COULD adopt. A simple example of this is permagardens. While the preconditions for each type overlap greatly, they are not identical. Hence, two different theories of change. 3. Next, based on two months of interviewing and in conjunction with TOPS Bridge leadership, we identified five “positive deviant” cases for deeper inquiry. The cases were posited to be areas where a) there was demonstrated change in IP behavior and b) qualitative data were thicker and richer, permitting greater learning potential. The exclusion of other TOPS’ trainings from this list should not be construed as any judgment or comment about them, or the extent to which they might have led to changed practices among IPs. TOPS Bridge leadership and the evaluation team agreed to deeper dives in relation to the following training efforts: TOPS Summative Evaluation FinalReport, March 2019 26  Commodity Management  Social and Behavioral Change  Permagardens  Resilience  Theory of Change Annex 2 contains data tables for each of the five cases. The tables do not present all data; the intent of the tables is to give readers a feel of patterns, ranges of response, and facts related to each element of the case’s theory of change. Commodity Management According to FFP staff with knowledge and access to the field, commodity management practices have been improved. All IP staff interviewed about this said the same. More so than any of the other emblematic cases, the CM efforts are praised for being field-driven, field-friendly, practical, and useful. Success factors 1. The CM task force was composed of agents who had responsibility for CM effectiveness in their organizations, and had an ethical commitment to “zero-defect” work due to the quite literal life-and-death nature of the commodity management function. 2. There was urgency both in terms of meeting regulatory requirements vis-à-vis the donor, but also because of critical incidents in which staff/community safety was an issue. Such safety concerns, in the past, would likely have remained siloed within one agency, but with the dialogic space opened by the task force, IPs were able to grasp that a) a number of them were experiencing the same problems and b) those problems were serious and failure to get better involved deep moral, ethical and professional identity motivators. 3. All materials were produced by practitioners of CM. There was no distant, detached, body of “experts” that declared the right way to dothings. 4. Working on those materials concretely aided many of the task force members in their day-to￾day work in their own agency. 5. The task force was able to tap into small grants to produce new tools/guidance. 6. The tools/guidance were translated into French and Spanish, making them more useful. 7. CM hired a full-time expert in training and instructional design. Challenges ahead 1. Commodity management is not part of IDEAL. It is unclear how future capacity building will be done. 2. The very champions who made the CM task force a success, and who produced materials commonly acknowledged as useful for all IPs, themselves will move to new jobs. As a result, it is not just front-line staff – the target population for CM training -- whose turnover presents challenges. Social and Behavioral Change In this emblematic case we looked primarily at capacity building around the specific tools/guidance “Designing for Behavior Change,” “Make Me a Change Agent,” “Care Groups,” and “Barrier Analysis.” We looked at a phase in which the SBC team was tightly focused on a handful of what it considered to be high leverage, high impact techniques. While this was happening, critiques existed in the wider system of this very approach. Data available to the evaluation team trend strongly toward the conclusion that this approach was strikingly powerful in its ability to spread voluntary practices, even though perhaps divergent from technical best practice. What seem to be the success factors at play? TOPS Summative Evaluation FinalReport, March 2019 27 Success Factors 1. The focus on a small subset of concrete practices in a larger, complicated field of intervention. This is universally reported by field staff. At the level of global technical staff in both FFP and IP HQs we find substantial critiques of TOPS’ SBC materials, and focus on the subset. If scale of uptake is important, the simplicity and focus worked. 2. Simplicity of guidance made non-specialists confident to experiment on their own. 3. Authentic passion for and belief in the value of the practices in saving lives, by TOPS promoters. The ties to high quality evaluative data demonstrating impacts helped. 4. Practices were promoted across multiple TOPS technical areas (ANRM, Nutrition) and guidancewas designed to be useful to many sectors. 5. Practices were not unique to TOPS, or to Food for Peace development programs. TOPS was able to leverage a much wider and larger set of thought and practice beyond IPs. Indeed, the Care Group method had been around for many years before TOPS began, promoted by CORE Group. Trials of Improved Practices (TIPs) – which has great resonances with TOPS’ SBC work – was first codified in the literature in 1997 and affinities abound with behavioral economics’ strategy of “nudging” which is drawn from Nobel Prize winner Dan Kahneman’s work. 6. Promoted practices were economical and able to be incorporated in early stages of CA start-up. Challenges Ahead 1. Timing and windows of opportunity are important, even for relatively low-cost, economical programming improvements. While there are good reasons to facilitate training and capacity building for programs and staff at any and all phases of implementation, the SBC case offers our strongest evidence that training during the earliest stages of a CA aids uptake, understanding, systematized and sustainable support to experiments in the field, and tight and clear connections to staff roles, job descriptions, and program M&E strategies. 2. While training of trainers (TOTs) is an obvious candidate for greater inclusion and emphasis in the future, SBC shows that a successful TOT is intensive, extensive, requires careful up-front planning, proactive links to back-home performance expectations and agreements, and then tracking, follow-up and support – coaching, mentoring, problem solving – from a distance. Permagardens/Ethiopia Permagardening gained a particularly strong foothold in Ethiopia. TOPS ANRM technical team conducted a training of trainers (TOT) for 28 participants in PY4, and 8 of those participants trained around 85 more people in the technique afterwards. The aim was: to cascade the training to other stakeholders, strengthening knowledge about the permagarden nationally and allowing the method to reach numerous smallholder farmers through multiple organizations and donors over time. By working with the Network, TOPS hopes adoption will occur more organically—i.e., driven by demand, leading to higher quality gardens—instead of program driven, where adoption targets are set and must be met in a short, fixed period of time.17 The technique was adopted in Ethiopia DFSAs and in other programs/organizations in that country. The government of Ethiopia was considering incorporating it into national strategies. The permagardens case demonstrates the power of TOPS connecting with existing infrastructures, programs, strategies and actors. Success Factors 1. The permagarden technique fit comfortably into the Ethiopia Government’s Productive Safety Net Program (PSNP) and international PVOs and local NGOs were central service deliverers within that program. The PSNP was not just any government program; it was of strategic political importance to Ethiopian leaders and its success in buffering shocks due to natural calamitieswas 17 The TOPS Program, “Cascading the Permagarden Method in Ethiopia,” May 2016, p. 2. TOPS Summative Evaluation FinalReport, March 2019 28 also strongly supported and financed by international donors. Permagardening also fit well into the small holdings of most Ethiopians benefiting from the PSNP, requiring very little land. 2. FAO had initiated a Home Gardening Network many years before the ANRM team’s work, that network had been incorporated into other programs and served as a host and disseminator of the technique. It was not TOPS staff – nor indeed FFP IP staff – that carried on the work post-TOT. Building the capacity of knowledgeable, skilled, and trusted local trainers was a force multiplier. 3. Technical support for the permagarden technique was aided greatly by the existence of Feed the Future’s Agriculture Knowledge, Learning, Documentation and Policy (AKDLP) Project which focused exclusively on Ethiopia vs. TOPS’ global mandate. 4. TOPS’ active learning approach using high quality adult learning methods was a breath of fresh air for a number of trainees and the practicality and feasibility of permagardens was made tangible, palpable, and credible within the TOT itself. 5. A web of personal contacts – predating TOPS itself – existed among a handful of stakeholders in Ethiopia leading to trust and mutual confidence, positioning the technique in the local context, and garnering support for permitting the TOT to proceed. Challenges Ahead 1. TOPS structural inability to engage in concerted follow-up in Ethiopia created an impression locally of disconnection, of parachute-in development, despite much hard work and conscious effort to reduce or prevent this on the part of the ANRM team. 2. Problems of obtaining approval needed within USAID/FFP for the training were odd, curious, and at times bewildering to other actors in the local environment. 3. TOPS was unable to actively engage in – collaborate with others on – rigorous impact evaluation of permagardening in the Ethiopia context. Given the lengthy history in that country of failed techniques regarding food security in vulnerable farming households this created an impression of over-promotion. 4. Collaborating with local actors requires a great deal of time, investment, trust building and discussion. When TOPS technical leads are stretched thin, despite best intentions, an element of instrumentalizing stakeholders inevitably arises. While for TOPS the local infrastructure can be a great opportunity for scale and sustainability, it can seem to those actors as if TOPS comes in with guns blazing, sprints to complete a training, then seems to essentially disappear. Resilience18 Resilience became USAID policy in 2013. Resilience concepts had in fact informed Food for Peace strategy as early as the late 2000s, and resilience metaphorically exploded inside the USAID institutional environment from about 2017 on. As new data about program impacts became available, the resilience staff infrastructure grew, and influential actors outside the USG – DFID, FAO, UNDP, the World Bank, and more – took the concept seriously. Furthermore, resilience capacity building also became intertwined with two other adaptive challenges within the FFP and TOPS performance arena: theory of change, and “Refine and Implement.” TOPS undertook a lengthy and consistent series of resilience trainings throughout much of TOPS’ life. While the majority were organized in Washington, D.C., resilience was also covered in KSMs. This series of trainings, orientations, and engagements contributed to changes in global policy and procedures within FFP and USAID more broadly, although determining precisely TOPS’ vs. other actors’ influence is not possible. 18 The TOPS Program became involved in the resilience conversation with FFP, the USAID Bureau of Food Security, and the International Food Policy Research Institute (IFPRI) in 2011. From the third year of the program onward, TOPS was an active participant in the global development of resilience as an operating concept in food assistance program design, implementation, and measurement. The overall resilience effort of TOPS extends beyond the strict confines of this evaluation question about training transfer. In addition, important resilience work has been undertaken through LWA awards; those awards were not part of this evaluation’s scope, however. TOPS Summative Evaluation FinalReport, March 2019 29 Success Factors The resilience “positive deviance” case is the story of how TOPS helped shape global strategic direction and priorities in FFP through training, education, information sharing, and advocacy. Five lessons emerge: 1. The resilience work within TOPS was thickly networked with expert and influential actors outside the Food for Peace development programming partners. TANGO – and even more specifically the President of TANGO – had long been connected to academics, think tanks, donors (multi-lateral, bilateral) and practitioners trying to move resilience (and what might be labeled pre-resilience) ideas, practices, and theory forward before TOPS. TANGO’s President actively worked those networks, brought in outside actors (outside the FFP IP world, that is) to produce foundational tools/publications, and so avoided what could have been perceived as narrow thinking. 2. TOPS provided a nexus in which the best, emerging thinking about resilience (all of which was coming from outside of FFP Implementing Partners) could be worked over, pushed and pulled, prodded, and shaped for the DFSA world. 3. The resilience agenda within the institutional USAID environment was informed by hard, empirical performance data from field programs, and key decision-makers in the organization knew and trusted the gatherers and purveyors of those data (TANGO, its evaluation partners). As TANGO presented and ran workshops within USAID, it mirrored those events – through TOPS – to the implementing partner community. 4. The structure of TOPS – specifically its Leader with Associate opportunities – facilitated resource investment in one very specific aspect of resilience: its measurement. The integration of TANGO as a consortium member meant that knowledge management/sharing around what was being learned would be prioritized, and done professionally. It also allowed implementing partners to directly shape how resilience would play out in Food for Peace’s strategy. 5. The resilience theory arrived at a time when conjunctural events – most specifically the 2011 drought – were making it clear to senior leadership in USAID’s institutional environment that something decidedly new was needed to produce sustainable development, and that pastpractices of spending large on disaster responses were ineffective and inefficient. Challenges ahead If the resilience case shows how a symbiosis of TOPS, FFP, and IPs can work to build consensus at a strategic level, it also shows systemic challenges in rolling out new practices to the field in a consistent and effective manner. 1. One organization cannot do global capacity building. Nor can two (Mercy Corps, and TANGO, for example). Local organizations exist around the developing world at this point that have quite long histories in capacity building with regard to resilience-type programming and it would aid in their sustainability – and sustainability of capacity building writ larger – were they to be a stronger focus for training of trainers. 2. ” Training” is only one part of what’s needed for a complex strategy – or goal – such as resilience. A more accompanying, coaching and mentoring approach is needed. 3. Due to the importance of context in resilience programming, training/capacity building for field staff should eschew regional events. Theory of Change TOPS’ engagement in the roll out of FFP’s theory of change initiative began in 2013 when the very first version of the “TOPS Program’s Theory of Change Facilitators’ Guide” was published. It was significantly revised to align to USAID’s 2016 Technical References for FFP Development Food Assistance Programs, and revised again in 2017. The guide and accompanying toolkit (with PowerPoint slides, tools, and handouts) is the most downloaded item in the FSN library as of November 2018. It was referenced as a “go-to” curriculum by FFP beginning in 2017. TOPS’ primary training engagement – the large majority of effort – focused on PVO HQ and FFP/DC. Between October 2014 and January 2018, 19 distinct TOC trainings were organized: 13 of these were concentrated between January 2017-18 and two of the 19 were conducted in the field (Dhaka March 2015, Kampala September 2016). TOPS Summative Evaluation FinalReport, March 2019 30 The data reveal unequivocally that these trainings were widely appreciated. End-of-course evaluations were nearly unanimous in saying that TOPS’ work had helped clarify TOC from both a technical (what it is, and why it is useful) and institutional (how TOCs fit within the larger DFSA process) perspective. The role, position, and use of TOCs in DFSAs became better defined and increasingly understood with TOPS involvement. The work of TANGO’s TOPS TOC specialist – brought on specifically to assist with TOC support in 2016 – was frequently mentioned by interviewees as a strength. The development of a TOC development/review checklist was also mentioned as useful by both FFP/HQ and NGO/HQ staff. FFP staff in both field and HQ who review theories of change see a difference in quality between 2014 and today, and attribute that to a number of factors including TOPS’ training efforts. Success Factors 1. The appointment of a dedicated TOPS technical lead for TOC training and technical assistance. TOC capacity building is complex and complicated and a devoted technical lead was a good move. 2. TOPS’ TOC written guidance, including the TOC manual (revised three times during TOPS) and TOC checklist. These materials allowed staff at multiple levels to understand key elements of using TOCs and how FFP was thinking about them. 3. Where TOPS training occurred in the field – Bangladesh and Uganda – field staff say they are clearer about the use of TOCs. 4. Visualization of causal pathways. The evaluation team encountered a number of critiques of the causal diagrams required. While too much complexity in such diagrams can be a problem, leaving one with the sense of looking at a bowl of spaghetti, requiring a crystallization of causal reasoning with some kind of graphic condensation is a useful tool. It forces a concentration of analysis, permits specific and focused questions about reasoning, and widens engagement to more visually-oriented learners, who may otherwise be excluded by monolithic reliance on lengthy narrative exposition. Challenges ahead 1. Expansion of TOC technical assistance and facilitation expertise. The process of developing a TOC is best done with external facilitation. Such an actor can make blind spots apparent, challenge hierarchies in the room that those within the organization cannot, ensure diverse voices are tapped, and synthesize views. Such a facilitator needs to be well-versed, however, in FFP requirements, non-negotiables, definitions, and the politics of TOCs within the agency19 and FFP needs enough of such human resources to serve the population from which it is requiring TOCs. There were many calls from IP interviewees for greater involvement of TOPS in the field in relation to TOCs. 2. Coordination among stakeholders. TOCs are reviewed by a large number of staff inside FFP, and there is a lack of coordination which results in feedback/advice that is overwhelming, sometimes contradictory, and leaves field staff with questions about what must be acted upon, what might be acted upon, what can be ignored, and whether others in the system understand their TOC in the same way. Better coordination is particularly important for the official “sign off” on a TOC and its connection with performance metrics (the IP’s M&E plan and framework). 3. When dealing with complex social changes – such as improved food and nutrition security – literally any TOC can be ripped apart from afar. Complex causal relationships sometimes are hard to communicate with words on paper, and often the writing may obscure the actual intention of the author. This can be exacerbated both by a desire to be succinct, or by a tendency towards verbosity. There are many layers of translation at work in this process: from the thoughts in an IP writer’s head, to what gets on paper, to the context of the FFP reviewers when they get to the document, to the assumptions and preconceived notions reviewers bring to the printed page, to how reviewer feedback is written, to the context of the IP writer when receiving the feedback. The process – even in the best of worlds, when the content is much more linear and simple – is fraught with opportunities of misunderstanding and half understanding. 4. Evaluation. Every DFSA’s mid-term and final evaluation should have a required component that looks at the utility, use, and quality of the program’s TOC. 19 By “politics” we mean the dance in any agency – any – of disciplinary power, hierarchical power, seniority, control of resources, and the informal organization. TOPS Summative Evaluation FinalReport, March 2019 31 Discussion These five positive deviant cases reveal success factors and challenges for consideration of future planners and decision-makers. The evaluation team hopes that the idea that there are theories of change for training transfer – just as there are theories of change for resilience, or food security – may help. While there were successes in transfer of training in TOPS, it is important that TOPS was not actually structured to ensure this outcome in the field, where implementation happens. And this was the most common critique of TOPS’ training that we heard from TOPS and IP staff alike in our interviews: You know…we weren’t able to fully interact and go in depth with the programs…. We would go in and do a training or do a workshop and then we would have to leave and not be able to fully follow up with them because we had other responsibilities and if we went back to that country then we would kind of be criticized for spending too much time in one place. We weren’t able to see it through to make sure it was done fully right. (TOPS Technical Lead) Another TOPS staff pointed to the project’s M&E plan, and noted that while all metrics on training were hit, “we didn’t do anything with the system around the training, whether it could be influential.” One IP HQ senior staff member offered a quote that was echoed in many other interviews with such respondents, noting that, “holding these HQ workshops and handing us tools and saying ‘you go train your staff’ is not enough.” Planning strategically with a theory of change related to transfer of training would be of use in the future. Conceptually, unpacking different types of training challenges could also help. The nature of TOPS’ training challenges was not monolithic. Analytically, TOPS faced four different types of challenge, created by the intersection of two variables: 1. Variable One: Regulatory (required) vs. Voluntary Regulatory practices are ones which implementing partners (IPs) had no choice but to learn and implement. Voluntary practice are practices that IPs could adopt if it made sense to do so in a particular context. While the lines between this can be blurry – and changing – that a line exists is recognized by IP field staff, one of whom commented, “There’s a tension in TOPS – not bad, just a tension – between regulatory things…and optional things.” 2. Variable Two: Technical (known good practice) vs. Adaptive A technical challenge is one for which a problem is clear and an expert can provide the right answer. An adaptive challenge is one for which problem definition and solutions require new learning, and good answers tend to emerge from many directions and types of actors. The intersection of these two variables produces four capacity building challenge, sketched in the 2x2 matrix in Figure 13 below. Figure 13: Typology of Capacity Building Challenges Type 1 Challenges: In some respect Type I challenges are the simplest. We know what the training is meant to accomplish; we know what good performance looks like after training. And those participating in the training are TOPS Summative Evaluation FinalReport, March 2019 32 motivated due to knowledge that the contents of the training are requirements in the future. Success is contingent upon training quality (including adequate human and financial resources, the right level of conceptualization, and trainer quality), appropriate timing, the right participants, and oversight of implementation post-training. Penalties for failing to deploy new practices need to be clear and commensurate with the nature of the breech. Commodity Management is mostly a Type 1 challenge. Type 2 Challenges: Type 2 challenges share the same challenges of training quality, timing, and correct participants. Unique to Type 2 challenges are: implementation structures flexible enough to meet the idiosyncratic timing and contexts of programs; negotiating access between trainers/DFSAs and artfully managing competing demands; continuity of expert assistance post training. Type 2 challenges, contrary to Type I, are demand rather than supply driven. SBC and permagardens are mostly Type 2 challenges. Type 3 Challenges: Type 3 challenges contain all of the Type 2 challenges. The nature of post-training support is not expert-driven, however. Rather, voluntary communities of practitioners learn together, building each other’s skills and knowledge as pilot efforts and learning experiments take place. Crucial here is clear messaging to IPs about the fact that the practices are importantly voluntary -- perhaps strongly encouraged, perhaps not by a CBO or other FFP official -- but at the end of the day optional. Sustained harvesting and promotion of success and failure stories is important if there is an implicit or explicit desire that such practices may crystallize into something more “technical” in nature. Resilience is mostly a Type 3 challenge. Type 4 Challenges: Type 4 challenges are the most difficult. They feel – as the old saw says – as if we are flying the plane while learning how to build it. Type 4 training/capacity building challenges are complex, particularly in loose knit networks such as the FFP IP community. They are a tightrope walk between emergent learning and set procedures, between requirements and improvisation, between “how to” and “how not to.” Top-down edicts created by distant experts fail due to decontextualization and distance while bottom-up emergence alone fails due to competing priorities, unclear incentive structures, and principle-agent slippages. They require courageous and risk￾taking leadership, accountability for both process and product, keen attention to pacing, over-communication, and capitalizing on low-hanging fruit to secure momentum, energy, and buy in. False starts, failures, and hiccups need careful management; small successes publicity. Champions of change need cultivation, support, and promotion. The larger bureaucracy around the Type 4 experiment needs to create a safe space for stop-and-start learning, the boundary around the space needs careful management, and influential decision makers or stakeholders need to be in the same book (though they may be on different chapters and only gradually get on the same page). TOC is mostly a Type 4 challenge. Type 1 and 2 challenges are handled fairly well within formal bureaucratic structures, rule- and compliance-based contractual arrangements, accountability norms, and control-oriented management cultures. Type 3 and 4 challenges are not. Rollout of these latter types of initiatives can be difficult, frustrating, and enervating. Collaboration, mutual learning, embracing mistakes and readiness to alter plans and strategies are all key, but this can be hard in certain performance environments. Type 3 and Type 4 capacity building challenges also put managers and leaders in a vulnerable position: by definition they are not sure of the right answer – nobody has the right answer – and the pathway forward is one of experimentation and rigorous learning from those experiments. Finally, in Type 3 or Type 4 challenges there are no individual heroes or villains: success is systemic. No single person or agency can create it. 5. How effective were the stakeholder consultations? TOPS conducted scores of meetings, events, workshops, briefings, and dialogues during its life. “Stakeholder consultations,” however, were a strategic subset within all of this convening activity: [S]takeholder consultations between FFP and members of the food security community…. provided implementing partner agencies and others with opportunities to gain clarity from FFP on specific topics affecting their work, and enabled FFP to consult with implementing agencies on compliance issues related to policies, regulations and guidelines. Before each consultation, TOPS staff interviewed both sets of stakeholders to collect detailed information about the consultation TOPS Summative Evaluation FinalReport, March 2019 33 topic and participant needs. They then created a consultation plan that promoted a solutions￾oriented dialogue between the two parties.20 Annex 3 contains a list of convenings that fall under the formal rubric of stakeholder consultations. What constituted a stakeholder consultation – versus, say, a briefing, listening session, training, KSM, or opportunity for input – was not necessarily clear to most participants, nor to most TOPS or FFP staff. IP partners, for example, did not necessarily understand that perhaps they were participating in a “single topic knowledge sharing” (of which there were 45 during the life of TOPS) event in DC, a briefing, an orientation, a webinar, or a “stakeholder consultation.” Interviewees, as a result, confounded various kinds of convenings when responding to the evaluation team’s questions about this aspect of TOPS. The framing of our question to interviewees accurately represented the spirit and strategy underpinning formal stakeholder consultations. We prefaced the question(s) with the statement that TOPS hoped that its intermediary role between FFP and IPs, its power of convening, and its active facilitation of dialogue would lead to a) better understanding among IPs of FFP policies, practices, and initiatives, and b) influence of IPs on FFP policies, practices, and initiatives. How effective was TOPS, we asked, at accomplishing this? With regard to (a), even though respondents were referring to a variety of types of convenings/consultations, the evidence all points to the formal stakeholder consultations as a great success, one of the most important added values of the TOPS program writ large. “Their convening power is amazing…They have done a fabulous job,” said one IP field staff, underscoring that TOPS’ ability to get decision-makers and implementers into the same room to talk about policies, practices, strategies, and procedures was unique in his 20+ years working within FFP development programs. “It has enabled the implementer and the donor to come together and consult one another and to be…on the same playing field, that we are all in [this] together,” said another IP field staff. The opinion that TOPS’ convening power mattered, and influenced FFP in positive ways from the perspective of IPs, tended to be more strongly felt and stated by IP HQ senior staff. “TOPS served as a forum for us to raise concerns and have consultations with FFP” outside of formal consultative structures “that don’t resolve anything,” said one such respondent. “TOPS has allowed us to get down to brass tacks…TOPS has a very strong “in” with FFP and is able to get people’s attention when individual PVOs are not.” Said another: TOPS…. provides that bridge between USAID and implementing partners. It decodes some of these complicated Food for Peace implementation guidelines. TOPS comes in to break that ambiguity…. There was important divergence of opinion among IP staff (HQ and Field) and FFP staff (HQ and field), however, about whether TOPS’ convening/facilitation role made a tangible and palpable difference inside of FFP with regard to policy, procedure, or practice. Yet the influence is concrete, factual, and empirical. Mentioned explicitly by FFP respondents in positions to know first-hand about such influence were the following changes in their practices:  M&E policy and procedures in a variety of ways, including indicators for food security programming  RFA guidance in many ways  resourcing for gender analysis  the advent of the Refine and Implement approach  social accountability as a concept and set of actions in DFSAs  elimination of a 5-page concept paper in RFA responses  rolling out of the resilience agenda and strategy  PREP guidance 20 The TOPS Program. 2018. The Technical and Operational Performance Support (TOPS) Program Final Report: August, 2010 to January, 2018. Washington, DC: The TOPS Program, p. 33. TOPS Summative Evaluation FinalReport, March 2019 34 The formal stakeholder consultations were appreciated – without exception – by FFP staff we interviewed and who knowingly took part in them. These respondents praised TOPS’ role in forging agendas, doing the hard work of qualitative inquiry and synthesis in advance of the consultations, and facilitating the half day sessions. IP staff who participated – most if not all would have been senior staff based state-side – were less sure that the consultations resulted in anything tangible. An important piece of the puzzle about stakeholder consultations and IP influence on FFP has little to do with TOPS. When asked about TOPS’ role in facilitating greater dialogue and respectful engagement between IPs and FFP, a FFP HQ interviewee noted, “It’s so much bigger than that [i.e., TOPS alone]. A lot of old-timers left [FFP]…. there has been a huge turnover, and generational shift in FFP.” The informant noted that “FFP’s resources have mushroomed…they have more staff…. they simply can devote more time to IPs…. Having more people within the office of FFP allows for more attention to a lot of things. 6. To what extent did the products of Small Grants Program-funded projects resultin food security IPs changing practices or methods? The Small Grants Program (SGP) distributed about $4 million to 27 organizations via 71 grants, and ensured that grantees engaged in dissemination of grant products and lessons via public presentations and documentation on the FSN Network web site. The program – which was stuttering along at the time of the TOPS mid-term evaluation – became much more effective in the 2014-18 period. While at the time of the mid-term this component of TOPS was likely the weakest, by the end of the program in January 2018 it had met or surpassed all performance targets. TOPS hired a consultant to do a deep dive into the effectiveness of the small grants program in 2016, who concluded: PVO partners, some dissatisfied at mid-term, now generally find the SGP to be a valuable learning mechanism, though some feel gaps remain. A very high percentage of Small Grant recipients surveyed reported that they completely or mostly met their Small Grant objectives. Applicants generally felt that the RFAs that TOPS issued were appropriate. TOPS and FFP improved the speed of application review since the mid-term.21 This turnaround is a good example of TOPS’ ability to learn and adapt, and turn the program into something of palpable value to the IP community, particularly small and mid-size NGOs: I have a built-in bias for the small micro-grants program. Especially being a mid-sized org that doesn’t have ability to do big events, big KS efforts on its own. But when we could leverage our own resources with the resources from the small grants, it allowed us to do things that we never would’ve been able to do. In terms of creating tools or guides and certainly in terms of positioning. Relationship-building with potential partners and donors. We really appreciated that aspect of TOPS. (IP HQ Sr. Staff) Small grants were of particularly strategic value when combined with technical task force priorities, enabling both the production of useful tools that otherwise would not have seen the light of day while also making it possible for smaller NGOs to participate and even lead processes of tool development. The December 2016 review of the small grants program revealed that 27 small grant tools and products on the FSN web site at that time had been downloaded approximately 3,000 times in total, an average of 111 times per tool or product. The top ten most downloaded small grant produced tools at that time – roughly 14 months before the end of TOPS – were:22 Small Grant Number of downloads 1. IYCF-E Toolkit: Rapid Start-up Resources for Emergency Nutrition Personnel 158 for introduction document 157 for summary sheet document 21 Patrick Connors, “Technical and Operational Performance Support Program: Small Grants Program Review,” December 23, 2016, p. 1. 22 Ibid., pp 36-7. TOPS Summative Evaluation FinalReport, March 2019 35 2. Focus on Families and Culture: A guide for conducting a participatory assessment on maternal and child nutrition 257 3. Care Groups in Emergencies: Evidence on the Use of Care groups and Peer Support Groups in Emergency Settings 142 for evidence document 95 for recommendations document 4. Nurturing Connections – Adapted for Homestead Food Production and Nutrition 143 for HQ document English 83 for LQ document English 8 for French document 5. Warehouse Safety Guide 212 for English 8 for French 9 for Spanish 6. Reading and Responding to Your Baby Lesson Plan 129 for lesson plan 81 for volunteer flip chart 7. Enhancing Nutrition and Food Security during the First 1,000 Days Gender-sensitive Social and Behavior Change 120 for technical guide 78 for the brief 8. Dietary Diversity and Determinants Survey Guide and User’s Manual 152 9. Resource Guide for Building Effective Collaboration and Integrated Programming 151 10. Case Study on Adolescent Inclusion in the Care Group Approach – the Nigeria Experience 115 In that report, five cases of small grant tool production were looked at with modest depth:  IYCF-E Toolkit (Save the Children)  MARKit (Catholic Relief Services)  Learning for Gender Integration (Lutheran World Relief)  Vulnerability Assessment (MercyCorps)  Reading and Responding to Your Baby (Food for the Hungry) The report concluded: These five examples of TOPS Small grants again demonstrate that knowledge from TOPS Small Grants is generally shared using a variety of approaches. Nonetheless, this review was not able to document a large number of examples of knowledge generated by Small Grants being used by other organizations. Interviewees and online survey respondents readily cited their own organizations’ use of the Small Grants tools they produced themselves, or their use of other tools produced by TOPS, but less so their use of tools and products from other organizations’ Small Grants. Twenty-five percent of unsuccessful applicants and 24% of successful applicants responded in the online survey that SGs implemented by other orgs did “not at all” enhance the knowledge of their organization.23 The December 2016 small grants review had acknowledged methodological challenges; the percentages cited above should not be taken as statistically valid, and the relative inability to document uptake by organizations other than the grant-receiving entity could be an artifact of the review’s methods and resources. But related to the question of spread and uptake, at the time of the small grants report only one of the small grant funded tools were recommended by a TOPS Task Force, and three of seven task force leads at the time of the review said they were not interested in promoting tools produced in this manner. 23 Ibid, p. 19. TOPS Summative Evaluation FinalReport, March 2019 36 The best answer to this evaluation question (“To what extent did the products of Small Grants Program-funded projects result in food security IPs changing practices or methods?”) is almost certainly, “In some cases a lot, and some cases not at all.” The TOPS final operational report captures this dynamic succinctly: When different sets and components of TOPS activities were combined, the potential and opportunities for long-term sustainability increased…. tools corresponded to core competencies, then helped to identify some of the subject material for capacity strengthening workshops, most of which were designed with replicability in mind. Technical materials were featured at knowledge sharing events, which in turn fostered networking and created social capital among implementers with shared interests. Shared interests, in an atmosphere of peer-to-peer community, increased the likelihood that a successful approach or promising practice would be replicated, expanded and/or improved upon. (TOPS Final Report, p. 25) When the magic happened vis-à-vis spread and uptake, it was more a matter of serendipity, of good fortune. There’s another analogy…you know those dandelions where you blow…you know, there’s a nice dandelion but you don’t know where those seeds are going and you’re not doing anything to plant the seeds. But I enjoyed it, a lot of good ideas, we were able to share ours… I can’t even pinpoint though, an area…what did we take from that that we applied differently, I wouldn’t be able to answer that. (IP HQ Sr. Staff) The SGP was designed to be an inclusive space, one responsive to ideas from the IP community. There was a strong underlying ethos of service to that community. TOPS prioritized this approach, while simultaneously seeking opportunities to align SGP with other components of the project, as one TOPS staff member told us: We tried to be more conscientious about having the small grants be aligned with TOPS/FFP priorities…. FFP did a really good job of having people review the applications, even the little ones, somebody actually reviewed them, made suggestions. I do feel that was a real valuable aspect of TOPS. But I think we could have been more strategic about describing the kinds of applications we wanted to receive. One TOPS technical lead commented that while the project had resulted in much more sharing, collaboration, trust, and learning among IPs, the fact of competition did not disappear: Everybody was very keen about collaborating about sharing about identifying lessons learned but if it came to saying, like, you’ve discovered this answer and its working we still would want to change it and adapt it and put our name on it and so that we’re not giving CRS and we’re not giving Save the Children and we’re not giving Mercy Corps credit and we’re not implementing their approach in our program… 7. How effective was the management of The TOPS Program? The TOPS program hit or surpassed nearly all of its performance metrics while taking on a very large amount of unplanned work and responsibility that supported FFP in myriad ways. As a senior FFP staffer noted, “TOPS was always under budget and ahead of schedule.” The professionalism and performance of TOPS was noted by many interviewees. One senior IP staffer articulated this well: I feel like it was a well-planned award. It accomplished what it set out to do which was to provide technical and operational support. Tremendous growth in learning and KS, the program was well￾managed here at the DC side. I really appreciated it. (IP HQ Sr. Staff) The items for which it was formally accountable -- KSMs, technical training, production of tools, creation of a wide community of practice meant to serve FFP development program IPs, distribution of small grants -- all received TOPS Summative Evaluation FinalReport, March 2019 37 very positive stakeholder reviews both throughout the life of program24 and during the 82 interviews conducted for this summative evaluation. The TOPS team engaged in continuous learning in admirable ways, consistently improving its performance in many aspects year after year. Relationships between TOPS and FFP leadership were a big part of the overall success of the program, with each supporting the other in formal and informal ways. In looking for pattern and clusters in our interview data, the evaluation team believes three aspects of TOPS’ management effectiveness merit highlighting: delegation and trust, adaptive ethos, and inclusiveness. Delegation and Trust TOPS Director deployed a delegatory and empowering approach to management, particularly of the technical leads. He “created an environment where you felt the freedom to pursue adaptive initiatives” commented one technical staff member. “I had a lot of freedom to run my technical sector as I saw fit. There was not micro-managing,” said another, adding “Because I had that freedom it allowed TOPS to do what it did best which is to identify key areas of focus within the technical sectors.” The creation of a team culture – overcoming the silos of individual consortium members and creating a community that was something truly apart from them – was a conscious priority for TOPS’ COP, and it was achieved. One TOPS staff noted: TOPS [was]…a very collegial environment, whereby people from organizations that might be competing from time to time could sit down and talk about what they’re doing, what’s working or not working and have those exchanges. Pretty remarkable. Another added: If you walked into the office you couldn’t tell who was Save, or Mercy Corps, or whatever. We were TOPS. There were dissenting voices about this, it should be mentioned. One TOPS staff acknowledged the strength of TOPS’ “very calm diplomatic way” of engaging with stakeholders across the system but argued that the balance tipped too far in that direction, and TOPS was not “pushing the program to be everything that it could be.” A number of TOPS staff also rued the fact that technical teams worked largely in silos, and felt that more could have been done – without sacrificing the spirit of delegation and trust – by TOPS leadership to encourage teams to collaborate, or to more actively shape each team’s operating plans. While the evaluation team acknowledges these critiques, we believe that it was a smart and strategic choice to err on the side that TOPS did, given the nature and context of the program. Adaptive Ethos FFP staff who were instrumental in developing the original RFA for TOPS knew that they were creating a program that was novel in both product and process. TOPS needed to grow into itself, the original conceptualizers understood, to learn what its value could and should be, and it needed to grow and learn not in isolation but with the very stakeholders (and donor overseers) it was meant to service or satisfy. TOPS needed to be formalistically accountable for planned, agreed outputs, just as any grantee, while engaging in emergent dialogue and thinking that prioritized responsiveness, changes in priorities, restructuring of resources, and double-loop (and triple-loop) learning processes about purpose and mission. By most measures, TOPS’ adaptive management was astute, deliberate, and empirical. The delegatory style empowered technical leads to engage in emergent learning at their level, helping TOPS to stay close to what practitioners wanted and said they needed vs. top down declarations. Adaptations happened fast: in a matter of minutes a new form of support to FFP in DC, something not in the annual operating plan, might be agreed. When this worked well it opened up rapid forms of resource deployment that FFP and IP staff had not witnessed before: [The most important value of TOPS was] the opportunity to access resources to do things that are beyond the remit of…our programs. Through a quick conversation with [TOPS and FFP senior 24 Empirical data for this claim include end of training and end of KSM event evaluations; external consultant reviews of training and the small grants program; surveys of users of the FSN web site; interviews for this summative and the mid-term evaluation. TOPS Summative Evaluation FinalReport, March 2019 38 staff] suddenly we were able to do things in short order that met our needs at that time. That’s rare. (FFP field staff) One IP senior staff echoed this sentiment, applauding the flexibility of TOPS to provide technical assistance on theories of change. “Within FFP there wasn’t technical support mechanisms and TOPS kind of filled that gap,” she said. Adaptive management would not have been successful without supportive dynamics within FFP itself. One FFP HQ staff noted the importance of a gap when FFP had no formal strategy during the first few years of TOPS, arguing that this opened up space for experimentation. The enabling environment for generative learning, this informant argued, was also aided by the arrival of a new FFP Director in 2012. “[She] came from the PVOs and we were lucky enough to have her for 6 years and she had a passion for the TOPS approach,” the FFP staffer said. Adaptive management takes great trust, and can be particularly challenging between grantor and grantee given norms and conventions of accountability, fiduciary trust, and in this particular case use of taxpayer funds. Many of the respondents we spoke to who felt able to speak with some authority and accuracy said that the relationship between the TOPS Director and FFP AOR should not be ignored if we are seeking to understand TOPS adaptability. Said one: [The TOPS Director] had a very positive and productive relationship with our AOR. And others on the FFP team. And I think that was also key…He made it so collaborative and our work plan was often all over the place and we could switch things up and change activities and communicate with FFP in a very open way. Which I haven’t experienced before. Inclusiveness In order to be successful, TOPS needed to enact inclusiveness with regard to multiple stakeholders:  the five organizations within the consortium  FFP IPs in both HQs and the field  FFP staff in both DC and the field The evaluation team operationalized “inclusiveness” as ensuring the spread of program benefits to all stakeholders, valuing the work (tools, ideas, skills, experiences) of all IPs, fostering consensus and equitable processes, allowing stakeholders to influence program priorities and activities, and giving credit where credit is due. “Save has been running these FFP programs for a long time, they have a lot of experience,” said one TOPS staff. “I think they were able to bring that experience and those connections and the diplomacy of leadership to the program.” TOPS created the PAC to ensure outreach and input from the wider IP community and, for the most part, this mechanism was perceived by PAC members themselves as useful: It was great to meet all together with all major implementing partners around the table and kind of talk honestly and candidly about the needs of our organizations and programs across everybody. That was very…helpful to know when we were all in the same boat about the same thing and we we’re able to then leverage that sort of collective voice. It was also great that the donor was…there at the table as well so they were able to listen as we while talking candidly about challenges or successes in our food program implementation. Another PAC member noted that the messaging around the PAC from TOPS’ Director and FFP’s AOR was crisp and consistent: “we want you at the Forum, we need you at the forum.” The PAC was not without challenges, however as one frequent attendee said: We would give advice, and that was listened to, and they would ask us questions for an up or down response. They would ask ‘what have we done well, what have we done poorly’…To be honest with you, I wondered what are the utility of these meetings. We were consulted, but we were not involved in actual decisions. TOPS Summative Evaluation FinalReport, March 2019 39 The preponderance of the evidence suggests that PAC participants found it an important and useful half day once a quarter, and thought the forum strategically important (how could you do capacity building and knowledge development with regard to FFP development programs without CRS, CARE, and World Vision being importantly involved, one of our interviewees asked rhetorically). The comment above, however, points toward an important question: the fundamental purpose of the PAC and the degree to which it was understood and reinforced. The evaluation team’s understanding of the established, written PAC purpose was not as a decision-making body. That does not mean that this was the best choice, or the best choice for a future iteration of TOPS. At a general level, the large majority of IP interviewees outside of the consortium felt that TOPS did a good job at inclusion. Respondents articulated that they appreciated the number of non-consortium members who won small grants, who were asked to present and run sessions at KSMs, or lead webinars or short knowledge sharing sessions of other kinds. IPs consistently identified as a major achievement TOPS’ power to convene, to develop collective voice among IPs and so engage with FFP as a sector (and not just individual NGOs), to authentically build consensus across the entire community of IPs. Looking beyond the subjectivity of such opinions, it is clear that TOPS paid attention to the distribution of project benefits, keeping careful track of data regarding small grant awards, participation in trainings and KSMs, outreach to the IP community to contribute ideas, tools, lessons and to have a stage at major events where they could burnish their personal and organizational brands. Once again, a handful of dissenting voices merit mention. “TOPS did some good things, but it was very apart from USAID technical team,” said one USAID interviewee. “It was a cooperative agreement and so should have substantial involvement of USAID in that; I didn’t see substantial involvement.” There were several comments that TOPS’ emphasis on service to, and inclusive engagement with, the IP community risked creating an environment closed to outside ideas and voices, although the evaluation team gathered no concrete examples of this. Improvements TOPS culture of management resulted in a lot of positives. But there were also several concrete shortcomings revealed in the data. “TOPS was very adaptive and iterative…. did a great job…making HQ staff aware, and good participation of HQ staff,” said an official of one of the consortium member, “Maybe the one thing, it…could have been more field-based and field-facing.” A TOPS staff member supported this view: It’s bizarre, but we didn’t have the closest connection to FFP programs in the field…I think we probably should have had somebody who was tasked explicitly with developing relationships with DFAPs. TOPS did not have any accountability for or authority over those programs, so [we were] super-cautious about pushing ourselves on those programs. I think we should have been a little pushier…We talked about it…and we tried to be familiar with what was going on in the field. But we didn’t want to foist ourselves on others. FFP field staff, even ones who had participated in a number of TOPS KSMs and country-specific trainings, told us that they continue – even in 2018 – to be unsure about what TOPS exactly is, what they can ask of it, how to make an ask, or how TOPS compares to other service providers under contract with FFP. They also note that TOPS is not present when country or regional plans/priorities are made each year and that it was hard to keep track of all the different technical groups/leads, have multiple conversations going, and to understand the liminal role that TOPS was meant to serve between FFP/HQ and the IP community. A long-time FFP field staffer offered that this challenge – full and rich integration with the field – could not be laid uniquely on TOPS’ doorstep: I don’t think the field orientation for TOPS was very strong. There is responsibility to be shared with FFP as well. If you’re contracting something and you want your field staff to utilize it, then you should do some work to make sure they know it’s there. [There should have been] more of a deliberate engagement of the field by FFP about how to utilize [TOPS]. Need to balance the ledger so TOPS isn’t going out to hustle up business. Demand should be built up by FFP, for FFP. An internal review conducted in TOPS in 2017 underscored the fragility of TOPS’ contact with the field: Despite the participation of hundreds of DFAP staff in training and knowledge sharing meetings, in addition to the existence of an effective newsletter and other services, respondent feedback confirms that TOPS does not yet have consistent outreach to the DFAPs and other PVO key staff. TOPS Summative Evaluation FinalReport, March 2019 40 Several field staff, including senior officers, express a lack of awareness or any contact with TOPS, and PVO technical leads (gender, M&E, ANRM, and others) and often do not have contact with the task force or TOPS focal point for those areas.25 A potentially useful and constructive way of analyzing the pattern of strengths and areas for improvement in TOPS management is shown in Figure 14 below. The evaluation team believes that certain perceived areas of improvement – places where TOPS might have been managed differently – are a result of three strengths of TOPS management. Figure 14: Strength-Created Weaknesses Empowering delegation, putting the field first, and adaptability/flexibility were important strengths of TOPS. Their very nature, however, created challenges. Delegation enriched and motivated technical staff, yet also leaned towards less collaboration between them rather than more. Three tops technical leads told us that they never stopped struggling to get included in FFP field plans or conversations. Two FFP field staff said that it was not the best situation that they had to have separate conversations with different technical leads. The ethos of putting the field first, of prioritizing bottom-up processes, created roles/purposes for the PAC, task forces, and the small grants program that likely prevented more rapid and global dissemination of innovations than a different ethos would have. The dance of asks between FFP and TOPS, in which TOPS was requested – not ordered – to take on new roles, responsibilities, activities, events, products and services, was in some ways destined to send mixed signals to COPS and FFP field staff. When combined with turnover within FFP itself it is surprising that there were not much bigger problems of coordination among actors in the field. 8. How sustainable are TOPS initiatives? A most important step towards answering this question lies in unpacking the notion of sustainability. In the context of TOPS, sustainability can be thought of in any of the following ways: 1. All products and services of TOPS should be carried on without any FFP funds. 2. Certain aspects of TOPS should carry on without need for external funding, while others not. 3. Certain outcomes of TOPS – accomplishments – can replicate themselves in the future without needing a TOPS infrastructure per se. 25 Hedley, Darren and Edith Mutalya. Technical and Operational Support (TOPS) Program Internal Review Report. Washington, DC, 2016, p. x. Flexibility/Responsiveness To new work requested by FFP Program Management Strength Strength-Created Weakness TOPS Summative Evaluation FinalReport, March 2019 41 4. FFP makes TOPS a core part of its operating and business model, and funds it (here, the sustainability challenge is between Congress, State, USAID, and FFP). Four axioms aid in analysis: 1. Not a single one of TOPS’ 2011-2018 products and services can be carried out without human and financial resources. There is no viable strategy in which they somehow get “folded into” the standard operating procedures of IPs as a standard cost of doing business. 2. Funding trends and strategies for PVOs continue to narrow the size of unrestricted funds available, meaning that there is less and less organizational slack and more and more staff tied very tightly to contractual deliverables. 3. TOPS’ most important value add involves creation and management of a public commons where public and common goods are exchanged. While it is possible for individuals to come together on their own to manage a commons – there is not always a tragedy of the commons in other words – the commons in question here is not one of basic survival for IPs. This reduces the urgency and salience. 4. Fee for service business models around knowledge sharing, building collective voice, and facilitating dialogue between a major donor and its grantees – in which grantees fund the initiative through fees to an established third party organization – are not promising. Given these axioms, it is difficult to imagine a future of TOPS in which any kind of maximalist strategy (all or most TOPS products and services on offer) and minimalist funding approach (no FFP or other donor funds) is viable. A quite radically scaled back “TOPS” – one that peels away products or services that are deemed inessential – is a distinct possibility, and would need to be combined with a mission-driven commitment from FFP to continue supporting it as part of its core business. Perpetuation of certain outcomes of TOPS without the TOPS infrastructure is also a distinct possibility. If FFP wishes the Care Group manual to be used, and be updated, it need only include this in specific RFAs, and stipulate in its guidance how the update needs to be done (à la TOPS, via participatory processes that engage other IPs, for example). This also carries interesting promise in the arena of knowledge sharing itself: were high quality KS to be required (and evaluated) in every DFSA, and were IPs to see that this mattered in terms of winning an award, some piece of TOPS’ KS outcomes could be sustained. Some substantial perpetuation of TOPS’ identified and documented good practices could happen through connecting them proactively to mid-term and summative evaluation of DFSAs in the immediate, mid-, and long-term future. With this approach FFP would move more strongly into making these practices standards and benchmarks and encourage – structurally – individual IPs to innovate on them. FFP would still need to devote personnel, however, to harvesting, synthesizing, and proactively promoting lessons. There are a number of areas of TOPS’ work that are unsustainable except in terms of a commitment to funding from FFP and additional infrastructure that is apart from IPs and FFP itself: the small grants program, technical training meant to raise all boats (and not just one organization), and maintenance of the FSN website and tools database likely fall into this category. V. Conclusion Contributions The TOPS program created a new, productive, and generative dialogue between donor and grantees. A new culture was, in essence, created. The new culture had palpable and tangible value and results. It led to changes in FFP policy, practice, procedure, protocols and resource flows. It led, too, to better grasp by PVO partners of priorities, rationales, and strategies. TOPS leadership and KS and learning personnel worked consciously and assiduously to create and hold this new container. For the first time – according to a number of interviewees – PVOs forged collective voice rather than negotiate individually with FFP. This amounts to a change in relations of power, and this points to courageous leadership within FFP, one rooted in good business sense: The ability of TOPS to convene PVOs is, for FFP, a true efficiency which led to better, more effective policies, practices, and approaches. The value arose from not one but a variety of TOPS activities, including stakeholder consultations, the use of a PAC, and other convenings. TOPS was created in the wake of failure of previous capacity-building efforts (ICB grants). The TOPS program radically increased knowledge and experience sharing between PVOs. To do this TOPS overcame territoriality, TOPS Summative Evaluation FinalReport, March 2019 42 intellectual property concerns, structural isolation of individual PVOs, and perverse systemic incentives and altered the culture of learning among implementing partners. This required establishment of an authentic and separate TOPS consortium identity. It required creation of active learning and knowledge sharing spaces and events that were seen as value-for-money by PVOs and FFP staff. It required an operation that was well run with motivated and committed consortium personnel who were champions of TOPS’ mission and purpose. It required, too, conscious planning and continuous monitoring of non-consortium participation and contribution in mechanisms like the regional KSMs, task forces, and the small grants program. Last but by no means least, it required highly professional, high quality planning, organization, facilitation, and documentation of knowledge sharing processes. At the same time, success required FFP to send consistent messages to their grantees about sharing learning and experiences through and with TOPS, embed new knowledge products into RFA guidance, and ensure FFP staff participation in TOPS activities. Regional KSMs – there were 10 during the life of TOPS – were highly lauded venues for experience and knowledge sharing. An important outcome of KSMs was the building of social capital between members of the FFP development programming community. More important was the role KSMs played to both remind about and reignite the common forge of commitment towards food and nutrition security that is stakeholders’ raison d’être. The small grants program provided flexible resources for practitioner-driven, practitioner-friendly program innovation and experimentation. Important to understand is that it benefited from TOPS’ rigorous approach to knowledge creation, documentation, and sharing, a massive change from the old Institutional Capacity Building (ICB) grants. Part of the “secret sauce” is that TOPS created a culture of mutual accountability among PVOs. Small grant awardees took seriously the act of documentation and sharing outcomes not to please the donor but, rather, out of a sense of professional obligation to other members of the PVO community. TOPS also actively demonstrated to grantees that their sharing mattered, helping organize webinars, prepare materials for storing on the FSN web site, and highlighting outcomes in the FSN Newsletter. All of these latter activities are rather hidden behind TOPS’ formal performance metrics but are crucial to understand – and replicate – in the future. Small grants allowed small￾to-mid sized PVOs to both experiment and document/share learnings in ways they otherwise could not. Learning processes scaffolded around small grants allowed practitioners to communicate with other practitioners in language that was accessible, actionable, and practical. While training transfer – changing organizational practices and having those practice make a positive impact on food security – was likely uneven, some training did succeed at both, and at scale. Evidence for this comes from interviewees placed advantageously to have seen before and after cases. Important to grasp, however, is that this occurred in competencies tied to mandated, required practices, practices that if not followed – and not implemented up to known standards – would obviously affect PVO performance, performance review, and possibly future grants. This contrasted sharply with a wide variety of TOPS-sponsored trainings, trainings that focused on what one might call optional practices or areas – such as gender equity or SBC – in which effective work is highly contextual and in need of continuous adaptation. Where training and capacity changed IP practices at scale and for the better, TOPS’ contribution to success was in a) highly professional design of trainings, b) care taken in participant selection, c) use of practitioners as trainers, d) existence of useful manuals designed specifically for DFSA work, and e) iterative engagement with field workers in training and tool design. FFP staff in both field and HQ were also clear in these areas, clear about accountability, decision-making, and messaging. Challenges Integration of TOPS within the global FFP structure and culture was a big challenge, one that produced conflicting messages, identity problems, and slotted TOPS into a liminal space. On the one hand, TOPS was a cooperative agreement, with the norms and standards associated. On the other hand, TOPS became an adaptive collaborator with FFP DC on new and emerging priorities and activities. On the one hand, TOPS interacted with the field hierarchy of FFP as a PVO – with all of the norms and conventions associated – while having a role meant to raise all boats among the IP armada, which did not align well with a standard PVO/FFP relationship. This integration challenge made TOPS’ role in relationship to FFP adaptive challenges – from field staff perspectives -- ambiguous. The learning process around the use of a theory of change in DFSAs is emblematic in this regard. FFP wished to learn together with IPs how best to use theories of change in its development programs. TOPS Summative Evaluation FinalReport, March 2019 43 FFP M&E staff knew authentic co-learning was the only way forward. TOPS proactively engaged early and often, primarily at the global level, to help IP and FFP staff in this rollout. Direct communications between FFP and IPs about TOCs – review and feedback of drafts, project start up trainings in the field – did not formally include TOPS, however. Different actors communicated different things about the quality/use of TOCs to IP field staff – TOPS being uninvolved -- leaving many confused and frustrated. TOPS was involved in only part of FFP’s overall TOC review system. TOPS was not set up to engage in the kind of monitoring and evaluation needed to give reliable and valid answer regarding the adoption/implementation of new practices, or of their implementation quality. Making TOPS contractually responsible for the behaviors of IPs would have backfired in myriad ways, in any case. However, it would be eminently feasible to ensure that such questions are embedded in mid-term and final evaluations of programs, and for a program like TOPS to be charged with meta-evaluative synthesis. TOPS deployed a strong set of values and axioms regarding field-focus, building agendas up from the bottom rather than cascading global strategies down, and responding to articulated needs of IPs (and FFP) instead of strongly shaping felt needs. This was neither wrong nor right but the focus made it harder to drive new practices through the entire community. There is a very interesting passage in TOPS’ final operational report, one which resonates quite strongly with the evaluation team: When different sets and components of TOPS activities were combined, the potential and opportunities for long-term sustainability increased…. tools corresponded to core competencies, then helped to identify some of the subject material for capacity strengthening workshops, most of which were designed with replicability in mind. Technical materials were featured at knowledge sharing events, which in turn fostered networking and created social capital among implementers with shared interests. Shared interests, in an atmosphere of peer-to-peer community, increased the likelihood that a successful approach or promising practice would be replicated, expanded and/or improved upon.26 This insight is accurate and valid; it also reveals a very difficult combination of factors to have come together within a strategy of responsiveness and bottom-up priorities. Ironically, despite the enacted strategy of bottom-up priorities and responsiveness to field practitioners’ articulated needs it was impossible for TOPS staff situated in Washington DC to engage with field staff at a level of ongoing support, coaching, and advising needed to help with transfer of training and knowledge to a program. Force multipliers were needed. 26 The TOPS Program. 2018. The Technical and Operational Performance Support (TOPS) Program Final Report: August, 2010 to January, 2018. Washington, DC: The TOPS Program, p. 25. TOPS Summative Evaluation FinalReport, March 2019 44 Reference List Baker, Sarah Elsie and Rosalind Edwards. “How many qualitative interviews is enough? Expert voices and early career reflections on sampling and cases in qualitative research.” National Centre for Research Methods Review Paper, Southampton, UK: NCRM, 2012. Connors, Patrick. “Technical and Operational Support Program Small Grants Review.” Washington, DC: TOPS, 2016. Change Theorists Wiki. “Perspectives on Change: Ronald A. Heifetz.” Change Theorists. http://changetheorists.pbworks.com/w/page/15475038/Ron%20Heifetz. (Accessed November 2018). Glenzer, Kent. “Mid-Term Evaluation of the Technical and Operational Support (TOPS) Program.” Washington, DC: TOPS, 2013. Hedley, Darren and Edith Mutalya. “Technical and Operational Support (TOPS) Program Internal Review Report.” Washington, DC: TOPS, 2016 Heifetz, Ronald A., and Marty Linsky. "A survival guide for leaders." Harvard Business Review 80, no. 6 (2002): 65-74. Lombardo, Michael M., and Robert W. Eichinger. The Career Architect Development Planner. Minneapolis: Lominger, 1996. Pascale, Richard T., Jerry Sternin and Monique Sternin, Positive Deviance: How Unlikely Innovators Solve the World’s Toughest Problems, Boston, MA: Harvard Business Press, 2010. The TOPS Program. “Cascading the Permagarden Method in Ethiopia.” Washington, DC: TOPS, May 2016. . “Technical and Operational Support Program (TOPS) Annual Progress Report Year 5.” Washington, DC: TOPS, 2016. . “Technical and Operational Support Program (TOPS) Annual Progress Report Year 6.” Washington, DC: TOPS, 2017. . “Technical and Operational Support Program (TOPS) Annual Progress Report Year 7.” Washington, DC: TOPS, 2018. . “Technical and Operational Support Program (TOPS) Year 1 Annual Progress Report.” Washington, DC: TOPS, 2011. . “Technical and Operational Support Program (TOPS) Year 2 Annual Progress Report.” Washington, DC: TOPS, 2013. . “Technical and Operational Support Program (TOPS) Year 3 Annual Progress Report. Washington, DC: TOPS, 2014. . “Technical and Operational Support Program (TOPS) Year 4 Annual Progress Report.” Washington, DC: TOPS, 2014. . “Technical and Operational Support Program (TOPS) Final Report August 2010 to January 2018.” Washington, DC: TOPS, 2018. Wenger-Traynor, Etienne, and Beverly Wenger-Traynor. “Introduction to Communities of Practice.” Wenger￾Traynor.com. http://wenger-trayner.com/introduction-to-communities-of-practice/ (Accessed November, 2018). Woodson, Lisa, Laurie Starr, and Nancy Mock. “TOPS Training Review.” Washington, DC: TOPS, 2015. TOPS Summative Evaluation FinalReport, March 2019 45 Annex 1 Evaluation Scope of Work I. Activity to be Evaluated Activity: Final Evaluation of The TOPS Program, Cooperative Agreement No. AID-OAA-A-10-00006 The assignment under this Scope of Work (SOW) is to assist USAID’s Office of Food for Peace (FFP), Save the Children (SC) and The TOPS Bridge program in conducting the final evaluation of the Technical and Operational Performance Support (TOPS) program. The primary focus of the evaluation will be to assess TOPS’ achievements in the four Program Results pillars – Knowledge Capture; Knowledge Generation; Knowledge Application; Knowledge Sharing – and their long￾term viability, and to highlight strategic lessons learned during the program. The evaluation recommendations will be available to inform the design and implementation of similar activities or initiatives. The anticipated period of performance for the assignment is September 17 – December 31, 2018. II. Background The TOPS program aimed to identify, synthesize, adapt, and share the highest quality information and tools, in order to build capacity and establish best practices among USAID’s development food assistance program partners. The TOPS Program was a seven-year Leader with Associates (LWA) Cooperative Agreement (August 13, 2010 – January 31, 2018) with a global mandate to enhance opportunities for sharing knowledge and best practices among practitioners in the multifaceted food security community. Upon completion of TOPS, SC was awarded a one-year associate award, TOPS Bridge, to continue some of the key functions of The TOPS Program in preparation for transition of the five-year follow-on program, IDEAL. In implementing TOPS, SC led a consortium of experienced food security organizations that brought specialized knowledge to program. The five principal consortium partners were CORE Group (knowledge management), Food for the Hungry (social and behavioral change), Mercy Corps (agriculture and natural resource management), and TANGO International (monitoring and evaluation). SC managed and directed TOPS, and supported nutrition and food technology, commodity management, and gender technical areas. TOPS established a Program Advisory Committee (PAC) composed of representatives of implementing organizations and other stakeholders. The PAC was a consultative group of major practitioners in food security and nutrition programming that guides and supports TOPS in considering strategic and tactical issues for program implementation. Since its inception in 2010, TOPS has focused on building a network of food security practitioners around the world to identify, analyze, improve, and share information, tools, and promising practices to improve program performance. The program used three fundamental approaches to address the food security community’s needs:  A directed program of capacity-strengthening activities for USAID’s development and emergency food assistance partners, led by TOPS technical staff;  An inclusive community of practice, the Food Security and Nutrition (FSN) Network, including its technical task forces, interest groups, and web portal (www.fsnnetwork.org), with open membership to all USAID grantees and other food security stakeholders;  A small grants program promoting the creation, use, and adaptation of capacity-building tools and encouraging evaluation and documentation to build the evidence base as part of TOPS’ strategies for the TOPS Summative Evaluation Final Report March 2019 46 dissemination of best practices to food security and nutrition stakeholders (grantees, local NGOs and other partners, as appropriate), with a focus on field-led and collaborative initiatives. It is important to note that, throughout the program, TOPS responded readily to frequent FFP requests to address topics and provide services that were not anticipated in the original work plan (e.g., resilience, commodity management, gender, USAID Forward, stakeholder consultations, Theory of Change training). TOPS undertook these activities because they were in line with its strategic objective and clearly a felt need of the community of practice. TOPS knowledge capture and sharing strategies were intended to be flexible, adaptive and responsive to changing donor needs and innovations in the field to enable TOPS to respond quickly and effectively to new requests from USAID, FFP, and the PVO implementing community. The TOPS Strategic Objective was highest quality information, knowledge, and best practices for improved methodologies in Title II food aid commodity program performance identified, established, shared, and adapted. The theory of change for the TOPS program followed a knowledge management results chain, which outlined the four program results that were hypothesized contributors to the program strategic objective. PR1: Knowledge capture: Knowledge and skill needs of audiences identified. PR2: Knowledge generation: Reliable, high-quality information synthesized and produced in user-friendly, appropriate formats. PR3: Knowledge application: Effective and appropriate traditional and non-traditional skill delivery approaches and systems / applications used; and PR4: Knowledge sharing: Information, skill and knowledge exchange supported and expanded. The schematic diagram of the results framework is presented below. TOPS STRATEGIC OBJECTIVE Highest-quality information, knowledge, and best practices for improved methodologies in Title II Improved Improved Practices Improved Systems Program Result 2: KNOWLEDGE GENERATION IPR 2.1 User-friendly tools identified,modifiedordeveloped, and endorsedbytheFSNNetwork. IPR2.2 Toolsadapted(created to fill gaps or modified) for use in appropriate formats. IPR 2.3 Packagedinformation on promising practices shared with food security community for discussion and continuous improvement. Program Result 1: Program Result 3: KNOWLEDGE APPLICATION IPR3.1 Capacity of food security practitionersstrengthenedthrough focused skilltransfer. Program Result 4: KNOWLEDGE SHARING IPR 4.1 Physical & virtual community exchange forums and knowledge sharing mechanisms supported and expanded. IPR 4.2 Local and regional knowledgesharingandnetworking strengthened. KNOWLEDGE CAPTURE IPR 1.1 Skill levels of food security practitioners assessed through participatory and external approaches. IPR 1.2 Relevant and current universe of appropriate tools identified. Food Security and Nutrition Network Contributing and Benefiting TOPS Summative Evaluation Final Report March 2019 47 The TOPS Program specialized in seven key technical and crosscutting areas:  agriculture and natural resource management  commodity management  gender  knowledge management  monitoring and evaluation  nutrition and food technology, and  social and behavioral change Relevant Performance Information Sources The following documents will be available for the selected consultant(s)’ review and reference:  Annual progress program reports (Program Years 1 through 7)  Mid-Term Evaluation report (September 2013)  Mid-Term Review of the Small Grants Program (September 2013)  Final Review of the Small Grants Program (December 2016)  Satisfaction data from TOPS workshops and trainings (Years 1 through 7) III. Purpose of the evaluation This evaluation serves the dual and mutually reinforcing objectives of accountability and learning: • Accountability – The evaluation will assess and report on TOPS’ achievements in the four Program Results pillars (Knowledge Capture, Knowledge Generation, Knowledge Application, and Knowledge Sharing) and the long-term viability of these achievements. • Learning – The evaluation will determine the reasons why certain results occurred or not to draw lessons, derive good practices, and identify recommendations for sustained application of achievements, practices, and lessons learned. IV. Overall Evaluation Approach The summative evaluation will root itself in good practice related to both utilization-focused evaluation27 and developmental evaluation,28 while incorporating principles and approaches related to participatory monitoring and evaluation.29 Scope The period covered by this evaluation is August 13, 2010 to January 31, 2018. Evaluation Questions Using the bolded key research questions below, evaluate TOPS’ achievements in the four Program Results pillars – Knowledge Capture; Knowledge Generation; Knowledge Application; Knowledge Sharing – include recommendations for long-term viability of these achievements among implementing partners, donor, and the overall community of practice in food security programming, and for incorporation into similar programs and initiatives in the future. 27 Michael Quinn Patton, Utilization-Focused Evaluation, Thousand Oaks, CA, London, New Delhi, and Singapore: Sage Publications, 2008. 28 Michael Quinn Patton, Developmental Evaluation, New York: The Guilford Press, 2011. 29 Irene Guijt and John Gaventa, “Participatory Monitoring and Evaluation: Learning From Change,” IDS Policy Briefing 12, Brighton, UK: IDS, 1998. TOPS Summative Evaluation Final Report March 2019 48 The (non-inclusive) list of sub-questions under each key question are illustrative. The Evaluation Team may revise, modify, or add to the list to further probe the participant learning experience, translation from learning to practice, and to highlight achievements in these key topics. Knowledge Sharing Meetings, Regional and Domestic To what extent did Knowledge Sharing Meetings contribute to Knowledge Capture, Knowledge Generation, Knowledge Application, and Knowledge Sharing?  To what extent did the knowledge shared at KSMs transfer to the workplace? Explore key themes of all KSMs and provide examples.  To what extent did TOPS KSMs result in the formation of new knowledge sharing relationships after the events?  To what extent did peer-to-peer knowledge sharing occur during unstructured times that went along with the KSMs (breaks, lunches, social reception)?  What opportunities exist to improve the effectiveness of KSMs within future programming efforts? (Including but not limited to: effectiveness of knowledge transfer to the workplace, formation of new knowledge sharing relationships, peer-to-peer knowledge sharing, etc.) Communities of Practice/ Task Forces What were the strengths and weaknesses of the topic-specific, sectoral task forces (e.g., Ag/NRM), in relation to knowledge capture, generation, application and sharing? Food Security and Nutrition Network (FSN Network) What were the most valuable components or uses of the FSN Network and why?  How effective was the FSN Network as a resource for information sharing and the dissemination of technical knowledge?  Which resources accessed through the FSN Network were most used/most valued? Which practices learned through the Network were most used/valued?  How effective was the FSN Network as a resource for building consensus on promising practices?  To what extent did the FSN Network contribute to shaping food security agendas and influencing donor priorities?  What opportunities exist to strengthen the FSN Network in future programming efforts? Capacity Strengthening Trainings To what extent did TOPS trainings translate to changed practices for food security practitioners? Please provide examples. Specifically, but not limited to:  Permagardens  NRM/water trainings  Commodity Management – FACM workshops, Fumigation and Pest Management training.  Theory of Change (ToC) development and use  Social and Behavioral Change and Communication (SBCC) – Care Groups, Make Me a Change Agent, Designing for Behavior Change.  Nutrition – TIPS for TOPS (Trials of Improved Practices method), joint nutrition/SBC training in Malawi  Monitoring and evaluation: annual monitoring workshops; qualitative research workshops, technical and guidance documents,  Resilience measurement  Gender integration in program design  What factors contributed to uptake of promoted practices? What, if any, barriers to uptake of promoted practices exist? Stakeholder Consultations How effective were the stakeholder consultations?  To what extent did consultations result in enhanced communication between PVOs and FFP?  To what extent did consultations result in modifications to FFPguidance? TOPS Summative Evaluation Final Report March 2019 49  To what extent did stakeholder consultations result in improved understanding of key issues for FFP? For implementing partners? Small Grants Program To what extent did the products of Small Grants Program-funded projects result in food security IPs changing practices or methods?  How widely was the Small Grant Program known; how well was it understood; how vigorous was uptake by the implementing community?  To what extent are food security IPs aware of small grant products produced under the TOPS Small Grants Program? Which Small Grants Program products were most/least useful and why?  What factors contributed to uptake of practices or methods promoted in Small Grants products? What, if any, barriers exist(ed) to uptake of practices or methods promoted in small grants products? General and Consortium Management How effective was the management of The TOPS Program?  How effective was internal TOPS management?  How effective was Save the Children’s management and oversight of the Cooperative Agreement?  To what extent did each of the TOPS Consortium members contribute to the performance of the team?  What activities did TOPS undertake outside the original scope of the RFA and project proposal (specifically, initiatives that responded to evolving food and nutrition security priorities of FFP and implementing partners)? What factors allowed TOPS to absorb and respond to emergent priorities? How effectively were the emerging priorities implemented? Sustainability How sustainable are TOPS initiatives?  What might be changed (practices, methods) to enhance or ensure the long-term viability of TOPS-initiated processes? Methodology A desk review of secondary literature and primary qualitative research comprise the key elements of this final evaluation. Analysis carried out during the desk review will directly inform the methods and research tools used to gather primary qualitative data. The evaluation team will propose the most appropriate menu of methods for this evaluation and share with TOPS leadership prior to data collection. Proposed methods should ensure the capture of input from the diverse group of TOPS stakeholders who have been involved over the life of the program. V. Evaluation Team Composition and Qualifications An external team led by an experienced team leader will carry out the final evaluation. The team leader will have extensive experience in program evaluation. Collectively the proposed team will have experience in capacity strengthening, knowledge management and networking, management development, and organizational learning. The following criteria will influence the selection of the team leader and team members:  Strong evaluation skills with food security programs (at least MS degree in related skills and 10 years of experience in program evaluation)  Strong communication/Interpersonal skills  Extensive familiarity with FFP-funded activities  Experience in conducting final evaluations for USAID projects  Experience working with Private and Voluntary Organizations (PVO)s or other international organizations  Strong organizational and reporting skills, attention to detail, and ability to meet deadlines  Proficient in use of Microsoft Office software (Word, Excel, PowerPoint)  Proficient in the design and analysis of simple online surveys using open-source, free software appropriate for low bandwidths. TOPS Summative Evaluation Final Report March 2019 50 VI. Team Leader’s Scope of Work The team leader is ultimately responsible for all components of the evaluation, including selection and management of the evaluation team, design of the research instruments, interviews, data validation meetings, integration of other members’ analyses, and compilation of the final report. The team leader will:  Develop an overall participatory framework for the evaluation, including the indicators of program progress, both in terms of outputs and in terms of the program implementation process;  Provide a point of contact for the TOPS Program Director and the Evaluation Team. Conduct planning meetings with the TOPS program management team to coordinate staff involvement;  Establish a strategy for involving institutional partners (primary consortium members, USAID partners) plus other partners in the evaluation;  Share a detailed evaluation plan with TOPS management and make necessary revisions based on mutual consent. The evaluation plan must include, methods, tools and a qualitative sampling strategy for the range of stakeholders (e.g., USAID mission and HQ staff; PAC members; participants of trainings, workshops, stakeholder consultations, and, knowledge sharing meetings; task force members; small grants recipients) to be included in the evaluation;  Create a calendar for data collection for each member of the team, including lists of people to be interviewed, the type of interview, and the method of reporting on each interview, then work with the TOPS management team to provide necessary logisticalsupport.  Meet in situ with the team to review the interview data and the results of thefieldwork;  Create an outline for the evaluation report and coordinate the writing responsibilities of individual team members;  Organize a review workshop with TOPS members to validate the initial observations andinterpretation.  Compile and prepare a draft report for review by TOPS and FFP;  Provide up to two revisions based on comments and submit the final report. VII. Deliverables 1. Draft report not exceeding 40 pages, including executive summary, that addresses the questionsspecified under Section III. Evaluation Scope of Work. The report will present a clear and concise summary of its findings, conclusions and a prioritized list of recommendations. Due date: November 30, 2018 2. Final report not exceeding 40 pages that addresses the areas specified under Section III. Evaluation Scope of Work and that has USAID/FFP and TOPS comments incorporated. The report will present a clear and concise summary of its findings, conclusions and a prioritized list of recommendations. Due date: December 31, 2018. VIII. Expected level of effort and anticipated calendar of deliverables The following table outlines the associated estimate of the LOE required of the evaluation team. The team leader may choose how to allocate LOE among team members. TOPS Summative Evaluation Final Report March 2019 51 Action Evaluation team LOE Deliverable due Document review 8 days One-day evaluation planning workshop in DC with TOPS team (and possibly others TBD) 4 days No later than September 24, 2018 Tool development, including online survey 5 days Data collection 30 days Check-in meeting (1/2 day) with TOPS team to engage in preliminary data analysis and to modify the evaluation’s action plan, if needed. 2 days Within two weeks of start of data collection Data analysis 9 days Team meeting in DC to validate initial observations and interpretations. 2 days No later than November 15th Draft and Final report writing by lead evaluator, including edits based on comments received after draft review. 9 days Draft due November 30, 2018; Final due December 30, 2018 TOTAL 69 days IX. Funding and Logistical Support All funding and logistical support for the TOPS’ final evaluation will be provided through TOPS Bridge program. Activities that will be covered include payment of team members; support for all expenses related to the evaluation; logistical support and limited distribution of the draft and final reports. Specifically, TOPS Bridge program will: 1. Carry out the necessary preparatory actions for the final evaluation, including sharing all relevant documents and reports for desk review and providing stakeholder sample frames; 2. Organize the evaluation planning meeting with the chosen consultants and FFP; 3. Organize a meeting(s) with TOPS Bridge program staff and former TOPS consortium members; 4. Organize validation meeting of initial observations and interpretations in Washington, DC, no later than November 15, 2018; 5. Submit a draft of the evaluation report to FFP and consortium members for comments; 6. Organize a debriefing of the evaluation to FFP if appropriate; and 7. Submit the final report to FFP. X. Relationships and Responsibilities TOPS Bridge program will provide all administrative support for the completion of the SoW. TOPS Summative Evaluation Final Report March 2019 52 Annex 2 Emblematic “Positive Deviant” Learning Case Data Tables Commodity Management Data Table In the table below, the evaluation team aims to give a feel for the data related to each precondition. Our aim is to select data that reveal patterns and ranges of opinion. We hope, too, that this level of granularity might assist future plans and strategies. Element of Theory of Change Evidence related to the precondition Training/tool/practice is easy to understand and readily adaptable by the target learners/adopters Warehouse guide “a deeply collaborative effort” (quote from Task Force participant)  Large number of unprompted positive comments about commodity management training/capacity building from interviewees in FFP and IPs.  Large comparative investment in tool design work (hiring of an instructional design expert) Time is allotted in trainings / KS events to conversations about contextual challenges and adaptation “The [CM] training was very well done. It was a living workshop with interaction” (training participant)  Numerous comments about practicality of training. Reasons for changes in practice are clear  The regulatory nature of CM training abundantly clear. But also: IPs themselves saw weaknesses (warehouse safety for example) and actively collaborated to crystallize practical guidance for the community. Trainees are the right ones for determining/deciding back-home adoption  Intentional process from the very top of TOPS to advocate for the right people in the room.  Several comments on the fact that regional trainings missed the mark slightly – front line CM staff less likely to attend in favor of their managers, or managers’ managers The new practices can be handled within existing budgets for the CA  Not a single shred of evidence that this wasn’t the case. FFP staff at different levels/sites reinforce practices consistently “In CM trainings…you always had FFP staff present, and IPs were very vocal about regarding challenges we were facing.” (IP field staff)  CM best practices strongly pushed by FFP  Strong recognition by FFP of the facts of IP business models – the loss of qualified staff is almost a genetic reality given how DFSAs are funded and the resource structure of most IPs – and therefore the value of the training. Technical staff available after training to provide ongoing support  Much evidence of TOPS’ CM team trying to meet post-event requests, but not able to meet demand. Big, well-resourced IPs stated they met thisinternally. Trainees can access additional advice after training  Easy to use manual, USB drive, posters, warehouse safety guide TOPS Summative Evaluation Final Report March 2019 53 “that toolkit has been in the hands of all of our chief of parties, commodity managers and managers of our food programs, even the non DFSA programs…. Every time we travel to the field we make sure that we have…. copies on hand…. I’ve trained my program managers here at the US level with…that guide and tool kit. So that’s been a huge resource which has really helped us to improve our commodity management as a whole…. And then also the warehouse management tools and the big posters that they had printed…we would take to our warehouse managers [so they would be] reminded of safety standards and procedures.” (IP HQ senior staff) TOPS Summative Evaluation Final Report March 2019 54 Social and Behavioral Change Data Table In the table below, the evaluation team aims to give a feel for the data related to each precondition. Our aim is to select data that reveal patterns and ranges of opinion. We hope, too, that this level of granularity might assist future plans and strategies. Element of Theory of Change Evidence related to the precondition Training/tool/practice is easy to understand and adapt by the target learners/adopters  Substantial evidence that Mercy Corps, Peace Corps, Helen Keller, World Vision, PCI picked this up.  High number of downloads  Many comments on the ease of use of the tools “We’ve trained now all of our people in Make Me a Change Agent…. [W]e learned to do a simplified model for our promoters…. We used our coordinator, who is a communicator by background, he read the tools, he never had training in the tools, but he read the documentation – he developed a training. Using that, we trained our staff. We have trained a lot of people. We did all this without any additional help.” (IP field staff) “I was in Haiti talking to a woman with…USAID funded project using Care Group. I talked to the doctor who [led a new USAID funded project using Care Groups]. Did she go to a Care Group training? No, she just downloaded the manual from the TOPS website. She launched a nation-wide food project working with no more technical input than that manual. So that’s fantastic. She’s a sharp person, but…the fact that she was able to simply take that manual and incorporate it into the program is amazing.” (TOPS technical staff) “For SBC specifically, SBC can be a nebulous practice area. What TOPS has very nicely done is to provide a good clear set of tools and approaches, that almost codify SBC practice. The people working with TOPS have really nicely pooled together SBC theory, these more theoretical things, and put them into practical actions. There was an awful lot of academic literature before TOPS….my impression is that previous resources were more theoretical. So, in those 3-4 key resources, TOPS pulls all this together and lays out a clear approach…. Their materials are very action-oriented. Theyare…directed towards action. I think that was missing from the field of SBC.” (IP staff) Time is allotted to challenges of adaptation during the training KS “In response to what was most valuable in the [SBC REALIZE] workshop, others said: “Hands-on experience using tools and talking about how we could use or adapt [them] for our own work.” …. In answer to what was most valuable, others said, “I most valued interaction with experienced DBC practitioners and the time to discuss and ask questions,” and, “Opportunity to refine my understanding and knowledge of Barrier Analysis and learn from other advanced practitioners’ experience.”” (TOPS Year 7 Annual Report) “The TOPS Nutrition and Food Technology Specialist developed this new methodology, designed to increase the overall quantity of food consumed by infants and young children via increased frequency of feeding, quantity of food per feeding, and use of responsive feeding techniques. Drawing on some of the essential elements of the Trials of Improved Practices (TIPS) approach, TOPS sought input from mothers in Malawi and Madagascar on small, doable actions that that were used to design a behavior change activity that could be implemented by field staff with little external technical assistance. The TOPS Nutrition team then facilitated an eight-day training-of-trainers (ToT) workshop, inviting staff from all FFP implementing partner organizations in Malawi and Madagascar, with the pre-negotiated agreement that they would replicate the trainings for other project staff and partners. The TOPS workshops included classroom learning, role-plays, and three half-day field experiences during which trainees practiced each step of the new TOPS Summative Evaluation Final Report March 2019 55 methodology. Equipped with all training materials used in the workshop, participants provided cascade training to field staff and community volunteers. In Malawi, trainees replicated the training for other project staff from all implementing agencies of the FFP￾funded project, and these staff then implemented the behavior…. In Madagascar, trainees replicated the training-of-trainers workshop for the eight partners in that country, who then supported each other to replicate the training, following a similar cascade model as in Malawi. This several-stage rollout occurred with only minimal support, via email, from The TOPS Program.” (TOPS Final Report)  Substantial time spent in all SBC trainings on adaptation (TOPS technical lead) The learner has an explicit performance commitment with the manager back home  One instance confirmed More than one person/project attends training  Always the case in country-specific trainings; not so consistently in regional knowledge sharing, or DC-based events. (TOPS participant list record) Trainees are the right ones for determining/deciding back-home adoption “Despite the enthusiasm for Care Groups, not all respondents had utilized the knowledge acquired at the workshop to implement changes or increase their application of these skills. Two respondents, as mentioned previously, do not work with Care Groups because the demand in their own organizations is low. While one organization attending the workshops was not looking to expand its capacity for Care Groups, the other organization was interested in using the Care Group model in the future.” (TOPS Training Review 2015)  Important emphasis on trainee type for nutrition-related SBC training Individual trainees have the resources to implement a newly acquired skill/tool/ “Most respondents were limited to sharing information informally. Of those respondents citing having internal support for Care Groups, only two of those had a budget available to conduct formal trainings. Two other respondents, with support from their organizations but with no budget, stated that although Care Groups are recognized as an effective tool in reducing child mortality and changing behaviors, they are only used when the context permits. Programs that respondents are currently advising may not be using the Care Group approach; therefore, organizations may not emphasize the need for more formal transfer of knowledge or skills. Three respondents were limited to sharing information by email or phone, often referring field staff to the manual or the caregroup.org website for further guidance. However, these materials were considered by one respondent as “very sophisticated” especially for those with language barriers. Respondents, overall, recognized the benefits of having in-person trainings because “this would have helped [staff] understand the concepts.” Although TOPS provides micro grants to workshop participants, this requires participants to allocate work time to apply, which may not be budgeted for within their organization.” (TOPS Training Review 2015)  15 IPs adopted/adapted Care Groups early in TOPS, eventually 30 organizations did with help from TOPS’ tools (TOPS staff)  Connection to approved CA budget not systematically approached. The skill Is relevant  While much evidence exists of the universality of SBC techniques to multiple sectors, and the spread of SBC in the last decade indicates relevance, the tight connection to job performance expectations, incentives, and rewards cannot be found in our data set. to the individuals’ job and performance review The skill is relevant to the DFAP project aims, including budget  SBC techniques are applicable in any program/sector “The majority of all participants from both training sessions state they have implemented some element of the Care Group approach since attending the training. One NGO was on TOPS Summative Evaluation Final Report March 2019 56 the verge of redesigning the health and nutrition aspects of its DFAP at the time of the training. The project enrolled and grouped beneficiaries based on the Care Group approach. Another project started several months after the July workshop. The decision to implement Care Groups had already been determined prior to attending the workshop, thus the majority of the workshop content directly informed how the project carried out its work.” (TOPS Training Review 2015)  Important number of comments from informants that budget an issue, and that it was hard to adopt/adapt once a DFSA was beyond the refine and implement stage. “What is needed to make Care Groups successful? What does an IP who wants to do it well do? 1) good training up front so that everybody understands the model. I’ve seen too many places where there’s too much self-teaching. Making sure everyone understands the model is key. 2) Staffing accordingly, you do need staff, you need to be intentional about #s of staff, if an org wants to reach 6000 people but they only have staff for 3000, it will fall apart. 3) Making sure that staff are well versed in facilitation techniques and abilities, so helping staff grow in those areas. Success of Care Groups depends on facilitators, those teaching the materials to volunteers.” (IP HQ staff) FFP encourages adoption, and/or signals that it would be welcome  FFP gave guidance on including SBC approaches/techniques in their RFAs. “that helped it spread” said a TOPS staffer. “Expectation from FFP was important. If you don’t get it, they’ll come down on you. There’s an indicator on it so there’s some “adoption”” (senior IP HQ staff)  At least one substantial disconnect between a DFSA’s implementation of TOPS-recommended SBC techniques and negative review of same from FFP technical people in DC Trainees can access additional advice after training  HKI story in Mali: Plugged into HKI regional resources, and got guidance from HQ lead who had participated in SBC trainings in DC. Able to articulate concrete adaptations needed in Mopti context.  Several IPs stated that they had internal expertise for follow-up, and didn’t need TOPS beyond the tools/manuals.  But strong overall feeling that the above was out of the ordinary, that the norm was a problem of ongoing advice (face to face, voice to voice, email) Individual(s) receive post-training/event prompts check-ins regarding implementation “[Our HQ lead] was very important: first, she briefed us about Care Groups. She linked [us] to [another of our organization’s projects in the region] to get technical assistance. Next, she came for a TDY to speak with implementing agents, and identified gaps and corrected gaps…. When we have any issue we refer to her and she can provide advice.” (IP staff in field) “we talked a lot about the need to go beyond….it seems that we have this model of doing a training and leaving…we wanted to see more of ongoing support to these programs through some sort of coaching and mentoring process, just longer term. Not that the trainings were bad but…it just seemed to be the primary model was…have a workshop….” (TOPS SBC staff) “Proactivity. She reached out and checked in, in a nice way, not pushy, how else can I help you? Excitement and commitment on her side was infectious. She’s excited, our team has seen it, it kind of motivated our team. She was excited, and it rubbed off. It’s contact and ingenuity in her communications with us.” TOPS Summative Evaluation Final Report March 2019 57 Permagardens Ethiopia Data Table In the table below, the evaluation team aims to give a feel for the data related to each precondition. Our aim is to select data that reveal patterns and ranges of opinion. We hope, too, that this level of granularity might assist future plans and strategies. Element of Theory of Change Evidence related to the precondition Training/tool/practice is easy to understand and adapt by the target learners/adopters “It was really a tangible activity that a lot of organizations currently do, different home garden methodologies but a lot of them don’t have a key approach documented for it, and so what TOPS was able to do was document this not only from a technical stand point but also…exactly how it should be done and how you can mobilize a program that has 50000, 100000 beneficiaries…[I]t was…very tangible, very clear instructions” (TOPS staff) “permagardens, how to shape the land, store the water…that stuff is very useful. The two consultants who came here were great. This is really concrete resilience, this is practical…. It’s not something theoretical. The TOPS trainers are experienced people from the field, they are not teachers coming from the university.” (COP) “a lot of people, you try to sell it, it’s not that cool, not that amazing. For the broader world [high level technical people] it’s not a big sell, but [for] people on the ground, it’s easy and simple to deploy.” Easy to adopt within existing project plans; a clear and measurable impact on food and nutrition security. (TOPS staff) “I think one of the things people liked about TOPS was that it was a participatory, a very practical, very hands-on way of doing training…. I think that appealed to people, Food for peace, NGOs, and to some extent to government. It was a hands-on, roll up your sleeves, do some digging, etc. etc. It all helped the sense that this was something useful.” (Tufts staff) In the three months since after permagarden TOT in Ethiopia, 8 of the trainees cascaded training to around 85; all of the training logistics, facilitation and expenses are covered by ToT participants and organizations, demonstrating a real commitment to sharing and learning (Permagarden Case Study) Time is allotted to challenges of adaptation during the training KS  Training is hands-on, field-based (in context) The learner has an explicit performance commitment with the manager back home  No evidence More than one person/project attends training  3/21 orgs in Ethiopia TOT had more than one staff participate Trainees are the right ones for determining/deciding back-home adoption  The permagarden toolkit -- notably, the most downloaded resource from the FSN Network website in PY6 -- was the basis of a training of trainers event in Addis Ababa, co-hosted by the Ethiopia Home Garden Network (which was developed with TOPS funds during PY4) and Tufts University’s Agriculture Knowledge, Learning, Documentation and Policy Project. All 28 participants received copies of the toolkit, and eight went on to replicate the training with more than 85 trainees--again using the toolkit--with no further support from TOPS. (TOPS PY6 Report) TOPS Summative Evaluation Final Report March 2019 58 Individual trainees have the resources to implement a newly acquired skill/tool/  The permagarden toolkit--notably, the most downloaded resource from the FSN Network website in PY6--was the basis of a training of trainers event in Addis Ababa, co-hosted by the Ethiopia Home Garden Network (which was developed with TOPS funds during PY4) and Tufts University’s Agriculture Knowledge, Learning, Documentation and Policy Project. All 28 participants received copies of the toolkit, and eight went on to replicate the training with more than 85 trainees--again using the toolkit--with no further support from TOPS. (TOPS PY6 report) The skill is relevant to the individuals’ job and performance review  Permagarden technique fit well into Ethiopian government program/priorities, which IPs were serving (PSNP project) “the permagarden plugged into a lot of things: strong programming (PSNP), it operates on a small scale fitting in with very small plots (average plot is ½ hectare or less), and then an ongoing, chronic nutritional problem. There’s also an emphasis on social and water conservation, and permagardens align with that too.” (Tufts staff) The skill is relevant to the DFAP project aims, including budget “The gardens – year around gardening – was a huge boost to the program.” “It changed our strategy.” (COP)  People and organizations had a vested interest in permagardens with or without TOPS FFP encourages adoption, and/or signals that it would be welcome Not a major factor in Ethiopia Trainees can access additional advice after training  Very difficult for TOPS to provide such services according to several TOPS technical leads  TOT led to replication  Independent, pre-existing home garden network and other organizations/actors in Ethiopia to continue promoting technique  Existence of AKDLP Individual(s) receive post￾training/event prompts check-ins regarding implementation  Very difficult for TOPS to provide such service  TOPS technical groups discouraged from focusing too strongly on any one country/DFSA/Region (TOPS staff)  November 2018: Feed the Future sponsored meeting in Ethiopia on permagardens TOPS Summative Evaluation Final Report March 2019 59 Resilience Data Table In the table below, the evaluation team aims to give a feel for the data related to each precondition. Our aim is to select data that reveal patterns and ranges of opinion. We hope, too, that this level of granularity might assist future plans and strategies. Element of Theory of Change Evidence related to the precondition Training/tool/practice is easy to understand and adapt by the target learners/adopters Global Level  Very long and consistent calendaring of resilience briefings, sharing of emerging practice, in DC.  Coalescence of resilience as a priority within FFP, as well as LWA award for resilience, indicates that DC level trainings have been successful.  5 “guidance notes” produced by REAL.  Development of resilience indicators: FFP drafted and then brought them through TOPS to the IP community, with extensive meetings and commentary periods. Said one IP sr. HQ staff, “My experience over 26 years is that when these changes are made by USAID they are presented to you and you live with them.” TOPS’ role in generating a participatory dialogue that led to changes greatly appreciated. Transfer of training to Field  people are “picking and choosing” parts of the resilience approach when they go back home, and that’s counter to the theory. (TOPS technicalstaff) “Recently in respect to Zimbabwe, [IPs]…said that the overall resilience [approach] was too much, they’ve picked and chosen…. but the whole idea of the resilience was that it was a package. If you are not just training on a set of techniques, then getting adaptive programming knowledge to people is more difficult.” (TOPS technical staff)  Mixed feedback from COPs and FFP field staff interviewed for the evaluation. “We’ve set up our own resilience monitoring, every six months. TOPS helped run the resilience conversation very productively. I hear different opinions from different organizations in the TOPS world. When I attended the REAL conference, there were clearly different theoretical approaches to resilience, and they don’t agree, and that’s not TOPS’ job to forge consensus. Each actor will use resilience in different ways, appreciating the differences is important, but if USAID sends out an RFA, we apply, and If we don’t agree we shouldn’t apply. What can TOPS do? I think in terms of being a knowledge base, of synthesizing, let’s talk about this new thing before it becomes a policy or an RFA. That would need FFP to be up front with TOPS or IDEAL, so they are not playing that catch-up game…. (Chief of Party) Time is allotted to challenges of adaptation during the training  Unable to address this in an empirical sense. The learner has an explicit performance commitment with the manager back home  No evidence that this was done. TOPS Summative Evaluation Final Report March 2019 60 More than one person/project attends training Nepal Resilience Design: YES (TOPS participant list) Gender and Resilience: NO (TOPS participant list) Resilience 2017: handful of IPs have two participants (TOPS participant list) Resilience 2017 Zimbabwe: main audience was GOZ (TOPS participant list) RD/Zim (Mercy Corps): YES (TOPS participant list) “We were all together with managers and a few field staff…I went through the training with them and we developed the action plan together. That’s how we were able to implement. It carries a lot of weight for a field person sitting there and the chief of party is also sitting there.” (Chief of Party) Trainees are the right ones for determining/deciding back-home adoption Unable to really answer this in an empirical sense. Individual trainees have the resources to implement a newly acquired skill/tool/ Unable to really answer this in an empirical sense. The skill is relevant to the individuals’ job and performance review Re., training around Resilience Design in Smallholder Farming Systems: “This was at the end of TOPS…we…. [put the] cart before the horse, we were promoting something that people don’t need.” (TOPS staff)  Resilience measurement required in endlines and baselines. Resilience part of FFP and USAID strategy.  Resilience not, per se, a requirement in RFA responses, although in measurement modules The skill is relevant to the DFAP project aims, including budget “I hear different opinions from different organizations in the TOPS world. When I attended the REAL conference, there were clearly different theoretical approaches to resilience, and they don’t agree, and that’s not TOPS’ job to forge consensus. Each actor will use resilience in different ways, appreciating the differences is important….” (Chief of Party)  Resilience task force begun through initiative of IP staff, “but we didn’t have a lot of time because TOPS is winding down, and the TF has wound down, and we were put on hold to some extent.” (TOPS staff)  Revise & Implement period exists, connection between resilience outcomes/TOC pushed by FFP M&E, TANGO asked to analyze all baselines for resilience and analysis used to engage in dialogue with COPs. “If you are going to do training on resilience design, it must come at the front end not at the end or midway. Implementing some of the things they want people to do, it takes time. You can’t come midway into the project, because remember you have to do mobilization of the communities…and people adopt…and the time they are exiting, that’s the time momentum is built. That first year is the time for training of partner organizations.” (Chief of Party) Re., training around Resilience Design in Smallholder Farming Systems: “This was at the end of TOPS…we…. [put the] cart before the horse, we were promoting something that people [didn’t] need.” (TOPS staff) TOPS Summative Evaluation Final Report March 2019 61 FFP encourages adoption, and/or signals that it would be welcome  Resilience measurement modules for baseline and endline required.  REAL LWA. SCALE LWA. Both indicate FFP direction. “USAID seems confused about resilience.” (Chief of Party) “resilience strategy, when it was revised, there were a new set of indicators that FFP drafted and brought them through TOPS to the IP community in extensive meetings around this and commentary periods and really looking at the indicators and do they make sense, are they measurable….and went through a rather long and intense process before finalizing them. My experience over 26 years is that when these changes are made by USAID they are presented to you and you live with them.” This felt authentically true, authentically participatory. This was a very new approach.” (IP senior staff, HQ)  “resilience” mentioned 48 times in 2016-2025 strategy. It is mentioned four times in 2006-2010. “Ten years ago we used to deal with food INsecurity…and FFP decided we worked on food security. Now, it’s resilience….I don’t look at resilience as a new concept. It’s kind of like a Phd theory of things I’ve been doing since 1994. Yes, we want people to have money in the bank, yes we want people to withstand shocks…. you want a roof over your head, assets in the bank…. that’s what resilience is…. We were doing this 20 years ago. (FFP field staff) Trainees can access additional advice after training  TANGO had been pretty much alone in trying to do this. (FFP HQ, Save HQ staff)  Mange Trust brought in as a local resource in Zimbabwe. PRA Kenya as well. “After the training [Mercy Corp’s], we held a technical working group for livelihoods and economic development team since we are implementing this project as a consortium. Then following the working group, we prioritized some of the areas that we needed to pursue and we also had a field visit to Mwanda where some of the resilience design principles are being implemented. And following that meeting and the visit, we then set out some action plans that each district field teams would then go and implement. And we are monitoring progress as a program.” (Chief of Party) Individual(s) receive post￾training/event prompts check-ins regarding implementation  No evidence exists for this outside the requirement for baseline and endline measurement “Those learning summits and trainings were very useful. I have a program that benefited…on resilience designs. Immediately after that we implemented some of the things that we learned. That is most of the concepts… But the implementation part, that’s where I have a problem. It’s more of good will of the organization, let me put it “good will” for lack of a better word. I think TOPS can do better by…after training, they continuously…look at what happened after that training. Let me give an example. After the resilience design training, no one has come down to see what we have done or if we have implemented what they trained us to. I think the monitoring…is key to that. Training is one thing, and implementing is another. We don’t want to just be sitting in the hotel…we want to be sure that the partners that are funded by USAID…they also implement some of these things that are recommended” (Chief of Party) TOPS Summative Evaluation Final Report March 2019 62 Theory of Change Data Table In the table below, the evaluation team aims to give a feel for the data related to each precondition. Our aim is to select data that reveal patterns and ranges of opinion. We hope, too, that this level of granularity might assist future plans and strategies. Element of Theory of Change Evidence related to the precondition Training/tool/practice is easy to understand and readily adaptable by the target learners/adopters Global Level  Among interviewees who participated in TOC trainings in DC during the life of TOPS: nearly unanimous praise on their quality and help with clarifying use of TOCs within DFSAs.  “Participants at all 12 individualized TOC workshops claimed that the one￾day sessions using their own TOC diagrams were extremely effective. As one participant said, they “walked away with a much greater understanding of how to meet FFP TOC requirements.” This sentiment was echoed at the July TOPS PAC meeting, by all PAC members whose organizations had participated in a workshop. The four organizations that invited TOPS to oversee Annual Review Workshops in April and May subsequently executed or scheduled an in-country TOC review. Said one attendee, “You made TOC less scary!”” (TOPS Year 7 Annual Report)  Checklist for TOC review appreciated by FFP  TOC manual praised by several interviewees Transfer of training to field  There is considerable positive feedback from field staff about the utilityof ToC training in preparing proposals, and FANTA praised the training as helping to improve the quality of proposals based on feedback from FFP. (TOPS internal review 2017)  FANTA-led trainings vs. TOPS led trainings are perceived differently by IPs. TOPS’ only involvement in field was Dhaka and Kampala and all interviewees who participated in them say they were high quality, and helped make the TOC process more understandable “When I look at [the way our chief of party in uses TOC] …he has the TOC printed off on massive poster paper….and they track how they are doing on different pathways [green means ok, yellow means not on track] ….and they review this as a whole team every quarter, and they are using it as a tool for adaptive management…. that change is massive [from 2013].” (IP HQ staff, talking about progress on utility of TOCs since 2013) “Our last experience – mid-term – here in , was the most successful experience. Before this, it was really conceptual, really theoretical, really hard for people in the field to really use and understand. But my last experience here, at the mid-term, was the most successful. We hired/recruited one member of TOPS to design our annual review, we used FFP’s guidelines from August 2017, so it was the last guidelines developed, and our DFAP was the first cohort, I think, using the August 2017 guidelines. It was very useful. We did it in a very participatory way. In our DFAP we have consortium partners, they participated, and then we followed the guidelines from August 2017, in a very practical way. Our previous TOC was 25 pages of text. Now, our TOC, in our diagram, we have a number of very useful information, key external actors, their role in contributing to the different pathways. We have a clear TOPS Summative Evaluation Final Report March 2019 63 Element of Theory of Change Evidence related to the precondition assumption, clear linkages within the purpose of our DFAP. We examined very critically our project logic. We are mostly using the diagram, and we don’t have these 25 pages of text. Of course, it is still a process for people at the implementation level to really use it. So we promote it, we continue to talk about it, in doing our last ARR we definitely used the TOC for that, and it is more and more easy to understand. We use it for adaptive management.” (Chief of Party) “ “We are getting the same message [of things being overly complicated, hard to implement]. In reality, for many of the theories of change we are getting, it is very complicated, it is very, very difficult to read. Difficult to read by us here in FFP. But, for me, my understanding, previously we used a kind of theory of change from outside the system, we had IPs that were using TOC before we promoted it. I feel like I understand and can explain TOC…. but many of our partners are struggling with theory of change right now. I really think it needs to be simplified. But how to simplify is the challenge.” (FFP Regional Staff) “I can get that partners think it is a nuisance. A TOC is good for visual folks. You read these proposals, and it’s hard to understand. I’m on the spectrum of, ‘give me bullet points’, but partners throw everything into a proposal. A TOC forces them – in the skeleton framework – to be clear about what they are doing and why. And often times, it doesn’t match (the TOC and their actions/strategies), and maybe that means the partners don’t know what they are doing…. [M]aking people work through those causal relationships, and explain in a few words, is challenging and creates frustration in many people…. [O]ne theory of change I’ve reviewed in the last couple of years, and it’s not written in American English, and so it’s really hard to understand. It wasn’t proofread….and it’s extraordinarily frustrating.” (FFP HQ) “I would get two different explanations from people on the same team that created the TOC regarding what the TOC meant.” (FFP HQ staff) Time is allotted in trainings / KS events to conversations about contextual challenges and adaptation “One hundred percent of the 25 participants who evaluated the four-day Theory of Change workshop (September 2016, Kampala) ‘highly valued’ the event. All indicated that the workshop provided them the necessary guidance to take the theory of change process forward in their organizations, and all stated that they will recommend the workshop to colleagues. The most oft-cited takeaway (56 percent of post-workshop evaluations) was the importance of using evidence to guide the entire theory of change process, and of keeping records of supporting evidence. Other major takeaways were: investing in the development of a problem tree as a first step, and devoting adequate time to the theory of change process. Participants explained that while the initial time investment may be large, they see its payoff in making indicator selection, logframe development, communication with project staff about the project vision, and project implementation and monitoring, more efficient.” (Annual Report)  High marks on DC-based TOC trainings run by TOPS, with open-ended answers pointing towards useful time/conversation for contextualization. TOPS Summative Evaluation Final Report March 2019 64 Element of Theory of Change Evidence related to the precondition  DFSA kickoff workshops are specifically focused in the field, on field programs, focusing on program-specific TOCs.  SABAL CLA case competition submission narrative reveals the utility of focus on adaptation, and how careful and persistent working through of TOC’s intricacies and utilities transformed staff attitudes andbeliefs about TOC. Reasons for changes in practice are clear “I think the whole M&E piece was also quite successful at raising the importance of M&E, trying to make sure that people were all on board and understanding the TOC and what that meant. Now we just need to make sure that were going to the next level now that everybody more or less understands the TOC and the importance of the TOC [and get to] how can we actually measure better the TOC.” (IPstaff) [TOPS] realized that TOC was being “shoved down people’s throats, and nobody really knows how to use them, adapt them over time. Within FFP there wasn’t technical support mechanisms and TOPS kind of filled that gap.” (IP sr. staff) “The big question is how FFP trains the field [and] the field was a little left out [of TOC training design]. I know that HQ people [in IPs] said [the DC training] was really good, but it wasn’t the same for field people.” (Chief of Party) “Yeah it’s been great for headquarters, not great for the field or people who should be using it… [It is] difficult to keep track of what’s happening with TOC because it’s all over the place and that’s what makes it hard.” (Chief of Party) “[I feel] that the theory of change process has gotten very bad in some ways. The demands and the expectations of USAID have become more and more onerous over the years. You spend a lot of time working on your TOC, and then you still continue to spend another year, if not year and a half, on your theory of change. And it has a lot of implications. And sometimes it is just a lack of clarity from USAID about what they want out of a theory of change. It needs to be streamlined, there needs to be a cut off, we should not be changing it a year or 1.5 years into the program.” (IP HQ staff) “What I feel that USAID has done…their expecting TOCs in the way that they want them. I don’t think that should be their role. I think if you want people to come up with a TOC…it is so incredibly complicated when you look at food security on a TOC, I don’t know how anybody makes sense of it, when you look at it. I think it’s gone from being a useful exercise in a simple manner to less useful when it is more complicated. And the stakes are high. And USAID is never satisfied. They seem quick to judge that it’s wrong…. Have you seen the diagrams? They are mess. I think the exercise is useful, don’t get me wrong…. We still don’t know what is expected. I talk to USAID mission staff, and they don’t know what is expected. They are just as confused as everybody else.” (Chief of Party)  SABAL (Nepal) CLA case competition submission reveals understanding as a longer-term process Trainees are the right ones for determining/deciding back-home adoption  Start-up workshops and Refine and Implement focused entirely on DFSA staff in the field. TOPS Summative Evaluation Final Report March 2019 65 Element of Theory of Change Evidence related to the precondition  DC trainings (run by TOPS) targeted at HQ-level staff of IPs to promote strategic understanding. The new practices can be handled within existing budgets for the CA “The amount of documents, the huge requirements for M&E, and all the documents, everything that goes into that, the amount of work that has to be put into the work around M&E in DFSAs it is just humongous. Of course DME is important so that we can capture learnings etc. It would be wonderful to find a way that it can be limited so that people can actually focus on implementing programs.” (Chief of Party) One informant laid out that the TOC leads to the logframe, the logframe leads to M&E indicators, and FFP needs to approve, and then this translates into a MEAL process. “As I’m speaking, it’s two years and counting, “that the MEAL plan/TOC still isn’t formally approved. “For us, the TOC is really helpful at the broad level…. but the relationship to the MEAL plan [and concrete resources for that] is not really clear.” (Chief of Party)  Resources are distinctly available for Refine and Implement phase FFP staff at different levels/sites reinforce practices consistently  Advent of Refine and Implement a consistent support to TOC ideals  DFSA start up workshops led by FFP emphasize flexibility and learning to finalize a more context-relevant program strategy, logframe, and M&E plan. FFP M&E staff have been very consistent on this over time  Alignment between TOPS’ TOC manual/materials and FFP aims greatly achieved by end of TOPS “I can look at it [a TOC submitted by an IP] one way and say it’s specific enough; another person will [look at the same TOC and] say, “it’s not specific enough.” The respondent added that there are many people looking at the TOC on paper, from a distance, and reading narratives about the diagram that may or may not be written well, and trying to interpret and then provide feedback/suggestions/demands; this system may foster confusion and frustration (FFP HQ staff) “To do a good solid review of a TOC takes concentrated time, probably ½ day is needed to do it well. I can’t take an entire day to do it, I try to break it down into sections…The checklist is great, but there’s a lot of things to consider. You are looking at pathways, causal relationships, a crosscutting feature, and even if you are looking at one SO, you have to look at the other SOs, so it is complicated, you come up with a lot of comments….is it color coded correctly? You have some very finicky comments at that level, then larger macro comments on the quality and causal relationships.” (FFP HQ staff) “For this DFAP, we submitted a ToC, and then USAID helped us during the R&I period, they helped us refine. We got a USAID M&E specialist [based regionally]. There was a workshop here [in country]. In the end the theory of change is way too complicated…. We spent months on it, revising it, the theory of change is related to the logframe, and the logframe is related to the budget and in the end it was all messed up…It is only at the end of year two that we have a MEL plan.” (Chief of Party) TOPS Summative Evaluation Final Report March 2019 66 Element of Theory of Change Evidence related to the precondition “[A]t the start it [TOC role and use] wasn’t laid out quite clearly. It was required in proposals. In [my program], what went into the proposal and what we are implementing now are totally different. We ended up totally changing it after the proposal was agreed. You’d say they are two different projects. I would say that the changing directions from FFP about how to do [TOC] were a definite problem…. Personally…I feel the TOC is an important and mandatory program. It really helps pull out these deep technical aspects and says here’s what we know, here are our assumptions…. I wish it would have been introduced a little better. It made life impossible for just about everybody….” (Chief of Party) “The downside is trying to adjust TOC [after it is finalized], with FFP it is not soeasy to do. The difficulty is, coming up with that TOC, it is embedded in desk reviews, and in proposals, but then the theories of change that are developed are more like academic research than real experiences in the field. In the field, it will not be a challenge to re-think something like the [role and use of] VSLs, but the challenge comes when you want to use your experiences as a project/program and then influencing the formal change in your TOC. The delay comes in when having to provide academic evidence, it’s biased towards you having some academic support of why the TOC should change. It’s like an iron-clad thing, there’s never that quick bottoms-up adjustment. We are not quick to value the feedback from extension officers, the experts we’ve put on the ground…. I don’t know how we can bridge the gap.” (IP field staff) I think it was a bit painful, for everyone…. And FFP was clear, ‘let’s learn from this together, none of us are experts.’ They had a pretty clear vision of what they wanted to see, we got a lot of flexibility….in a CA, some of our hardest deadlines are the M&E plan…. FFP said, ‘we’ll cut you the slack to learn together.’ Looking back we did not have the right people in the room for the TOC training. It was such a high level…. My colleagues, we’ve gone through the Refine and Implement process, it’s become more systematized. Kudos to the FFP team to learn as we went along. I would say it was a learning process, we were guinea pigs. I know in other regions they did not have the same level of support from their regional FFP process, and it wasn’t a great experience.” (Chief of Party) Technical staff available after training to provide ongoing support “TOPS could do better in the area of TOC: instead of stopping at providing the overview of what is TOC and how do you “do” one [they] might have gone down the road of developing facilitators in each organization who would be able to carry out those trainings independently” (IP HQ)  Largest of IPs have internal expertise; smaller ones may or may not Trainees can access additional advice after training “[W]e need a training of facilitators and make sure that people who get exposed to TOC are technical people, not just MEAL people. We are using TOC as a tool for implementation and there is not a broad enough understanding.” (IP HQ) “Look, there’s only one Laurie Starr. We’re reaching the limit of having that capacity, to really push this to the next level. Laurie’s done a great job, she wrote the guidelines for it [but] there needs to be a deeper bench in the next iteration of TOPS to support TOC. And not have it fall on Laurie’s shoulders.” (Chief of Party) “[The FFP M&E team] has grown pretty quickly. I’m not sure how the vision and…the capacity to support the IPs…has been communicated...These are young [staff doing the TOC reviews], they are very busy and they are being pulled in a TOPS Summative Evaluation Final Report March 2019 67 Element of Theory of Change Evidence related to the precondition thousand ways and we don’t have the time to get involved in the depth we used to. The critique [that IPs get conflicting feedback from FFP staff regarding their TOC] is almost certainly valid. There’s a there there.” (FFP HQ) TOPS Summative Evaluation Final Report March 2019 68 Annex 3 Formal Stakeholder Consultations These events were implemented via extensive qualitative inquiry prior to the event, with formal documentation in advance of positions of actors, questions to be engaged during the consultation, and required extensive effort on the part of TOPS staff to prepare, research, synthesize, implement, and then follow up. 2017 Stakeholder Consultation on Indicator Challenges for Food for Peace-funded Development Food Security Activities (Washington, DC: July 6, 2017) 2016 Stakeholder Consultation on Indicators for USAID Office of Food for Peace New 2016-2025 Food Assistance and Food Security Strategy(Washington, DC: October 5, 2016) Stakeholder Consultation on USAID Pipeline and Resource Estimate Proposal (PREP) Guidance (Washington, DC: July 27, 2016) Stakeholder Consultation on USAID Office of Food for Peace FY16 International Emergency Food Assistance Annual Program Statement (Washington, DC: March 29, 2016) 2015 Stakeholder Consultation on USAID Office of Food for Peace Policy and Guidance for Monitoring, Evaluation, and Reporting (Washington, DC: October 20, 2015) 2013 Stakeholder Consultation on USAID Office of Food for Peace FY14 RFA Guidance (Washington, DC: June 27, 2013)