i Mid-Term Evaluation of The Technical and Operational Performance Support (TOPS) Program Cooperative Agreement No. AID-OAA-A-10-00006 September 2, 2013 “We yearn for frictionless, technological solutions. But people talking to people is still the way that norms and standards change.”1 Dr. Kent Glenzer Lead External Evaluator Monterey Institute of International Studies kglenzer@miis.edu 1 Atul Gawande, “Slow Ideas,” The New Yorker, July 29, 2013, pp. 36-45: 37. ii Acknowledgements The evaluation team thanks the TOPS Program Management Team members, Program Advisory Committee members, Food for Peace DC and field staff, Title II implementing staff, and consortium member senior staff for their availability throughout this evaluation. The evaluation was designed to be collaborative, participatory, and emergent, and our intent was to work with critical stakeholders to find consensus on program strengths and areas for improvement. As lead evaluator, I wish to extend a heartfelt note of appreciation to others on the evaluation team: Tim Ogborn, who did the heavy lifting on the Small Grants Program inquiry, and Centrality Consulting LLC – headed by Shawn Harris and Omar Salem – which designed, implemented, and analyzed three distinct online surveys and input on the web metrics of the FSN Network portal. Lastly, I want to thank the extra effort put in by the Evaluation Advisory Committee, made up of Michelle Gamber from FFP, Brian Kriz from Save the Children, and Tim Frankenberger from TANGO International. The purpose of the EAC was not to influence or shape evaluation findings or recommendations but, rather, to assist me in being clear, politically sensitive, factually accurate, and just broadly well￾informed about the TOPS institutional context in my written products throughout the evaluation process. I use this kind of mechanism often in my work as an internal and external evaluator, and I found the EAC’s questions, concerns, comments, and advice particularly useful in this instance. Kent iii Table of Contents ACRONYM LIST .................................................................................................................................................. 1 1 Executive Summary ........................................................................................................................................ 2 2 Summary of TOPS Program ............................................................................................................................ 5 3 Evaluation Design, Approach, and Methods .................................................................................................... 6 4 Detailed Findings and Recommendations ........................................................................................................ 8 4.1 Program Result Area 1: Knowledge Capture ......................................................................................... 9 4.2 Program Result Area 2 : Knowledge Generation .................................................................................. 10 4.3 Program Result Area 3 : Knowledge Application.................................................................................. 12 4.4 Program Result Area 4: Knowledge Sharing ........................................................................................ 15 4.5 Small Grants Program .......................................................................................................................... 18 4.6 Unplanned Activities and Their Impact on TOPS’ Planned Performance ............................................... 21 4.7 TOPS Management Effectiveness ........................................................................................................ 22 4.8 Save the Children’s Consortium Management ...................................................................................... 24 4.9 TOPS Strategy and Long-Term Sustainability ...................................................................................... 26 ANNEX 1: Evaluation Scope of Work ................................................................................................................. 28 ANNEX 2: Evaluation calendar ........................................................................................................................... 30 ANNEX 3: TOPS Monitoring and Evaluation Plan .............................................................................................. 31 ANNEX 4: Results from surveys of portal users, trainees, and knowledge sharing event participants .................... 34 Survey Demographics ....................................................................................................................................... 34 Trainee Responses ............................................................................................................................................ 35 Knowledge Sharing Event Participants .............................................................................................................. 36 Portal Users ...................................................................................................................................................... 38 Small Grants Respondents ................................................................................................................................ 40 ANNEX 5: A Management Dashboard for TOPS ................................................................................................. 41 ANNEX 6: List of Interviewees ........................................................................................................................... 45 1 ACRONYM LIST ANRM Agriculture and Natural Resource Management CG Care Group CM Commodity Management COP Chief of Party CRS Catholic Relief Services CY Calendar Year FANTA Food and Nutrition Technical Assistance Project FFP Food for Peace FH Food for the Hungry FSN Network Food Security and Nutrition Network ICB Institutional Capacity Building KM Knowledge Management KS Knowledge Sharing LWA Leader with Associates M&E Monitoring and Evaluation MC Mercy Corps MG Micro Grant NGO Non-Governmental Organization OAA Office of Acquisition and Assistance PAC Program Advisory Committee PDQ Partnership Defined Quality PIA Program Improvement Award PMT Program Management Team PR Program Results PVO Private Voluntary Organization RFA Request for Assistance SBC Social and Behavior Change SC Save the Children SG Small Grants SMILER Simple Measurement of Indicators for Learning and Evidence-based Reporting SOW Scope of Work TANGO Technical Assistance to Non-Governmental Organizations TF Task Force TOPS Technical and Operational Performance Support Program USAID United States Agency for International Development 2 1 Executive Summary TOPS is a big player right now. I wasn’t sure at the start that TOPS even had a niche…. It was created, and then after the utility floated into it. FFP sees TOPS as a very good tool to coordinate with a large number of stakeholders rather than be trapped by a few loud voices. FFP sees TOPS as an organizer and interlocutor between themselves and the broader PVO community. And the PVO community sees TOPS as a way of organizing our thoughts and being our megaphone. From both PVO and FFP side….we need TOPS. Senior, Domestic-Based PVO Official I’m just thinking that [the TOPS, rather than the previous ICB] approach is more feasible, in terms of managing changes and knowledge. Individual PVOs have been receiving that grant, whatever knowledge is produced is more for internal uses, and not for cross-sharing and scaling up. But this one, a consolidated approach, is mainly focusing on knowledge sharing and…focusing on how it is happening…whether knowledge is shared among those who are in need of it. I…support the consolidated approach. Senior PVO Official, who runs a Title II program in the field The Technical and Operational Performance Support (TOPS) Program (Cooperative Agreement No. AID-OAA-A￾10-00006) focuses on USAID development food aid assistance program partners. It aims to identify, synthesize, adapt, and share high quality information and tools, build capacity, and establish best practices. TOPS is a five-year Leader with Associates (LWA) Cooperative Agreement (August 13, 2010 - August 12, 2015) with a global mandate. TOPS was built on lessons learned from the generation of Institutional Capacity Building (ICB) grants accorded by Food for Peace (FFP) to individual Private Voluntary Organizations (PVOs) implementing Title II programs. Foremost among these lessons was that PVOs did not actively share best practices among each other, tools were kept inside individual organizational boundaries, and there were, as a result, less than desired wider benefits to the entire Title II implementing community. TOPS was designed to overcome these challenges. Rather than support individual PVOs, the TOPS RFA put much of FFP’s capacity building resources for PVOs into one basket. A consortium of PVOs – Core Group, Food for the Hungry, Mercy Corps, TANGO International, led by Save the Children – was awarded the Leader with Associate (LWA) cooperative agreement. The consortium aims to foster knowledge sharing, promote useful tools, and conduct capacity building and training workshops for PVOs. . This mid-term evaluation rooted itself in principles of participatory monitoring and evaluation, appreciative inquiry, and utilization-focused evaluation. A principle interest of both TOPS management and FFP was identifying areas for improvement, for enhancing the impact of TOPS on Title II implementation practice. The evaluation was structured and constructed as a classic formative effort, one less concerned with assessing the achievement of outcomes or impacts and more concerned with program progress, process, and trajectories. Strategic Strengths/Achievements of TOPS to Date 1. The creation of an inclusive global platform for PVOs implementing Title II programs. The TOPS consortium could have organized itself very differently. An early and persistent strategy by TOPS – Save the Children’s Chief of Party merits specific mention here – was and is to reach out beyond the winning consortium, to forge a strategy of inclusion, of generation of benefits for all Title II PVO implementing agencies. This platform – and interest and voice aggregation it entails – is paying dividends both for the PVO community and for FFP. 2. Clear Knowledge Sharing Strategy and Very Satisfied Knowledge Sharing “Customers.” TOPS decided to invest a great deal of resources in “Regional knowledge sharing events,” large-scale, three day (usually) interactions bringing together PVOs, FFP staff, and others. All attendees touched by this evaluation mentioned these events as one of the principle strengths of TOPS to date. Attendees applaud the participatory and dialogic nature of these events, the variety and utility of the sessions, and the opportunity 3 to engage in face-to-face discussions with peers, researchers, and donors. What is noteworthy is a) TOPS is working off a clear and compelling – and evidence-based – theory for learning and knowledge sharing, one which treats learning is a social phenomenon, and b) participants have already shown themselves willing to cost-share. TOPS’ knowledge sharing events are, in other words, acting as a multiplier for additional resources being channeled into the TOPS effort. TOPS’ technical lead for knowledge sharing merits high praise for strategic leadership; however, the accountability for the success of these regional events is legitimately spread across all technical leads – who put in a great deal of effort – and the TOPS administrative and logistical support staff. 3. High quality technical trainings. Satisfaction scores from trainees are high. Trainee self-reports of transfer of skills from the classroom to the workplace are promising. Pre- and post-test knowledge tests in all trainings show substantial gains. Trainings are well-organized, are becoming increasingly contextualized, and are rooted in a substantial effort to access PVO and FFP field staff input into actual needs. 4. Formation of a highly motivated, collegial, collaborative, and committed Program Management Team. TOPS staff are motivated staff, love their work, appreciate the environment of freedom and delegation that the Chief of Party has created, and know that the Chief of Party has their backs. 5. A simple, practical, straightforward, and feasible monitoring and evaluation plan. TOPS is an emergent design. . TOPS’ M&E plan reflects this, permitting flexibility while – concurrently – holding a set of fundamental indicators steady. TOPS’ information system is simple and straightforward, and basic information about progress is readily available. The M&E approach matches well the resources available for M&E, is clear and explicit about proxies, quantifies what it reasonably can, and is clear about what it is unable to measure due to cost of measurement. This has been a true team effort – all TOPS technical staff have played a role in this – with a firm vision for M&E emanating from TOPS’ M&E technical lead. 6. Stellar FFP-TOPS Relations. The culture of collaboration, mutual respect, and still constructive mutual challenge is impressive. FFP staff engage strategically on the program’s Program Advisory Committee, and communication is frequent and fruitful on more operational affairs. Both sides are to be commended in stretching the run-of-the-mill comfort zone of donor-PVO relationships. There seems a very useful symbiosis at work here. Strategic Challenges/Opportunities to Improve Any new and complex program can find a large number of quotidian, operational improvements. The full evaluation report contains a number of such recommended improvements. However, from a management perspective, a good to ask the question is, “what are the two or three most crucial improvements we MUST make? The following three improvement opportunities rise to the top. 1. Transfer of training, tools, and knowledge sharing to the workplace. While extant qualitative and quantitative evidence triangulates in the right direction, TOPS needs to organize more structured and vigorous inquiry into the transfer of training, tools, and knowledge sharing events to the workplace. Both TOPS management and FFP staff are clear that TOPS is not accountable for transfer. Understanding if transfer is happening, and if not why not, is different from being accountable. Currently, TOPS does not have a robust strategy for acquiring a “good enough” idea about this transfer. The evaluation team recommends that TOPS contract out this inquiry to a firm or consultant and that the inquiry be done annually – part of operational M&E – over the remaining life of the project. 2. Small Grants and Program Improvement Awards. TOPS management organized an in-depth look into its Small Grants (SG) and Program Improvement Awards (PIA) as part of this evaluation. It did so because it already knew there was substantial room for improvement. Procedural and administrative fixes are in place, and this will speed up and streamline awards. What is important now is to better align the two forms of grantmaking with TOPS’ global strategy and raison d’être. The awards should be able to contribute to and move forward – for example – priorities identified by technical task forces. Awards should be considered for supporting operational and action research related to particular tools or emerging ideas/practices identified as promising by technical leads. RFAs should be more focused. Grantmaking mechanisms other than RFAs can be developed, giving the program more flexibility. The Program Advisory Committee (PAC) should play a role in shaping the award strategy. Save the Children should renegotiate its contract with FFP, permitting TOPS to approve up to $100,000. Finally, the evaluation team strongly recommends the hiring of a senior level Director of the Small Grants Program and Program Improvement Awards. 4 3. More strategic engagement with, use of, and contributions by the Program Advisory Committee (PAC). The PAC was established by TOPS management as a site of inclusion for PVOs that did not win the TOPS LWA. It was also intended as a site for coordination, collaboration, and information sharing with the Title II PVO community. Examples exist already of a more strategic role for the PAC: the decision to move towards more engagement with the resilience agenda came primarily from the PAC. This engagement has given PVOs and their concerns a concrete seat at the resilience table. Perhaps more important, it has provided a greater diversity of perspectives, interests, and approaches regarding resilience, ones that usefully engage other major actors such as Feed the Future and the Bureau of Food Security. The PAC has been important, too, in TOPS’ bridging activities with the Food Security Information Network. Robust exchanges have happened as a result between Rome and DC. Yet all PAC members interviewed for this evaluation said they thought the PAC should do more. Indeed, the evaluation team believes that the PAC should be a mechanism that a) helps shape TOPS’ top-level strategy and annual objectives more substantially, b) tasks senior leaders of PVOs on the PAC with important roles for positioning TOPS strategically vis-à-vis other actors and emerging trends in food security, c) serves to generate shared ideas about how individual PVOs can both leverage – and be leveraged by – TOPS, and d) aids in the positioning of the US PVO food security community as an influential player, beyond Title II. The evaluation team recommends that TOPS management be intently focused on single-loop learning related to the current core business of the LWA. Meanwhile, the PAC should be the site for double-loop learning and this will produce a stronger aggregate voice among PVOs than if such strategic thinking were left to the five core consortium members. Save the Children senior leadership should organize a day-long retreat, led by a skilled strategic planning facilitator, during which PAC members themselves jointly flesh out this new role. 4. In relation to specific tools and approaches, deepen and continue the already present shift of emphasis of task forces from discrete numbers of tools uncovered, promoted, or “recommended” to the degree to which such tools and approaches are a) meeting critical needs of PVOs (client perspective), b) opening up new arenas of performance for PVOs (innovation perspective), and c) are being used and are changing performance for the better (impact perspective). TOPS task forces have developed rigorous procedures through which they recommend tools to the Title II implementation community. The idea behind this is logical: task forces are intended to vet tools developed by an agency, or program, a process which then says to other Title II implementers that the tool has been empirically shown to be useful. The tool or approach can be adopted and adapted with confidence that there is actual evidence that it works. To date – as of July 2013 and as per the FSN Network portal where tools appear as “recommended” or not – 12 tools and six sub-tools have earned the “recommended” label. While one can easily critique the pace of production of “recommended” tools and approaches, almost certainly it is a strategic mistake to start focusing on the mass production and dissemination of “recommended” tools. While the evaluation team advises all task forces to have a discussion about the pace of their production, more important is for task forces to be able to a) articulate the most crucial needs of PVO Title implementers, b) demonstrate how the tools/approaches they are working on align with these, and c) collaborate with an external firm or consultant on the question of use and impact, in an ongoing manner, for the rest of the project. These are the most crucial strengths and opportunities for improvement. The evaluation document contains a larger number of more minor and/or operational recommendations, related to specific evaluation questions. 5 2 Summary of TOPS Program The Technical and Operational Performance Support (TOPS) Program (Cooperative Agreement No. AID-OAA-A￾10-00006) focuses on USAID development food aid assistance program partners. It aims to identify, synthesize, adapt, and share high quality information and tools, build capacity, and establish best practices. TOPS is a five-year Leader with Associates (LWA) Cooperative Agreement (August 13, 2010 - August 12, 2015). TOPS has a global mandate. TOPS is managed and led by Save the Children (SC). SC also has technical leadership for gender integration, nutrition and food technology, and commodity management. The other four members of the TOPS consortium are the CORE Group (technical lead for knowledge sharing and management), Food for the Hungry (technical lead for social and behavioral change), Mercy Corps (technical lead for agriculture and natural resource management), and TANGO International (monitoring and evaluation). TOPS established a Program Advisory Committee (PAC) composed of representatives of implementing organizations and other stakeholders. The PAC is a consultative group of major practitioners in food security and nutrition programming that guides and supports TOPS in considering strategic and tactical issues for program implementation. TOPS is a new strategy for Food for Peace (FFP). It was developed in the wake of lessons learned about the generation of Institutional Capacity Building (ICB) grants that numerous PVOs were awarded. ICB grants provided resources to individual PVOs for training and capacity building needs related to Title II programs. The hope was that tools and good practices would be shared among PVOs. This proved not to be the case. TOPS is a consortium that centralizes much FFP Title II capacity building related to PVOs. The program uses three fundamental approaches to address the food security community’s needs: • A directed program of capacity-building activities for USAID’s development and emergency food assistance partners, led by TOPS technical staff; • An inclusive community of practice, the Food Security and Nutrition (FSN) Network, including its technical task forces, interest groups, and web portal (www.fsnnetwork.org), with open membership to all USAID grantees and other food security stakeholders; • A small grants program promoting the creation, use, and adaptation of capacity-building tools and encouraging evaluation and documentation to build the evidence base as part of TOPS’ strategies for the dissemination of best practices to food security and nutrition stakeholders (grantees, local NGOs and other partners, as appropriate), with a focus on field-led and collaborative initiatives. The TOPS Strategic Objective is highest quality information, knowledge, and best practices for improved methodologies in Title II food assistance commodity program performance identified, established, shared, and adapted. The theory of change for the TOPS program leads to the logical specification of four program results (PR) areas: PR1: Knowledge capture: Knowledge and skill needs of audiences identified. PR2: Knowledge generation: Reliable, high-quality information synthesized and produced in user-friendly, appropriate formats. PR3: Knowledge application: Effective and appropriate traditional and non-traditional skill delivery approaches and systems / applications used; and PR4: Knowledge sharing: Information, skill and knowledge exchange supported and expanded. The project document specifies clear intermediate results for each of the. These are shown on the diagram below: 6 TOPS Program Framework, and Theory of Change 3 Evaluation Design, Approach, and Methods Overall Evaluation Approach The evaluation was a formative, mid-term inquiry focused on improvement. The evaluation rooted itself in good practice related to both utilization-focused evaluation2 and developmental evaluation,3 while incorporating principles and approaches related to participatory monitoring and evaluation.4 The primary thrust was qualitative and interpretive. Triangulation among TOPS stakeholders was central to developing findings and recommendations. Evaluation Team Structure and Composition Lead External Evaluator: Kent Glenzer External Evaluator for Small Grants Component: Tim Ogborn5 External Survey Research Evaluator: Centrality Consulting LLC 2 Michael Quinn Patton, Utilization-Focused Evaluation, Thousand Oaks, CA, London, New Delhi, and Singapore: Sage Publications, 2008. 3 Michael Quinn Patton, Developmental Evaluation, New York: The Guilford Press, 2011. 4 Irene Guijt and John Gaventa, “Participatory Monitoring and Evaluation: Learning From Change,” IDS Policy Briefing 12, Brighton, UK: IDS, 1998. 5 Ogborn has produced a separate, detailed report on the Small Grants Program assessment. TOPS Strategic Objective Highest-quality information, knowledge, and best practices for improved methodologies in Title II food aid commodity program performance identified, established, shared and adapted Project Result 4: NETWORKING Information, skill and knowledge exchange forums supported and expanded. Project Result 3: KNOWLEDGE SHARING AND APPLICATION Effective and appropriate information and skill delivery systems/applications used. Project Result 2: KNOWLEDGE GENERATION Reliable, high-quality information synthesized, produced & delivered in user-friendly, appropriate formats & venues. Project Result 1: KNOWLEDGE CAPTURE Knowledge and skill needs of audiences identified. Improved Knowledge Improved Practices Improved Systems Food Security and Nutrition Network Contributing and Benefiting PR 2.1 Recommended tools/methods generated & validated for segmented audiences PR 2.2 Information adapted to Title II in user-friendly & appropriate formats PR 4.1 Physical & virtual community exchange forums supported and expanded PR 4.2 Local networks & capacity building resources linked to FS Network activities PR 3.1 Effective traditional skills delivery systems used PR 3.2 Effective online & virtual information & skill delivery systems used PR 1.1 Relevant and current universe of appropriate tools/methods identified PR 1.2 Knowledge & skills levels of grantees assessed through participatory approaches PR 1.3 Gaps between knowledge/skills/needs & tools/methods documented 7 Evaluation Advisory Team: Tim Frankenberger (TANGO), Michelle Gamber (FFP), Brian Kriz (SC) The role of the Advisory Team was to provide input to the Lead External Evaluator on the evaluation design/approach, preliminary analyses, and recommendations. Everything in the present document is the sole responsibility of the Lead External Evaluator. Evaluation Questions The evaluation focused on the following seven sets of questions: a) What progress has been achieved to date in TOPS’s four Program Results areas and how might plans be improved for the remaining years of the project? b) How effective are the technical capacity strengthening workshops/trainings provided by TOPS program in core technical areas? How might this service be improved? c) How effective are the strategies used by TOPS to strengthen the Food Security and Nutrition Network and what can be done to improve them? d) How effective is the TOPS Small Grants program and how could it be improved? e) What work, assignments, or activities has TOPS undertaken outside the original scope of the RFA and project proposal? How did these originate, by whom, and how effectively have these emergent priorities been implemented? What might be done to permit TOPS to absorb and respond to other emergent or unforeseen priorities in the future? f) How effective is the management of the TOPS Program, both (a) internal TOPS management by the program management team and (b) and the management and oversight of the Cooperative Agreement by the Leader, Save the Children? What changes or improvements could be recommended in TOPS management for the rest of the project? g) How sustainable is TOPS in its current form? What might be changed to enhance the long-term viability of TOPS-initiated processes? Methods The evaluation relied heavily on semi-structured interviews. Overall, evaluators interviewed 54 people, a number of them more than once. In addition, three distinct online surveys were done, each for a different kind of TOPS participant: FSN Network portal users, trainees, and knowledge sharing event participants. These were available in English, French, and Spanish. A total of 207 people responded to one of the three surveys, a response rate of about 16%. The intent of the survey was never to achieve statistical validity but, rather, to serve as a qualitative set of data to triangulate with interviews. A great deal of time was spent studying project documentation such as tools on the web site, training plans and curricula, monitoring data, and portal metrics. Limits of Methods and Data Three important limitations of the evaluation approach bear mention. First, the evaluation methodology did not permit direct observation of TOPS staff in action. Instead, the evaluation team triangulated on performance through staff retrospection, compared these to client and other stakeholder perspectives, and also compared this to known good practice as it relates to knowledge sharing, management, learning, capacity building, and fostering of communities of practice. Lack of direct observational techniques was also addressed through use and analysis of regular monitoring data – data frequently gathered during key TOPS activities such as training, capacity building, dialogue facilitation, and knowledge sharing events -- on product and process quality related to most facets of TOPS’ work. Second, the method did not permit in-depth interviews of an adequate number of Title II implementing staff in the field. The resource requirements to get a high level of engagement from these stakeholders were discussed during the development of the evaluation Scope of Work, and we decided that such an expenditure was not warranted at this formative stage. Particular attention should be paid to this stakeholder group, however, in the remaining life of the project and at the time of summative evaluation. Third, no detailed, in-depth appreciation of every recommended task force tool, or every training workshop, was conducted. This too was discussed during the finalization of the Scope of Work. As with field staff participation in the evaluation process, the resources required to carefully analyze and evaluate these important TOPS deliverables in an utterly external fashion were not deemed a good use of funds at this time. Existing, regular, and ongoing 8 monitoring data collected by TOPS staff during the first 2.5 years of the project were deemed sufficient, combined with semi-structured interviews. 4 Detailed Findings and Recommendations Annex 1 contains the TOPS Monitoring and Evaluation plan. This plan makes explicit the final and intermediate results for the program. It also articulates the indicators to be tracked associated with these results. The tables of results and indicators below are copied from this plan, revised and finalized by the TOPS M&E lead in May 2013. TOPS has a simple, direct, and feasible M&E framework. Great care has been taken to make outputs, outcomes, and their indicators align with the project’s resources. This is strength of the project. The M&E plan is very clear on what indicators mean, how data will be collected, when, by whom, and the most important analytical procedures that will be applied. One aspect of the M&E framework is the use of a set of indicators related to the project goal. While this evaluation was not set up to assess progress toward this final goal, it is still worth documenting what we know at this point. In addition, each final goal indicator also relates directly to one of the four program result areas that were included as objects of analysis in this mid-term evaluation. As such, omitting them from the evaluation would leave an important gap in understanding of TOPS’ progress and opportunities for improvement. TOPS Program Five-Year Strategic Objective Highest quality information, knowledge, and best practices for improved methodologies in Title II food aid commodity program performance identified, established, shared, and adapted. Strategic Objective Indicators Year 2 6 Target Actual Dashboard Assessment7 1. Number of tools in each phase (identified, recommended, shared, adapted) 33 36 2. Percent trainees scoring 70% or higher in post-test assessment 60% 87% 3. % recommended tools incorporated into FFP guidelines, information bulletins 20% 64% 4. Percent Program Improvement Award (PIA) projects that meet objectives N/A N/A 5. Number of organizations served by micro grants 15 69 TOPS has surpassed targets on four of five strategic indicators. While PIA monitoring is only due to begin after PIA awards end, a red flag exists due to the slow pace of fund disbursement for this important component of TOPS. A caution is signaled around the indicator related to the micro grant program too, even though the project’s target was far exceeded. This relates to fund disbursement but, also, to changes in micro grant rules and procedures that 6 Year three data collection against all indicators is in process. This evaluation itself was not designed to collect such granular data. As a result, performance against Year Two targets – which represent the situation as of early CY 2013 – are used throughout this report. Qualitative judgments of the evaluation team are offered about probable progress against Year Three targets where salient to the foci of this evaluation. 7 The “Dashboard Assessment” is a quick-and-dirty rating – by the lead evaluator – of progress. Green = everything ok, no need to change anything. Yellow = a few concerns, “let’s keep them from getting out of hand.” Red = fix this now. This kind of top-level monitoring system is common in the private sector. It amounts, largely, to a “management by exception” approach. The Dashboard Assessment is, therefore, introduced in this evaluation as a recommendation – itself – to TOPS management. Such an approach reduces information overload on managers, can communicate crucial info more quickly and visually to donors and senior leaders in the consortium. The approach, too, relies on team “swarming” on red or yellow light issues. The management team – together – is responsible for tackling those items. The system is particularly appropriate in contexts where overall performance is reliant on interdependence between members of a team, and where underperformance in one area can affect view on performance in another area. Annex 1 shows what the current TOPS strategic M&E plan would look like, were TOPS to adopt such a dashboard. 9 may be desirable. These are covered below in more detail (Section E). Recommendations regarding Indicator 1 are also addressed below in Section B. 4.1 Program Result Area 1: Knowledge Capture RESULT AREA Indicator Target Dashboard Assessment Program Result 1: KNOWLEDGE CAPTURE: Knowledge and skill needs of audiences identified. By end of year five, an effective system is in place for periodic knowledge and skills self- assessment of Title II functional areas by Title II implementing organizations. N/A Intermediate Result 1.1: Skill levels of food security practitioners assessed through participatory and external approaches 1.1.1 Percent of Title II core competency areas defined by the Task Forces. 80% Intermediate Results 1.2: Relevant and appropriate tools identified 1.2.1 Number of [already existing] tools relevant to food security programs identified. 32 1. Progress against plan IR1.1: In year one, as TOPS designed more detailed strategies and work plan, TOPS focused on developing participatory approaches to the assessment of Title II practitioner skills. The use of a participatory assessment approach was chosen in order to build a sense of community among the practitioners. The approach called for TOPS task forces to develop core competencies for their technical areas. Competencies identified, task forces developed self-assessment instruments for Title II implementers. These two products – and the aggregated results of the self-assessments across the technical areas focused upon by TOPS – constitute the indicator for PR1. It is important to note that while this was the formal indicator, the full needs assessment strategy of TOPS is wider than this: it includes substantial conversations with PVO and FFP staff, TOPS staff’s own knowledge and insight, keeping apprised of trends in the food security programming world, and bringing all these strands together in task force deliberations. Six task forces – Agriculture and Natural Resources Management (ANRM), Nutrition and Food Technology, Social and Behavior Change (SBC), Monitoring and Evaluation (M&E), Commodity Management (CM) and Gender – have defined competencies. The Knowledge Management (KM) task force determined that such an approach would not be useful to Title II implementers. It chose instead to identify good and promising practices among Title II agencies for effective knowledge management and sharing. The change of methodology was smart, a process with a higher degree of potential sustainability. IR1.2: This indicator targets the identification of pre-existing tools. It is related to TOPS succeeding the Institutional Capacity Building (ICB) grants as an FFP strategy for building PVO capabilities to implement Title II programs. Technical leads and task forces scoured the existing literature – gray and formal, PVO and governmental, academic and practitioner – during the first two years of the program. Appeals were continuously made to current Title II implementers through the Food Security and Nutrition (FSN) Network portal, newsletter, and during regular task force meetings. By the end of year two, 15 tools were formally identified as useful and worth promoting. . This should not be a worry. The assumption upon which IR1.2 was built – that a wide array of useful tools have already been proven as useful – is not justified. The FSN Network portal contains more than 300 items in its resource library, and the resource library receives high marks for its utility. 2. Major Accomplishments The shift to a self-assessment system rooted in task force recommended core competencies was an important improvement. Building a global culture of dialogue around core competencies – rather than trying to account for and meet unique or highly contextual capacity building needs – offers some modicum of strategic focus for both 10 FFP and the PVO community. Placing the onus on implementers and staff to think critically about their own skills and development needs reduces the likelihood of creating a culture of dependency. 3. Opportunities and Specific Recommendations Opportunities to Improve Specific Recommendation Enhanced transparency and communication to stakeholders regarding aggregate results of self￾assessments Each task force should post aggregated results of self assessments on the FSN Network portal. Wind down investigations into useful tools from the ICB period, and build on the already emerging emphasis on uncovering current Eliminate indicator 1.2.1. Allow indicators 2.2.1 and 2.2.2 to suffice. 8 Account for reality of continuously and always emerging capacity building needs related to core competencies Update indicator 1.1.1 to indicate periodic updating of core competencies and associated self-assessment forms and processes by task forces. Seek structural solutions for implementing rolling self￾assessments with the possibility of aggregation for capacity building planning purposes FFP to require self-assessment during start-up phase of future funded programs, and share these with TOPS task forces. 4.2 Program Result Area 2 : Knowledge Generation RESULT AREA Indicator Year 2 Target Dashboard Assessment Program Result 2: KNOWLEDGE GENERATION: Reliable, high-quality information synthesized and produced in user-friendly appropriate formats. Average score on user perception survey of appropriateness, efficacy and satisfaction in use of FSN TF recommended tools 70% Intermediate Result 2.1: Existing user￾friendly, effective tools endorsed by the FSN Network Task Forces 2.1.1 Number of effective and user friendly existing tools endorsed by Task Forces 16 Intermediate Result 2.2: Tools adapted (created to fill gaps or modified) for Title II grantee use in user-friendly and appropriate formats. 2.2.1 Number of existing tools modified or new tools created and endorsed. 6 Intermediate Result 2.3: Packaged information on promising practices and approaches generated 2.3.1 Number of packaged approaches and promising practices generated and disseminated. 3 Intermediate Result 2.4: Knowledge and tools generated through the TOPS small grants and fed into the dissemination channels 2.4.1 Number of tools created, field tested or validated using small grants 1 1. Progress against plan Only anecdotal data exists regarding PR 2 at this point. A formal, statistically reliable satisfaction survey of tools users was not implemented as planned during year three, and such a survey was judged to be not a good use of evaluation time and resources due to inherent and known problems with PVO staff – field and HQ – responses to online surveys. The SBC task force conducted such a survey during Year Three, and numbers of responses were too low to make the satisfaction scores useful for this evaluation. Interviews for the evaluation suggest, however, that TOPS is on the right track with regard to the usability of recommended tools. As part of the evaluation, all task force recommended tools were subjected to a readability analysis. The average number of grades that a reader would need to complete to understand recommended tools was 12 (U.S. education system), with an average Flesch ease of reading score of about 40. A Flesch score of 60 is – generally – considered an easy read. Recommended tools are – generally – well structured, clear, and logical. 8 See Annex 3, “TOPS Monitoring and Evaluation Framework,” for details on this. 11 IR 2.1 and 2.2: Judging the number of “endorsed” task force tools is an inexact process. For this evaluation, we took a very restrictive and tight position: an “endorsed” tool was one that shows on the FSN Network portal as “recommended.” Overall, by the end of year two, the TOPS performance plan called for 22 endorsed tools, and 39 by the end of year three. TOPS has produced 18 as of this writing in August 2013. TOPS is also on the verge of publishing what appears to be a very important paper on promising practices. IR2.3: No formal data exists on this indicator. There are definitional issues to work out with technical leads and task forces in order to ensure consistent tracking of the indicator. Some of the recommended tools (see IRs 2.1 and 2.2) are, actually, approaches or generalized collections of good or best practice, but for purposes of counting these distinctions are not clear. IR2.4: No progress to date (See below, “Small Grants Program” for more coverage) 2. Major accomplishments TOPS task forces and technical leads are to be commended for their diligence in trying to identify useful tools from the ICB period, and from other sources. This has been a time consuming process. It has also led to useful and generative discussions in all task forces about what constitutes a “tool,” and what makes a tool endorsement ready. Tools that have been endorsed by the FSN Network task forces are – for the most part – being downloaded in reasonable numbers.9 Trainings related to the tools are also receiving high ratings, another useful indicator that the tools are, indeed, meeting the needs of the target audience. The focus on tools that are proven as useful through objective, reliable monitoring and evaluation and/or research processes is laudable. It moves Title II program PVOs into the current powerful discourse on evidence-based decision-making, rigorous assessment of impact, and results-based management. TOPS technical staff are making a direct assault on PVO practices that have for far too long been mostly anecdotal in nature, more art than science, and unable to stand up to scrutiny. An irony, however, emerges here: a very deep and devoted effort to separate the proven tools “wheat” from the anecdotal “chaff” implicitly turns the TFs into a somewhat conservative structure, one more focused on adherence to certain standards rather than the more risky business of innovations in practice and approaches. It is easy to imagine a vision for TFs that has less to do with vetting “tools proven to work for producing X outcome” and more on developing potentially useful – if yet unproven – tools for solving new problems or challenges. Useful examples of this approach are already present in the current SBC TF. 3. Opportunities and Specific Recommendations Opportunities to Improve Specific Recommendation Shift emphasis of task forces from discrete numbers of tools discovered, promoted, or “recommended” to the degree to which tools and approaches are a) meeting critical needs of PVOs client perspective), b) opening up new arenas of performance for PVOs Instead of a binary endorsement (either a tool is or is not endorsed), use a multi-criteria rating system. Criteria could be akin to the following: 1. Tool recommended by at least one practitioner who has used it 2. Tool recommended by a PVO (i.e., beyond a single person’s opinion) 3. Tool has been subjected to external evaluation/review (i.e., beyond one person, or one organizations) 9 Reference population for this conclusion: the number of Title II programs currently running. “I think the TF’s are the right vehicle, however…I think they could be pushing the envelope a bit more and advancing the field more aggressively” 12 RELEVANCY OF TRAINING TO YOUR JOB (N=49) ALMOST NONE 2% A LITTLE 0% SOME 8% A LOT 55% A VERY LARGE AMOUNT 35% ( (innovation perspective), and c) are being used and are changing performance for the better (impact perspective). 4. Tool has been recommended by more than one actor 5. Tool has been recommended in more than one operating context 6. Tool has been recommended or already become standard practice in an organization other than the original recommending organization. 7. Tool has been subjected to rigorous academic study, and shown to produce results that are better than other tools meant to do the same thing. Such a rating system would be very quick to apply, and remove the necessity of gaining large-scale consensus among task force members. Use the small grants program to allow task forces to commission specific research on particularly controversial or promising tools See also recommendations for an external consultant to conduct annual reviews of uptake of training, tools, and knowledge by PVO implementers Program Result Area 3 : Knowledge Application RESULT AREA Indicator Year 2 Target Dashboard Assessment Program Result 3: KNOWLEDGE APPLICATION Effective and appropriate traditional and non-traditional skill delivery approaches and systems/applications used Percent of Title II implementers aware of at least 50% of key FSN TF-recommended tools 50% Percent of projects reporting use of FSN TF-recommended tools 40% Average score (normalized) on post-training satisfaction survey 80% 1. Progress against plan To date, TOPS has surpassed all of these targets. According to year two monitoring data, 73% of Title II implementers are aware of at least 50% of key tools, 52% reported using recommended tools, and average post￾training satisfaction is about 90%. Trainees have scored on average 87% on post-training knowledge tests, far surpassing the 70% target, and representing a notable and clear gain from knowledge baselines done at the start of the trainings. 2. Major accomplishments TOPS trainings are receiving high marks from participants. The manner in which TOPS technical leads identify training needs is well structured and clear and strategies for prioritizing capacity building needs represent a useful combination of bottom-up and top-down inputs. Qualitative data from interviews and an online survey of training participants leans towards a tentative conclusion that TOPS training is probably translating into alterations in practice in the workplace. 13 CONTENT APPLICATION AFTER RETURN TO WORK (N=49) ALMOST NONE 0% A LITTLE 8% SOME 29% A LOT 55% A VERY LARGE AMOUNT 8% Strengths of the Trainings (n=47) Relevant and Quality Material 28% Effective Facilitation 23% Inclusion of M&E and Data Analysis 18% Diverse Participants 15% Suggested Improvements for the Trainings (n=41) Need more time to cover designated topics/too condensed 21% Increased focus on the field 21% More Data Management and M&E Focus 16% follow-up Sessions/Check-ins (Post-Training) 12% Participants with Similar Experience Level 9% Respondents to the mid-term evaluation’s online survey almost all said that the training contents had either “a lot” or “a very large amount” of relevance to their actual jobs, and more than half said that they have been able to apply “a lot” or “a very large amount” of the training in their jobs. Food for Peace staff who were deeply involved in training – the M&E and Commodity Management trainings involved very close collaborations – unanimously said that TOPS trainings in these areas have made a palpable difference. Meanwhile, those responsible for implementing the trainings all report a high level of ongoing engagement – queries, questions, additional requests for guidance – from trainees after the workshops. TOPS’ overwhelming reliance on traditional training – face-to-face – is a major strength and this fundamental strategy should continue. The overwhelming majority of Title II field staff have little experience with Web 2.0 and distance learning strategies, these are not institutionalized within HR departments of PVOs, and training venues themselves forge social connections, social capital, and ongoing relationships that are important. Online survey respondents were asked two open-ended questions about the strengths and areas for improvement of TOPS’ training. Respondents were asked to list the three most important strengths, and three most important areas for improvement. The survey results on “strengths” dovetail well with interviews and end-of-training questionnaires. TOPS is currently organizing relevant, high quality trainings that are well-facilitated and address issues that trainees identify as practical, useful, and important to their work. One of two most commonly mentioned improvement areas – needing/wanting more time in training events – is another indicator that TOPS technical leads are organizing trainings that are relevant. That 21% of responses on improvements highlighted the need for an increased focus on the field should be interpreted, therefore, not as a lack or gap in TOPS’ training strategy but, rather, as an indicator of the difficult any global initiative would have in fully pleasing Title II staff spread across three continents. Some technical leads have encountered unsurprising and predictable challenges in coordinating training with FFP field staff. Initial contacts must be made through FFP in DC, field staff are swamped already, and the formal protocol of FFP communication doesn’t always result in the right field person contact for the particular TOPS technical training area. TOPS technical leads have learned a great deal in the last 18 months however about the necessity to coordinate, and FFP staff in DC have also recognized they have a crucial role in facilitating this relationship. The coordination efforts have meant that technical training efforts – in some areas – are less “nimble.” Worse, in three cases TOPS has had to cancel trainings in the field because of delays in securing FFP field staff approval. This represents a very large amount of time essentially wasted by TOPS technical leads. It also prevents time-sensitive training – in one case, the training was designed to help a PVO with a baseline study that could not be delayed – from happening, hence reducing TOPS’ impact and effectiveness. “We’ve gotten decidedly less nimble. It does endanger us reaching our targets” 14 It is important to point out that when coordination works – such as with commodity management and M&E trainings to date – all stakeholders benefit. While – officially – TOPS technical leads do not need formal permission from FFP field staff to undertake training requested by and agreed with Title II PVOs, there is a political need to coordinate. Host governments may well query FFP – not TOPS, nor trainees – about the nature of the training event held on their soil. FFP staff need to reply with accuracy and promptness. In addition, TOPS technical staff may not necessarily be fully informed about the politics of particular contexts: the problems TOPS had coordinating a training effort in Ethiopia with FFP is a good case in point. TOPS technical staff were simply incognizant of the nature of that government’s relationship with international donors and particularly NGOs. “I think they need clearer and more open conversations with the mission,” said one FFP field staffer, “What’s hard to understand is that I don’t really know what I should expect out of TOPS. Should I be expecting a lot more out of them? Or a lot less? This has to do with our own [FFP’s] internal communications, too. It’s not all on TOPS.” TOPS is constrained from planning training and capacity too far in advance. TOPS wishes to remain flexible, lithe, and responsive to needs of PVOs, and be able to mobilize and respond in good time. A four-month planning horizon for countries and/or regions – for most technical areas – seems possible; whether four months is enough time for FFP is an open question. 3. Opportunities and Specific Recommendations Opportunities to Improve Specific Recommendation Stronger and more proactive promotion of TOPS’ unique value￾add when it comes to training and capacity building, so as to better shape stakeholder expectations, coordinate actions with them, and engage in longer-term planning horizons. Post one-page, rolling, training/capacity plans – for each technical area – on a new TOPS web site in ways that are easy for visitors to see. (More on the need for a TOPS specific web site in the next section.) Clarify – and post on a new TOPS web site – a statement of what Title II implementers and FFP field staff can expect from TOPS’ capacity building and training efforts, how these relate to training and capacity building that PVOs should expect to do themselves, and how TOPS’ efforts relate to FANTA’s. Investigate how direct voice conversations – rather than email responses – can be used for crucial decisions. TOPS Chief of Party needs to strengthen personal contacts and relationships with FFP and USAID field decision-makers. The evaluation team has strong doubts that any new rules or procedures meant to make decision making more efficient will be very useful. Instead, building of stronger social capital between the Chief of Party and field staff is needed. FFP DC to clarify TOPS’ authority to organize training and capacity building with decision-makers in the field. Greater investment in understanding degree of transfer of training and capacity building to the workplace Engage an external consultant or agency to conduct annual inquiries into training transfer. The method needs to consist of very active outreach to trainees – not passive web surveys – and guidance for COPs and FFP field staff to easily observe transfer or not. This consultant or agency should also look closely at the current training of trainers – or training of project coordinators who are then expected to transfer the competencies to others in perhaps less formal ways – approaches of all technical leads. Greatly enhance opportunities for PVOs to get high-quality training Technical leads should spend more time building up a stable of training consultants who can then expand the opportunities for field staff to get high quality capacity building. These consultant lists should be readily available online, for Title II staff to find. This should be considered an important element in TOPS’ long-term sustainability. Technical leads should place more emphasis on doing training-of-trainers rather than direct, hands-on training of Title II staff themselves. “We really stepped in it in Ethiopia.” 15 22% 11% 34% 33% 0% KS Event Content Transfer to the Job (n=27) Very little A little Some A lot A very lot 4.3 Program Result Area 4: Knowledge Sharing RESULT AREA Indicator Year 2 Target Dashboard Assessment Program Result 4: KNOWLEDGE SHARING: Information, skill and knowledge exchange forums supported and expanded. 4.1 Number of members regularly participating in FSN Network technical task forces 70 4.2 Number of participants attending FSN Network technical knowledge sharing meetings 500 4.3 Average score (normalized) on post FSN knowledge sharing meeting satisfaction survey 80 4.4 Average number of members participating in FSN Network listserv and discussion groups 850 4.5 Number of downloads from the FSN website 1000 4.6 Number of members regularly participating in FSN Network technical task forces 70 Intermediate Result 4.1: Physical and virtual community exchange forums and knowledge sharing mechanisms supported and expanded 4.1.1 Number of hits on FSN Network website 10,000 4.1.2 Number of regional interest groups actively sharing knowledge on key implementation issues 5 1. Progress against plan All targets have been surpassed, most by a very large amount. Only indicator 4.1.2 has questions around it, and this is already a priority for TOPS’ portal staff, with plans to enhance the utility to Title II and other food security field staff of these fora. 2. Major accomplishments TOPS’ regional knowledge sharing events are one of the programs’ most important – and high profile – contributions. Participants are nearly unanimous in their praise for these events. They provide crucial opportunities to bring PVO, FFP, and other actors together to share experiences and through doing so build social capital in the larger Title II and food security and nutrition epistemic communities. Participants appreciate the engaging, participatory, and active nature of the regional knowledge sharing events, the careful preparation done with presenters/facilitators so as to avoid “chalk-and-talk” or death by PowerPoint kinds of sessions. While anecdotal in nature, the data from the online survey points in a heartening direction regarding transfer of contents to the workplace (see box to the left). Each event has been treated as a unique whole, and TOPS leadership responsible for organizing the events has allowed each to develop its own character without imposing a cookie cutter formula. While 16 suggestions for improving these events are easy to hear – longer, shorter, more free time, less free time, more deep dives, more topics and experts – the leadership for these events is advised to keep doing what it is doing: let each event take on a life, structure, and process of its own, which will never be perfect for the hundreds who attend these events. Themes Covered in TOPS’ FSN Network Regional Knowledge Sharing Events, 2010-2012 Due to FSN Network’s vision of reaching beyond the Title II world, we analyzed the degree to which actors other than Title II implementing PVOs led sessions at these events. Analysis revealed that 36% of all presenters/facilitators came from TOPS consortium members, 27% from TOPS collaborating partners (other Title II PVO implementers, for the most part), 10% from other NGOs, about 8% from FFP, and 9% from academia. The high participation from Title II PVOs demonstrates a true commitment of them to these events, and the degree to which non-TOPS consortium PVOs have been given space in these events is a testament and tribute to TOPS desire to build a very wide community. The table above shows the distribution of themes/topics covered in the four regional knowledge sharing events to date. Not surprisingly, the majority of time (about 74%) in these knowledge sharing events is devoted to themes associated with one of TOPS technical task forces. This shows how active all TOPS consortium members – each with their technical area of focus and responsibility – are in these events, and also shows how no single task force is dominating. The one non-task force theme – Food Aid/Security Policy, Strategy, and Theory – that hit double digits was driven primarily by the focus on resilience in the Washington DC event. Task Forces are a very important part of the FSN Network’s strategy to foster, catalyze, and promote communities of practice around important technical areas for Title II – and wider – food security practitioners. Task forces provide crucial person-to-person exchanges, permit more in-depth discussions and dialogues, and serve to connect “Not only do these events carry out what they are tasked with doing, but it also puts an element of public accountability out there, which is important and aids in PVO utilization and buy-in with what TOPS is doing” Technical Area Number of Sessions Total Hours % total of all hours ANRM 8 12 8.9% AG/Nutrition 5 8 5.9% SBC 11 16 11.9% Commodity Mgmt. 5 8 5.9% M&E 9 13.5 10.0% Nutrition and Foot Tech 12 17.5 13.0% KS/KM 7 11.5 8.6% Gender 8 12 8.9% Management of Food Aid 2 3 2.2% Food Aid./security Policy, Strategy, Theory 12 17.25 12.8% Emerg. Response 2 2.5 1.9% Comms/Pub. Rel. 2 3 2.2% HIV/AIDS 1 1.5 1.1% Keynotes/Other 6 8.75 6.5% TOTALS 90 134.5 1 17 STRENGTHS OF THE FSN PORTAL (50) TOOLS ARE CONCISE, TIMELY AND RELEVANT 43% USER-FRIENDLY 20% GLOBAL COLLABORATION CAPABILITY 11% and knit together tools, capacities, training, and individual growth. While almost all Task Force leaders express concern about field participation in Task Forces – only the M&E task force exhibits any consistent participation of this kind of actor, and this through heroic efforts by the former TF lead – the current level of participation is neither surprising nor necessarily cause for alarm, or cause for great changes. To their credit, TF leads have tried a very large number of ways to augment field participation in the task forces and their continuous search for solutions and commitment to field participation is strength. TOPS’ web strategy relies on a useful resource library as a broad pull and benefit. The Portal users survey provides anecdotal data that this strategy is working. The FSN Network portal is, for the most part, cleanly designed, simple to navigate, fast, and delightfully free of clutter. Aspects of the web site that are mildly confusing are all known to the current portal strategist, and effective action in the works. Web statistics for the site are heartening, with users staying on the site, investigating multiple pages, and engaging in good numbers of downloads. The portal users survey done for this evaluation reveal that users rate almost all features of the portal as more valuable to them than other, similar web sites they visit for similar information, knowledge, and updating requirements. Portal users, as the table below reveals, clearly find the portal of high value. Perceived Value of Primary Elements of the FSN Portal in Comparison to Other Online Resources (N=47) NOT VERY USEFUL A LITTLE USEFUL USE￾FUL VERY USEFUL I DON'T KNOW/HAVE NO OPINION RESOURCE LIBRARY 2% 10% 27% 24% 8% EXTERNAL LINKS 5% 7% 35% 31% 22% FSN NETWORK DISCUSSION EVENTS 2% 10% 32% 46% 10% ONLINE DISCUSSION GROUPS 2% 15% 30% 42% 12% FSN NETWORK UPDATES 3% 10% 27% 52% 8% AVERAGE 3% 10% 30% 45% 12% REASONS FOR VISITING THE PORTAL (N=64) RESOURCE LIBRARY 73.1% FSN NETWORK UPDATES 21.3% FSN NETWORK DISCUSSION EVENTS 52.5% ONLINE DISCUSSION GROUPS 68.9% EXTERNAL LINKS 42.6% SUGGESTIONS FOR IMPROVEMENT REGARDING THE FSN PORTAL (32) IMPROVE NAVIGATION FUNCTIONS 30% MORE INTEGRATION OF FIELD LEVEL ISSUES AND EXPERIENCE 21% UPDATE RESOURCE LIBRARY MORE OFTEN 21% MORE WEBINARS AND EVENTS 12% 18 FSN Network’s semi-weekly newsletter is nearly universally praised as concise, readable, and useful. A number of interviewees mentioned that while they are on many mailing lists of similar newsletters, the FSN Network’s newsletter is one they actually open and read rather than simply delete from their inbox. The portal users online survey asked respondents two open-ended questions. One requested they list three strengths of the portal. The second asked to list the three most important improvements. While not statistically valid, the survey results triangulate convincingly with interviews, the evaluation team’s hands-on assessment of the portal, Google metrics for the site, and TOPS’ staff opinions. All of the suggested improvements are, in fact, being actively implemented by the current portal manager. 3. Opportunities and Specific Recommendations Opportunities to Improve Specific Recommendation Diversify what it means for field staff to participate in Task Forces Institute one meeting of TF’s each year that is face-to-face, and begin with travel stipends for field staff to attend, gradually phasing these out as TF participation gains its own value inside of field staffs’ programs. Use this meeting as the central planning and strategizing moment for the entire year. Reinforce norms in task forces with regard to members’ obligations to seek front-line staff input on important issues. Not all participation should be thought of – in other words – as measured by who actively participates either in task force meetings, or online discussion associated with a task force. Give each task force a formal space, time, and place during regional knowledge sharing events to build community and reach further out into the field for engagement. Inquire more deeply into what is transferring from regional events into the workplace Engage an external consultant or consulting firm – preferably the same as for training/capacity building follow up – to strongly pursue this inquiry on an annual basis for years 3, 4, and 5. Invest strongly in this question during the final evaluation. 4.4 Small Grants Program 1. Progress against plan The Small Grants Program (SGP) experienced significant difficulties during the first round of RFAs. An important message, however, is that much has already been changed about procedural problems in both SC and FFP related to grant approval and administration. As a separate, full report on the SGP is called for in relation to this mid-term evaluation, details of these challenges and solutions already agreed and implemented will be telescoped here. The initial RFA process was cumbersome, prolonged, and difficult. In short: many informants say that the hoops they had to jump through – and the extensive set of questions they had to respond to and the length of time they had to wait for notification of approval – were not worth the small amount of funds in question. Since then, FFP has established new review deadlines and authority has been delegated by Save the Children to the TOPS team to issue their own awards. The SGP is seen by members of the Title II PVO community as an important source of funding to address the development of new tools, for carrying out research, and for funding the sharing of tools and learning. The biggest attributes of the SGP are seen to be: I. It is a unique source of funding that is demand driven without an agenda behind it and which can fund activities that are not usually funded by donors, this is particularly appreciated by those PVOs that do not have large unrestricted income flows II. It should have a quick turnaround enabling the grants, in particular the Micro Grants, to be responsive to rapidly changing needs 19 III. It has a strong requirement for sharing and benefitting the wider Title II community. The SGP is significantly behind its disbursement schedule. Only four Micro Grants (MGs) have been awarded, of which three have been completed, with six more grants in different stages of review at this moment. Five Program Improvement Awards (PIAs) have been made with the second round RFA issued in June 2013. Only $584k has been awarded out of a possible $1.8m small grants budget for years 1 through 3 – less than 32% . The slow SGP implementation hurts TOPS’ ability to achieve its overall goals. 2. Opportunities for Improvement and Specific Recommendations There are 4 key areas of opportunity which, if attacked effectively, should lead to significant improvement in the SGP. Combined, they will help address the most often-mentioned reasons that portal users, trainees, and knowledge sharing event online survey respondents mentioned for not applying for a program (see box to the right). These are:  New integration of the SGP into the overall strategy of TOPS;  More predictable and transparent approval process;  Increasing both application and acceptance rates; and  Better communication a) SGP integration with overall TOPS strategy To date the SGP has been positioned as a responsive funding mechanism reacting to ideas and proposals submitted in response to broad and open RFAs issued by TOPS. This positioning is actually well appreciated by most PVOs and is listed as one of the key attributes of the program. This positioning can be respected moving into the future, while also adopting the following recommendations:  Hire a senior SGP Director, at the same level of experience as technical leads. Make this position the official deputy of the Chief of Party.  Develop RFAs that drive a broader, TOPS strategic agenda. TOPS’ strategic objective is broad and inclusive, and the project should not be shy about its importance;  Incorporate SGP responsibilities explicitly into all Technical Leads job descriptions in two ways: o Responsibility for utilizing the SGP to move forward Task Force priorities o Responsibility for participating in grants review activities.  Consider different types of RFAs and granting mechanisms that enable TOPS and Task Force leaders to be more directive. Possible examples are: o Subject-specific RFAs (already being considered by TOPS) o Reserving part of the small grants fund specifically for Task Force developed priorities o Provide a grant to a PVO to contract and oversee research done by an academic to produce a state of play/position paper on a particular issue.  Develop a grants manual that allows for innovative approaches to grant-making that is not constrained by the typical USAID hands-off competitive model. USAID rules and regulations are very flexible about how grants can be issued and TOPS should make as much flexibility as they can to use the SGP to advance the TOPS agenda b) More predictable and transparent approval process Much of good sense has already been changed about the cumbersome first RFA process. Both TOPS and FFP staff are to be commended for these changes. Notwithstanding the improvements, there is one big question to be asked: Why is USAID so substantially involved in the approval of such small grants? Why Have You Not Applied for Small Grant or PIA Awards? REASON # of Times Mentioned I didn’t have an idea that qualified for funding 18 I didn’t know how to apply 17 I don’t know why I did not apply 13 My organization is not eligible 12 The application process was too burdensome 8 I don’t have the authority to apply 8 Timing wasn’t right; plan to apply next time 5 20 Originally FFP agreed to have TOPS review the small grants and then issue grants without involvement of FFP officers. This is typically how small grants programs under Cooperative Agreements operate. The exceptions to this are usually when the operational environment is such that very significant concerns can be raised about potential problems with small grants programs, e.g. in countries like Iraq and Afghanistan with grants being issued to local NGOs. Despite this the OAA determined that USAID needed to review and approve each project individually. This has put both SC and FFP in an unenviable spot. This can be rectified by:  Save the Children should renegotiate their agreement with USAID to permit TOPS to approve grants up to $100k without further approval by USAID  FFP should be substantially involved in the small grants program through engagement on the setting of SGP strategy and approval criteria, rather than project by project approval  TOPS should publish the expected turnaround time for small grant approvals in their RFAs and on appropriate websites c) Increasing application and acceptance rates There have been a total of 44 proposals submitted, 22 MGs and 22 PIAs. However, of these 2 MGs and 9 PIAs were outside the scope of the SGP, and so effectively there have only been 33 proposals, 20 MGs and 13 PIAs, in the running for awards. This is not a high number considering the amount of funding available and the number of potential grantees. All key applicant informants mentioned that the $100k maximum and the 1 year time limit were both factors limiting their interest in applying. In addition, the number of rejected awards is also high. 57% of the submitted proposals have been rejected so far with only 20% awarded, and 23% pending review. Even if we exclude the 11 proposals that were outside the scope of the SGP, the rejection rate drops to 42% with only 27% awarded, and 31% pending review. These proposals have, in the main, been submitted by long time experienced Title II implementers which know Title II and food and nutrition security pretty well. There are two main reasons for the high rejection rate:  The strategy of issuing RFAs and waiting for proposals has led to a scattergun set of applications which do not necessarily align with priority areas for TOPS and, in 11 cases, ended up with proposals outside the scope of TOPS completely.  The proposal review process is unreasonably rigorous and rigid for proposals of this small size. Applications could be increased, and acceptance rates improved by:  Simplify the proposal review process focusing the review on whether or not the project satisfies the specific criteria in the RFA  Limit the scope of issues questions to ones addressing the small grants strategy, and not ask questions about the already approved project strategy to which the small grant is attached.  Extend the time limit for projects to two years where appropriate, for example to include more than 1 agricultural season. d) Improving communications The main mechanism for communication about the SGP is through the FSNN e-Newsletter. This is adequate for announcing and publicizing RFAS. However, there is no central repository on the FSN Network portal where comprehensive information about the small grants can be found. The RFAs are available, although the links to them are somewhat hidden at the bottom of the page, and there is no information about what projects have been funded, what progress has been made, and no list of project final reports. From the positioning on FSN Network’s portal, once would conclude that the SGP is peripheral to TOPS, rather than a central lever for program improvement, tools development, and capacity building. Three interviewees called SGP communications a “black hole.” Two used the term in reference to the application process and the other to the lack of communication after an award was issued. Also two members of the TOPS staff indicated that internal communications updating them about the progress of small grants was inadequate. After reviewing and providing feedback on small grant applications, no systematic update was provided on the final decision. 21 Clearly there is a significant problem with transparency regarding the SGP. There is no intention to hide the workings of the SGP, but insufficient thought has been put into the types of communications that are necessary to ensure that all members of the PVO community and FFP are adequately informed of its progress. Communications could be substantially improved in the following ways:  Make the SGP very prominent on the FSN Network portal, preferably as one of the key headings across the top of the page. It is a very significant resource for members of the network and should be advertised on the network website as such. It should be presented as an integral part of the resources available to network members and not hidden away at the bottom of the front page  The SGP page on the FSN Network should be a comprehensive record of all activities of the SGP. The page should be a one-stop information resource for anyone interested in applying for a small grant, or just interested in the state of play of the different grants that are being implemented. It is recommended that the following should be kept updated on the SGP page: o Current RFAs o Policies, procedures, submission dates and corresponding approval turnaround times o Contact information for key SGP staff o All successful proposals (minus confidential financial information) o Current statistics of the SGP program – numbers of applicants, amount of funding available and obligated, etc. o Final reports, meeting reports, and other small grants outputs o Progress of implementation of all small grants – start date, dates of events (see bullet below as well), significant outputs, etc. o Consolidated calendar of events funded by the SGP  Standard communications with small grant applicants should be planned such that each applicant receives detailed information as to the progress of their proposal. If unexpected delays take place, then applicants should be informed as to what is happening and why, and how long it is expected to take to resolve the issue  Internally, TOPS should have regular written communication about which projects have been funded and which not, and the reasons why. No TOPS staff member should be unclear as to the status of any small grant, particularly ones they have reviewed. 4.5 Unplanned Activities Performed Upon Request of FFP and Their Impact on TOPS’ Planned Performance Since inception TOPS has responded readily to FFP requests to address topics and provide services that were not anticipated in the original proposal. All activities have been aligned with TOPS strategic objective. Examples of this responsiveness to unplanned needs include: B. Taking on the role of commodity management capacity building when the Capable Partner project ended in late 2011; C. Organizing a two-day USAID conference addressing resilience strategies in the Horn of Africa; D. Facilitating PVO input into a new RFA being planned, prior to its finalization; E. Facilitation of dialogue among PVOs regarding FAFSA 2 findings and recommendations; and F. Assistance organizing and facilitating dialogue related to FFP’s evaluation policy. Food for Peace staff in Washington, DC greatly value and appreciate TOPS’ staff engagement with these important events and dialogic processes. High quality discussion and participation matters a great deal; FFP’s ability to call on TOPS’ expertise is helpful to the agency and to the wider food security and nutrition professional community in and around DC. High quality events burnish FFP’s image among important stakeholders and authorizers; they also serve as high-profile public relations and brand-building opportunities for TOPS. Both TOPS and FFP staff recognize this value. The evaluation scope of work called for an inquiry into whether these ad-hoc activities were in any way compromising TOPS’ delivery on the core business of the LWA. TOPS senior staff say that, thus far, 22 responsiveness to ad-hoc demands has not affected tools development, training, or knowledge sharing priorities. FFP DC staff are very clear on the issue: There is no strategic drift or confusion. TOPS has been asked if it can assist a) because of specific competencies, but also b) because involvement in event leadership can contribute to brand￾building and even long-term sustainability of TOPS. If TOPS requires additional financial resources to take on new work, they simply need to make the case. To date – commodity management aside – TOPS has not requested any additional resources, human or financial, to assume ad-hoc responsibilities requested by FFP. To date, TOPS management has not presented any arguments – using, for example, annual plans and their stipulated achievements – to make it clear that the opportunity costs are simply too high. Reliance on logical straight talk coupled with good will, political and social capital, and cordial yet still frank relationships cuts through a great deal of bureaucracy, clutter, and red tape: in the best of times, it is a win-win for both TOPS and FFP. It is a sign of the excellent working relationships between FFP/DC and TOPS management that it has – for the most part -- worked well so far. But it is likely time for a few simple rules of the game. Opportunities To Improve and Specific Recommendations Opportunities Specific Recommendation There’s nothing essentially “broken” here, so build on the useful flexibility and fluidity that current FFP and TOPS collaboration exhibits by instituting two commonsense rules/procedures. FFP/DC and TOPS should agree on what constitutes reasonable lead time to request TOPS assistance in unplanned events. For major processes/events, three months seems a useful benchmark. For more minor processes/events, one month seems reasonable. TOPS annual plans should include a new column, indicating the number of days that TOPS staff estimate are needed for their strategic deliverables. In essence, this recommendation asks TOPS staff to think a bit more like consultants. When FFP – or perhaps another actor – approaches TOPS for help, the Chief of Party should be able to point to a planning document that shows what kinds of space staff have in their current plans to respond to ad￾hoc requests. 4.6 TOPS Management Effectiveness The inquiry into TOPS’ management effectiveness looked into questions of performance planning, documentation, and monitoring; staff management and satisfaction; conflict management and resolution; communication with stakeholders; donor relationships and reporting; and leadership. Broadly speaking, the effectiveness of TOPS management is lauded by stakeholders. All recognize that the consortium represented a strange animal to start with, tackling a new kind of task, under an innovative structure. Rather than institute a cumbersome TOPS consensus process for project decisions, the Chief of Party chose a deeply delegative approach, one which provided ample space to all consortium member technical leads – and subsequently technical task forces – to independently plan and implement their work. This approach is highly praised and valued by all. The Chief of Party is universally praised as supportive, helpful, as “having our backs”, and in actively shaping the collaborative space with FFP. A central mechanism in TOPS’ management is the Program Management Team (PMT), consisting of all staff who work in the TOPS DC office, all consortium member technical leads, and a Save the Children finance officer. The PMT meets semi-weekly. The primary purpose of the meetings is information sharing, coordination, and blocking￾and-tackling new operational issues. In other words: single-loop learning. Facilitation responsibilities are shared, with the Chief of Party ready to step in and move agendas along if needed. When complex issues arise they are delegated to a subset of the PMT rather than dealt with by the full body. “I don’t know what they are not doing because of things that I’ve requested. It’s not totally clear to me what is being taken away, and how much I can ask TOPS to do. I haven’t asked as much as I really would want to, because I’m unclear….Everybody I’ve talked to [in TOPS] is amazingly flexible and open to listening to me.” 23 PMT members in general express satisfaction with these meetings, saying in particular that they have improved greatly since the first year of the program. PMT members also think the frequency of the meetings is good. Two areas were cited for improvements: first, a number of PMT members said they wished the meetings would result in clearer action items, and that these items be more rigorously followed up on in subsequent meetings. There is a sense, in other words, that issues sometimes are raised but left hanging, with staff unsure what will be done about them. Second, while delegation and freedom to act is highly valued – nobody in TOPS feels micro-managed – some PMT members say they wish that “more substantive” discussions arose and were allowed to happen within PMT meetings, from time to time. There is a clear feeling of camaraderie among TOPS staff, and personality and professional conflicts have been dealt with as they arise in a conscious, planned, and professional manner. Hard, fraught, or difficult decisions between consortium members are handled outside the PMT, through bilateral relationships between the COP and consortium member senior staff. There is a great deal of mutual respect among TOPS staff, and the COP spends important time and effort on enhancing these relationships of trust and collaboration. TOPS internal annual planning process and format is not mentioned as an issue by any staff. Plans are kept flexible, and are revised mid-year to account for change. This is good practice, although TOPS management likely needs to spend more effort on contextualizing and explaining the importance of changes to stakeholders. TOPS monitoring and evaluation plan is very good. It is clear about its theory of change, reasonable in its selection of indicators and methods, and transparent about what the program will and will not measure, and how. The TANGO M&E technical lead who works in TOPS has assiduously gathered data on a regular basis, the data is stored using the minimal necessary technical level – Excel and Word files suffice just fine – and the M&E plan is revised and updated annually. The simplicity of TOPS’ M&E approach is a strong asset. It is also courageous, in that it makes very clear tradeoffs between “must know” and “could, or would be nice to, know,”, and eschews all forms of the latter. Annual reporting – from an M&E perspective – is comprehensive and aligns 100 percent with the project proposal and the program’s theory of change. While this is true, the report itself may not be as useful to FFP, consortium members, and the wider field of collaborating partners as it could be. A common observation from nearly all stakeholder groups was that while TOPS activities and products are high quality, it was hard to understand more broadly what TOPS does, doesn’t do, and fundamentally what it is achieving. TOPS is achieving much, producing important deliverables, but the current reporting format may not communicate this in the best way. Annex 5 contains a quick and dirty version of a “Management Dashboard” that might be useful. TOPS’ relationship with FFP in DC is excellent. There is frequent communication, much dialogue, FFP staff participate in task forces and on the Program Advisory Committee, and there is clear mutual respect and admiration. FFP staff in the field that have worked closely with the technical leads from the M&E and Commodity Management task forces are strong supporters of both, and admire how field driven, contextually appropriate, and core business focused trainings have been. Both of these task forces included key FFP field staff early in their planning processes, FFP staff were present for some or all of the training process, and FFP staff feel some responsibility for understanding whether or not transfer of training is occurring. Beyond these two instances, FFP field staff want more advanced notice regarding both training and knowledge sharing events that TOPS plans for particular countries. It is important to note here that FFP field staff wanted more advanced notice of events, more opportunity to contribute to shaping agendas, not in the name of control but, rather, for the purpose of contextualization in particular operating environments. An important element in this relative gap in TOPS’ effectiveness: while called for in the original project proposal, TOPS has not had a) a full-time deputy to the COP, nor b) three full-time staff who were to be placed in the field. There has been a rich, ongoing debate inside of TOPS regarding the overlap between TOPS and the Food Security and Nutrition Network. TOPS – as an FFP funded LWA program – is deliberately downplayed on the FSN Network portal. This was an explicit decision by TOPS management, and was done because it was deemed important for promoting the post-TOPS sustainability of the FSN Network. This decision has, however, left TOPS without a web footprint of any magnitude. Couple this with the fact that TOPS is nearly absent on the web sites of consortium members – only on the CORE group site is there substantial coverage or connections made – and this leaves TOPS in a communicative bind. One can imagine a wide number of things that would be useful to have on the web regarding TOPS – its annual plans, for example, or its annual reports – and might assist in small ways with informing stakeholders about TOPS, but have not been deemed appropriate for the FSN Network portal. 24 Opportunities To Improve and Specific Recommendations Opportunities Specific Recommendation Free up the Chief of Party to make more field visits, more voice-to-voice contacts, and to generally build stronger social capital with FFP field staff and to communicate in a concise and strategic manner TOPS’ priorities Use the recruitment of a Small Grants Director to enhance the role of Deputy Chief of Party. In other words: make the new Small Grants Program Director to Deputy Chief of Party and delegate activities currently being performed by the COP. This delegation will free the COP up for more interaction with decision makers in the field. Make TOPS planning and reporting more tailored to the needs of stakeholders Use unspent resources from personnel not hired to bring on board a communications specialist. The specialist would take TOPS internal planning/reporting material, and produce different products from them for different audiences. The specialist would also play a role in developing a communications strategy for engaging with Title II COPS and key FFP mission staff (with the appropriate and necessary collaboration with FFP DC). The specialist would also be charged with helping to carve out a clearer TOPS footprint on the existing FSN Network portal. Get TOPS’ main annual priorities, deliverables, and how these will benefit Title II implementers and FFP field staff out into the stakeholder community in more concise, communicative, and high-level ways. TOPS’ COP (and Deputy COP) need to have more voice-to-voice conversations with FFP field staff. These need to be short – no more than 30 minutes – and focused on decisions/actions, not chit￾chat. FFP DC staff clearly have an important role to play in paving the way for these conversations. Create a TOPS web site. 4.7 Save the Children’s Consortium Management “Consortium management” and “TOPS management” are analytical categories that can and do bleed into one another. For purposes of this evaluation, “consortium management” was understood in this evaluation to include: 1. Management of institutional relationships between consortium members on issues that outstrip the pay grade of the Chief of Party 2. Leading the promotion and positioning of the consortium in relation to Associate Awards 3. Driving strategic thinking – double-loop learning – about the nature, purpose, role, and value add of the consortium, and how the consortium might more smartly position itself vis-à-vis trends, other actors and programs, shifting politics and policies, and opportunities With regard to #1: little has been done. It is, in part, a result of very good internal TOPS management: there have been no conflicts that really required such escalation of involvement from very senior staff. But if we look at #1 as being as much about taking advantage of the TOPS opportunity for larger purposes beyond the TOPS contractual obligations themselves – the over-riding concern of the Chief of Party and TOPS staff, after all – the lack of activity in this domain seems a wasted opportunity. With regard to #2: our data are very mixed on whether or not the opportunity costs of an aggressive, strategic push to unlock associate awards are worth it. While it seems true that SC could have done more to promote, market, and seek to spread the value of such awards, in the particular context of this LWA in this context, the data suggest this would have been largely wasted effort. An alternative strategic focus for augmenting TOPS’ influence, relevance, member resources, and all this to the benefit of both PVOs and FFP seems evident. TOPS’ Chief of Party proactively organized a “Program Advisory Committee” (PAC) early in the project. The PAC consists of senior staff from all consortium members – usually at the VP level – as well as staff from collaborating partners such as CRS, Land o’ Lakes, and CARE. The PAC was established with explicit aims related to consortium management roles #1 and #3 above: “The PAC is a consultative group of major practitioners in food security and nutrition programming that guides and supports TOPS in considering strategic and tactical issues for program implementation… A crucial intent for the PAC was to provide a “portal” for 25 implementing PVOs that were not members of the original Save the Children-led consortium to welcome them into the TOPS process and to enable them to participate fully.” “TOPS established the PAC…as a strategic “think tank”….TOPS encourages food security managers from all FFP implementing partner PVOs to participate. The PAC meets twice a year. The TOPS Director frequently contacts members of the PAC…to confer on programming and strategic issues…. In the recent past, the…PAC has advised TOPS on…the growing resilience agenda and in the design of the Small Grants Program.”10 The PAC meets infrequently yet regularly, and most PAC members describe the meetings as more focused on updates on progress by the COP, information sharing, with modest efforts to shape TOPS activities or priorities by PAC members. These latter efforts – according to many PAC members – lack concrete follow-up and they declare themselves unsure what happens to them. Nearly all PAC members said they wished the PAC were a site of more high-level, strategic discussion about TOPS purpose, ability to influence USG priorities beyond FFP, and priorities. TOPS is – as mentioned above – relatively absent from consortium members’ web sites. There are many reasons for this, and each member has its own, unique set of reasons. It is the opinion of the evaluation team that the strategic value of TOPS – or potential strategic value – is simply unknown or subject to debate inside each consortium member’s senior management team. A careful, robust, and ongoing dialogue about TOPS’ strategic positioning – both within the wider food security and nutrition community domestically and globally, as well as within the US PVO Title I community – in the PAC seems needed. PAC members may well help each other consider and reconsider the role and value of TOPS to their home organizations, and how this unique collaboration can be better leveraged to have impacts on the world’s food insecure. The PAC is the exact right place and space for this discussion. TOPS staff themselves – while not in any way resistant to such transformational – or potentially transformational – dialogue rightly say their main focus is not in rethinking, repositioning, or leveraging TOPS. Their task is to implement TOPS, as agreed. It seems to the evaluation team that senior-most staff of consortium members – those senior leaders who supervise TOPS technical leads – have been curiously passive in this regard. While wishing to engage in the PAC in more strategic dialogue about how to leverage TOPS, they have waited for the current Chief of Party to drive, facilitate, and promote the process. Opportunity There is an important opportunity at hand, which is to build on the original intent and shift the mental model of the PAC from inward-facing inclusion (of PVOs that did not win the TOPS CA) – mission accomplished already – and single loop learning (asking the PAC to provide input on agreed goals/activities/ objectives) to outward-facing leverage and double loop learning. The graphic on the next page summarizes the differences between single-loop and double-loop learning. 10 “Technical and Operational Performance Support Program (TOPS), Year 2 Annual Progress Report, Reporting Period: 1 October 2011 – 30 September 2012,” pp. 11-12. “I think [the PAC] has been a very good forum…. It does give us very good access to TOPS top management, we’re kept in the loop. We can discuss, and we’re listened to….” PAC member “It’s a role that could be strategic and could be vital…but it’s not right now. It’s been under￾utilized. I don’t want to see it go away” PAC member 26 11 Opportunities To Improve and Specific Recommendations Opportunities Specific Recommendations Transform the PAC into a space of strategic dialogue about how to leverage TOPS Conduct run-of-the-mill updates over email, not during PAC meetings Make the PAC more a venue for thinking about TOPS positioning in the wider universe of USAID strategies, and how PVOs in particular can better leverage TOPS for their own missions but – ultimately – for the benefit of the world’s food insecure. At the next PAC meeting, draft a new statement of purpose for the group. Best would be to have this meeting externally facilitated by a consultant experienced in such products. Allow six hours for this, including in that time for discussions among PAC members – face to face – about the strategic opportunities to leverage TOPS that actually exist. 4.8 TOPS Strategy and Long-Term Sustainability What part of TOPS should be sustainable without any additional support from Food for Peace? In other words: what should TOPS be building now, in the expectation that funding from FFP is not forever? What part of TOPS should be sustainable without ongoing support from consortium members? In other words, what should TOPS be building now, in the expectation that the current consortium might well change sometime in the distant future? What part of TOPS should be considered “one-off” – i.e., we have no need to keep doing the activities, or producing the deliverables, after we’ve done them once – and which outputs or outcomes require ongoing implementation, into the mid- or long-term future? What part of TOPS – currently 100 percent supported by FFP – might be supportable through contributory funding streams, of which FFP might be one that is reduced but ongoing? At the heart of TOPS is an important hypothesis: the needs and wants of all Title II implementing agencies, when it comes to capacity building in technical areas, can be met best through a shared, collaborative structure rather than through supporting each agency individually. TOPS is, after all, built on the lessons of that latter approach (embodied in ICB grants). At the heart of TOPS is a second important hypothesis: that there is substantial implicit and explicit knowledge among PVO staff, and that providing ways to capture both will lead to better programs, and so greater impact. There’s a third hypothesis that may seem obvious, but is actually quite complicated: that trained PVO staff will do better programs, and that the (relative) handful of staff who get TOPS training can make a big enough difference in their projects to lead to higher impacts for the world’s food insecure. Finally, at the heart of TOPS is a fourth important hypothesis: PVOs – as institutional actors and not as individual staff – want to learn from one another. The truth or falsity of all four hypotheses will affect plans for sustainability. These hypotheses are in the process of being tested in TOPS, so conclusions are not possible right now. Yet, we know some few things quite clearly, in the absence of support or refutation of these hypotheses: PVOs experience substantial turnover rates, and so training 11 http://www.afs.org/blog/icl/?p=2653 accessed July 26, 2013. 27 never goes away as a need. PVOs have a terribly hard time getting donors to support staff development. PVOs cannot pay the salaries that for-profit firms can pay to hire staff who do not need training. PVOs are short-staffed and overworked, making efforts to share lessons, learning, and practices among themselves – in a way that others can act on – is a low(er) priority. PVO staff value knowledge sharing, but incentives are not aligned with this. PVOs are – at least sometimes – in competition with one another, know it, and need to keep something back as a comparative advantage vis-à-vis donors. In light of these (as yet unproven) hypotheses, and some of the institutional realities outlined in the previous paragraph, TOPS has already made some important moves regarding longer-term sustainability. PVOs cost share for training, and for participation in regional knowledge sharing events. This shows that – at least to date – a wide variety of organizations value these venues and are willing to pay something for their existence. This is allowing FFP to leverage resources outside of its investment in TOPS in an important way. All task forces are seeking to – as quickly as possible – stand alone without the need for leadership from a TOPS consortium member technical lead (who is being paid by TOPS). TOPS’ strong emphasis on tools – “explicit knowledge” – and tools that are quite self-explanatory (i.e., can be used off the shelf, without expensive training) is also an important strategy for sustainable impacts of the project: while TOPS can end, and the consortium disperse, the tools live on and can be used indefinitely into the future. Opportunities to Improve and Specific Recommendations Opportunities Specific Recommendations Think more divergently about what FSN Network and TOPS is, or could be, as an organizational structure. Investigate the possibility of turning the FSN Network into a hybrid organization, one that combines nonprofit and for￾profit business models. Develop a formal, draft, hybrid business plan to identify what could become a profit stream, and what should and must remain nonprofit/charitable. Make the contribution of other actors to TOPS more explicit Include participants’ financial support for training, knowledge sharing, and other TOPS and/or FSN Network activities in a revised M&E plan. 28 ANNEX 1: Evaluation Scope of Work12 The Consultant has agreed to: assist USAID’s Office of Food for Peace (FFP), Save the Children (SC) and the TOPS consortium members in conducting a midterm evaluation of the Technical and Operational Performance Support (TOPS) program. The evaluation will provide an opportunity to the FFP, and TOPS program management team to determine what strategies are functioning effectively, identify contributions of TOPS in improving knowledge and skills of food security practitioners and providing them with effective tools; make any midcourse corrections in program strategy and address implementation issues that will assist greater effectiveness and sustainability. The primary focus of the evaluation is to review implementation strategies and processes, identify the elements of the successful strategies that would make them effective, provide input on creative ways to respond to challenges and suggests options for areas needing improvement and document lessons learned. This service shall include, but is not limited to: all components of the midterm evaluation, including team building, interviews, data validation meetings, design of the research instruments, integration of other members’ evaluations, and compilation of the final report.  Develop an overall participatory framework for the evaluation, including the indicators of program progress, both in terms of outputs and in terms of the program implementation process;  Provide team members with the necessary orientation to create a common sense of mission;  Provide a point of contact for the TOPS Program Director and the Evaluation Team. Conduct planning meetings with the TOPS program management team to coordinate staff involvement;  Share detailed plan with TOPS Management and make necessary revisions based on mutual consent;  Create a program of data collection / interviews for each members of the team, including lists of people to be interviewed, the type of interview, and the method of reporting on each interview, then work with the TOPS PMT to provide necessary logistical support to the fieldwork activity;  Create a sampling framework for the range of stakeholders and beneficiaries (participants of training, knowledge sharing meeting, task force members) to be included in the evaluation;  Establish a strategy for involving all institutional partners (primary consortium members, USAID partners, and other partners) in the evaluation;  Meet in situ with the team to review the interview data and the results of the fieldwork;  Create an outline for the evaluation report and coordinate the writing responsibilities of individual team members;  Organize a ground-truthing workshop with TOPS PMT members to validate the initial observations and interpretation.  Compile and prepare the final draft report for discussion with team members, TOPS and the consortium members;  Revise and submit the final report (see Schedule B for components to include in the report). In addition, the Team Leader will assume responsibility for ensuring that the following questions are answered during the evaluation. S/he may delegate the responsibility to an appropriate team member(s). 12 After agreeing on this SOW, TOPS Chief of Party agreed to supplement the Lead Evaluator with a devoted external evaluator who looked at the small grants program, and a survey research firm to develop, administer, and analyze three online surveys of TOPS participants. Those SOWs are not included here. 29 1. What progress has been made to date in TOPS’s four Program Results areas and how might plans be improved for the rest of the project? 2. How effective are the capacity strengthening workshops/ trainings provided by TOPS program in core technical areas and how might this service be improved? 3. How effective are the strategies used by TOPS to strengthen the Food Security and Nutrition Network and what can be done to improve them? 4. How effective is the small grants program, and how can it be improved for the rest of the project? 5. What has TOPS done outside the scope of the RFA and agreed project proposal, how effective have these emergent priorities been implemented, and what might be done to permit TOPS to absorb and respond to unforeseen priorities in the future? 6. How effective is the management of TOPS and how can this be improved over the rest of the project? 7. How sustainable is TOPS in its current form, and what can be changed to enhance its long-term viability? Calendar of Activities Action Team leader Documentary review 3 days One-Day evaluation planning workshop in DC with Core Team (and possibly others TBD) 4 days Individual, in-depth stakeholder interviews 12 days On-line survey creation (two separate surveys needed), implementation, data analysis (including social network analysis) 2 days Mid-term check-in meeting (1 day) with Core Team in DC to engage in preliminary data analysis and to alter the evaluation’s action plan, if needed 3 days Data analysis for final Core Team workshop 2 days Core Team meeting in DC to agree on findings and future improvements 3 days Final report writing by lead evaluator 2 days 31 days 30 ANNEX 2: Evaluation calendar June 24- 30 July 1- 7 July 8- 14 July 15-21 July 22-28 July 29- Aug 2 August 5-11 August 20-28 Sept 2 Finalizing Evaluation team, evaluation strategy, methods, data gathering tools Key informant interviews for entire evaluation Key informant interviews for Small Grants Online surveys Focus Group discussion on emerging evaluation findings with PMT Full draft evaluation report Draft report discussion with PMT Feedback on draft from PMT, PAC, consortium senior management, FFP Final Evaluation report submitted 31 ANNEX 3 TOPS Monitoring and Evaluation Framework13 13 May 2013 revision. 32 14 While it is recognized that this is an output indicator (i.e., more under the control of TOPS), the cost of measuring outcomes of such small grants would be prohibitive. Table 1: TOPS INDICATORS SUMMARY (Outcome and Impact indicators Only) Program Strategic Objective: Highest quality information, knowledge, and best practices for improved methodologies in Title II food aid commodity program performance identified, established, shared, and adapted 1 Number of tools in each phase (identification, recommendation, share, adaptation) 2 Percent of Title II implementers scoring 70% or higher in post-test assessment on key knowledge, skills and practices for improved program performance. 3 Percent of FSN Task Force recommended tools incorporated into FFP guidelines, information bulletins 4 Percent of small grant (PIA) projects (US$100,000 or under) that meet their planned objectives 5 Number of organizations served by micro grants14 PROGRAM RESULT 1: Knowledge Capture: Knowledge and skill needs of audiences identified 1a An effective system in place for periodic knowledge and skills self-assessment of Title II functional areas by Title II implementing organizations IPR 1.1: Skill levels of food security practitioners assessed through participatory and external approaches 1.1.1 Percent of Title II core competency areas defined by the task forces IPR 1.2:Relevant and appropriate tools identified 1.2.1 Number of tools relevant to food security programs identified PROGRAM RESULT 2: Knowledge Generation: Reliable, high-quality information synthesized and produced in user-friendly appropriate formats 2a Average score on user perception survey of appropriateness, efficacy and satisfaction in use of FSN TF recommended tools IPR 2.1:Existing user-friendly, effective tools identified and endorsed by the FSN Network 2.1.1 Number of effective and user friendly existing tools endorsed by the task force IPR 2.2: Tools adapted (created to fill gaps or modified) for Title II grantee use in user-friendly and appropriate formats 2.2.1 Number of existing tools modified or new tools created and endorsed IPR 2.3: Packaged information on promising practices and approaches generated 2.3.1 Number of packaged approaches and best practices generated and disseminated IPR 2.4: Knowledge and tools generated through the TOPS small grants and fed into the dissemination channels 2.4.1 Number of tools created, field tested or validated using small grants 33 PROGRAM RESULT 3: Knowledge Application: Effective and appropriate traditional and non-traditional skill delivery approaches and systems/applications used 3a Percent of Title II implementers aware of at least 50% of key FSN TF-recommended tools 3b Percent of projects reporting use of FSN TF-recommended tools 3c Average score (normalized) on post-training satisfaction survey PROGRAM RESULT 4: Knowledge Sharing: Information, skill and knowledge exchange forums supported and expanded 4a Number of members regularly participating in FSN Network technical task forces 4b Number of participants attending FSN Network technical knowledge sharing meetings 4c Average score (normalized) on post FSN knowledge sharing meeting satisfaction survey 4d Average number of members participating in FSN Network listserv and discussion groups 4e Number of downloads from the FSN website IPR 4.1: Physical and virtual community exchange forums and knowledge sharing mechanisms supported and expanded 4.1.3 Number of hits on FSN Network website IPR 4.2: Physical and virtual community exchange forums and knowledge sharing mechanisms supported and expanded 4.2.1 Number of regional interest groups actively sharing knowledge on key implementation issues 34 ANNEX 4: Results from online surveys of portal users, trainees, and knowledge sharing event participants Survey Demographics Language Selection: Survey: English French Spanish All 178 (86.0%) 29 (14.0%) 0 (0%) Portal 91 (94.8%) 5 (5.2%) 0 (0%) Training 36 (64.3%) 20 (35.7%) 0 (0%) Knowledge Sharing 51 (92.7%) 4 (7.2%) 0 (0%) Type of Organization Respondent Works In Type of Org All Portal KS Training PVO/NGO 139 (81.3%) 54 (83.0%) 34 (69.4%) 18 (100%) Academic/ Research/ Think Tank 5 (2.9%) 4 (6.2%) 1 (2.0%) 0 (0%) United Nations 6 (3.5%) 1 (1.5%) 2 (4.1%) 0 (0%) Donor 7 (4%) 2 (3.0%) 5(10.2%) 0 (0%) Private Sector 5 (2.9%) 4 (6.2%) 1 (2.0%) 0 (0%) Other 9 (5.2%) 0 (0%) 6 (12.2%) 0 (0%) Country of Work UNITED STATES 32.4% WEST AFRICA 10.7% EAST AFRICA 23.1% SOUTH ASIA 11.2% CARIBBEAN 1.8% CENTRAL AMERICA 1.2% SOUTH & CENTRAL AFRICA 14.8% NORTH AFRICA (ME) 3.0% 35 Trainee Responses 1. Training Session (n=52) # M&E, Aug 2011, Liberia 2 M&E, Sept 2011, Mozambique 7 Qualitative M&E, Feb 2012, Burkina Faso 8 Mid-Term Prep April 2012, Bangladesh 7 Qualitative M&E, April 2012, Bangladesh 7 M&E, June 2012, Ethiopia 6 Quantitative M&E, March 2013, Niger 2 DBC, September 2011, Niger 4 DBC, January 2012, Burundi 5 DBC 2012, Ethiopia 5 DBC, July 2012 Mozambique 2 Nutrition, Sept 2011, Mozambique 2 Nutrition, April 2012, Bangladesh 3 Nutrition, June 2012, Ethiopia 5 Agriculture, Sept 2011, Mozambique 1 Agriculture, June 2012, Ethiopia 5 RELEVANCY OF TRAINING TO THEIR JOB (49) ALMOST NONE 2% A LITTLE 0% SOME 8% A LOT 55% A VERY LARGE AMOUNT 35% CONTENT APPLICATION AFTER RETURN TO WORK (49) ALMOST NONE 0% A LITTLE 8% SOME 29% A LOT 55% A VERY LARGE AMOUNT 8% STRENGTHS OF THE TRAININGS (47) RELEVANT AND QUALITY MATERIAL 28% EFFECTIVE FACILITATION 23% INCLUSION OF M&E AND DATA ANALYSIS 18% DIVERSE PARTICIPANTS 15% PRACTICAL TOOLS 9% HANDS-ON INSTRUCTION 4% PRE AND POST-TRAINING SESSIONS 4% SUGGESTED IMPROVEMENTS FOR THE TRAININGS (41) NEED MORE TIME TO COVER DESIGNATED TOPICS/TOO CONDENSED 21% INCREASED FOCUS ON THE FIELD 21% MORE DATA MANAGEMENT AND M&E FOCUS 16% FOLLOW-UP SESSIONS/CHECK-INS (POST-TRAINING) 12% PARTICIPANTS WITH SIMILAR EXPERIENCE LEVEL 9% INTEGRATION OF MORE MULTI-MEDIA 6% MAKE MORE AFFORDABLE 4% IMPROVING AND EXPANDING NON-ENGLISH TOOLS AND RESOURCES 4% BETTER TOPS-PARTICIPANT COORDINATION 3% CERTIFICATION OPTION/TOPS TRAINING MANUAL 2% - The percentages here reflect the percent of responses (each individual gets three) that related to the specific category - Interesting Idea from Suggested Improvements: Online and video trainings Notes: - 56 started the survey (19% response rate) - 52 continued past first question - Numbers in parenthesis are the number of respondents per question - Open-ended questions answered in all languages are included in this summary 36 Knowledge Sharing Event Participants 3. Knowledge Sharing Event (45) # Baltimore 2011 6 Mozambique 2011 7 Washington, DC 2012 11 Ethiopia 2012 7 I did not attend any of these events 19 CONTENT APPLICATION AFTER RETURNING TO WORK (27) VERY LITTLE 22% A LITTLE 11% SOME 33% A LOT 33% A VERY LOT 0% PERCEIVED VALUE OF THE KNOWLEDGE SHARING EVENTS (22) Not valuable Somewhat valuable Valuable Very Valuable BALTIMORE 2011 0% 33% 33% 33% MOZAMBIQUE 2011 0% 0% 33% 67% WASHINGTON, DC 2012 8% 8% 42% 42% ETHIOPIA 2012 0% 0% 100% 0% AVERAGE 2% 10% 52% 35% Notes: - 55 started the survey (20% response rate) - 45 continued past first question - Numbers in parenthesis are the number of respondents per question. 37 Strengths and Improvements STRENGTHS OF THE KNOWLEDGE SHARING EVENTS (19) PRESENTATION OF PRACTICAL TOOLS AND INFORMATION 29% INTER-ORGANIZATIONAL EXCHANGE 25% NETWORKING 18% DISCUSSION GROUPS 10% BUILDING CAPACITY OF PARTICIPANTS 8% EVENTS IMPROVE OVER TIME 6% MATERIALS AVAILABLE ONLINE POST-EVENT 4% SUGGESTED IMPROVEMENTS FOR THE EVENTS (17) FOCUS ON FEWER TOPICS 19% INCREASE EVENT-LEVEL PARTICIPATION OPPORTUNITIES 19% FOLLOW-UP EVENTS AND INFORMATION ACCESS (POST-EVENT) 16% INCREASE INTERNATIONAL AND DIVERSE ATTENDENCE 16% PRE- AND POST-EVENT TRAININGS 10% TRANSLATE RESOURCES INTO NON-ENGLISH LANGUAGES 10% DECREASE FOCUS ON PROCESS 3% INTEGRATION OF QUANTITATIVE SKILLS 3% BETTER COMMUNICATION BETWEEN TOPS AND PARTICIPANTS 3% - The percentages here reflect the percent of responses that related to the specific category 38 Portal Users REASONS FOR VISITING THE PORTAL (64) RESOURCE LIBRARY 73.1% FSN NETWORK UPDATES 21.3% FSN NETWORK DISCUSSION EVENTS 52.5% ONLINE DISCUSSION GROUPS 68.9% EXTERNAL LINKS 42.6% PERCEIVED VALUE OF PRIMARY ELEMENTS OF THE FSN PORTAL IN COMPARISON TO OTHER ONLINE RESOURCES (47) Not very useful A little useful Useful Very useful I don't know/have no opinion RESOURCE LIBRARY 2% 10% 27% 24% 8% EXTERNAL LINKS 5% 7% 35% 31% 22% FSN NETWORK DISCUSSION EVENTS 2% 10% 32% 46% 10% ONLINE DISCUSSION GROUPS 2% 15% 30% 42% 12% FSN NETWORK UPDATES 3% 10% 27% 52% 8% AVERAGE 3% 10% 30% 45% 12% TOP 10 OUTSIDE ONLINE RESOURCES IDENTIFIED (45) ONLINE RESOURCE Mentions 1. USAID SPRING, MCHIP AND WIKIS 14 2. FAO - SECURE NUTRITION PLATFORM 13 3. CORE GROUP 5 4. AGRILINKS 5 5. IFPRI FOOD SECURITY PORTAL 5 6. UN STANDING COMMITTEE ON NUTRITION 4 7. FANTA 4 8. FEWS NET 3 9. GLOBAL FORUM ON FOOD SECURITY AND NUTRITION 3 10. CARE GROUPS 3 HOW OFTEN THEY ACCESS THE PORTAL (65) NEARLY EVERY DAY 10.8% A TIME OR TWO EACH WEEK 32.3% A FEW TIMES A MONTH 36.9% RARELY ACCESS THE WEBSITE 20.0% NEWSLETTER SUBSCRIPTION RATES (63) SUBSCRIBE 60.3% DON'T SUBSCRIBE 23.8% UNSURE 15.9% 39 Strengths and Improvements STRENGTHS OF THE FSN PORTAL (50) TOOLS ARE CONCISE, TIMELY AND RELEVANT 43% USER-FRIENDLY 20% GLOBAL COLLABORATION CAPABILITY 11% TOPICAL FOCUS 7% TASK FORCES AND INTEREST GROUPS 7% NEWSLETTER 6% RESOURCE DOCUMENTS 6% SUGGESTIONS FOR IMPROVEMENT REGARDING THE FSN PORTAL (32) IMPROVE NAVIGATION FUNCTIONS 30% MORE INTEGRATION OF FIELD LEVEL ISSUES AND EXPERIENCE 21% UPDATE RESOURCE LIBRARY MORE OFTEN 21% MORE WEBINARS AND EVENTS 12% TOO MANY EMAIL NOTIFICATIONS 5% ABOUT US SECTION 5% STRENGTHEN TASK FORCES 2% PORTAL FOR PROPOSAL DEVELOPMENT 2% ALLOW RESPONSES TO THE RESOURCES POSTED 2% SUGGESTED IMPROVEMENTS FOR THE NEWSLETTER (9) INCLUDE CASE STUDIES 57% MORE VARIETY OF NEWS SOURCES 29% INCLUDE NON-FSN TRAININGS AND EVENTS 14% STRENGTHS OF THE NEWSLETTER (27) RELEVANT AND TIMELY CONTENT 43% CONCISE AND EASY TO READ 21% FREQUENCY 18% EVENT UPDATES 9% JOB POSTS 5% ALLOWS FOR COLLABORATION 2% REMINDER TO VISIT PORTAL 2% - The percentages here reflect the percent of responses (each individual gets three) that related to the specific category 40 Small Grants Respondents Have you heard of the TOPS small grants program? [Response Rate: 151 (72.9%)] Did you apply? [Response Rate: 104 (50.2%)] Why not? (*Multiple Response) Aggregate I didn’t have an idea that qualified for funding 18 I didn’t know how to apply 17 I don’t know why I did not apply 13 My organization in ineligible to apply 12 The application process was too burdensome 8 I don’t have the authority to apply 8 Timing not right; planning to apply next time 5 Other: 8 *Percentages not calculated due b/c multiple answers allowed. Other Response Summary (Aggregate): 4 respondent indicated they do not have the resources/time to apply for/ execute the program. 2 respondents indicated they do not have a current need to apply/ see it as a priority. 1 respondent indicated he/she missed the call to apply. 1 respondent indicated he/she considered applying but did not. . AGGREGATE PORTAL KS TRAINING YES 104 (68.9%) 43 (70.5%) 27 (65.9%) 34 (69.4%) NO 47 (31.1%) 18 (29.5%) 14 (34.1%) 15 (30.6%) Aggregate Portal KS Training Yes 18 (17.3%) 9 (20.9%) 7 (25.9%) 2 (5.8%) No 79 (76.0%) 33 (76.6%) 15 (55.5%) 31 (91.2%) I don’t know 7 (6.7%) 1 (2.3%) 5 (18.5%) 1 (3%) 41 ANNEX 5: A Management Dashboard for TOPS 42 15 Of which 6 reach “recommended” stage, 5 adaptation. SBC and M&E TFs responsible for all of these. 16 End of year three is September 20, 2013. Program Strategic Objective: Highest quality information, knowledge, and best practices for improved methodologies in Title II food aid commodity program performance identified, established, shared, and adapted Planned Accomplis hed Rating ACTION REQUIRED NOW 1 Number of tools in each phase 33 (Yr 2 cum) 3615 (Yr 2 cum) 2 Percent trainees scoring 70% or higher in post-test assessment 60% 87% (year two) 3 % recommended tools incorporated into FFP guidelines, information bulletins 20% (Yr 2) 64% 4 Percent PIA projects that meet objectives N/A N/A 5 Number of organizations served by micro grants 15 (Yr 2) 69 PROGRAM RESULT 1: Knowledge Capture: Knowledge and skill needs of audiences identified 1a. An effective system in place for periodic knowledge and skills self￾assessment of Title II functional areas by Title II implementing organizations 1 1 IPR 1.1: Skill levels of food security practitioners assessed through participatory and external approaches 1. 1. 1 Percent of Title II core competency areas defined by the task forces 80% (Yr 2) 80% IPR 1.2:Relevant and appropriate tools identified 1.2. 1 Number of pre-existing tools relevant to food security programs identified (and deemed worth pursuing more, or recommending) 32 (Yrs 1 and 2) 15 (yrs 1 and 2) PROGRAM RESULT 2: Knowledge Generation: Reliable, high-quality information synthesized and produced in user-friendly appropriate formats 2a Average score on user perception survey of appropriateness, efficacy and satisfaction in use of FSN TF recommended tools 70% (end yr 3)16 Unknown IPR 2.1:Existing user-friendly, effective tools identified and endorsed by the FSN Network 43 2.1. 1 Number of effective and user friendly existing tools endorsed by the task force as is 21 (yrs 1 and 2) 11 IPR 2.2: Tools adapted (created to fill gaps or modified) for Title II grantee use in user-friendly and appropriate formats 2.2. 1 Number of existing tools modified or new tools created and endorsed 6 (Yrs 1 and 2) 4 (Yrs 1 and 2) IPR 2.3: Packaged information on promising practices and approaches generated 2.3. 1 Number of packaged approaches and best practices generated and disseminated 7 (Yr 3) 0 (Yr 2) IPR 2.4: Knowledge and tools generated through the TOPS small grants and fed into the dissemination channels 2.4. 1 Number of tools created, field tested or validated using small grants 3 (Yr 3) 0 (Yr 2) PROGRAM RESULT 3: Knowledge Application: Effective and appropriate traditional and non-traditional skill delivery approaches and systems/applications used Percent of Title II implementers aware of at least 50% of key FSN TF￾recommended tools 50% (Yr 2) 73% (Yr 2) Percent of projects reporting use of FSN TF-recommended tools 40% (Yr 2) 52.3% (Yr2) Average score (normalized) on post￾training satisfaction survey 80% (Yr 2) 91.5% (Yr 2) PROGRAM RESULT 4: Knowledge Sharing: Information, skill and knowledge exchange forums supported and expanded 4a Number of members regularly participating in FSN Network technical task forces 70 (Yr 2 cum) 192 (Yr 2 cum) 4b Number of participants attending FSN Network technical knowledge sharing meetings 500 (Yr 2 cum) 533 (Yr 2 cum) 4c Average score (normalized) on post FSN knowledge sharing meeting satisfaction survey Yr 1: 80 Yr 2: 80 Yr 1: 74 Yr 2: 80 4d Average number of members participating in FSN Network listserv and discussion groups Yr 1: 500 Yr 2: 850 Yr 1: 750 Yr 2: 1100 4e Number of downloads from the FSN website 1000 (Yr 2) 2078 (Yr 2) IPR 4.1: Physical and virtual community exchange forums and knowledge sharing mechanisms supported and expanded 44 4.1. 3 Number of hits on FSN Network website 10,000 (yr 2) 20,734 (Yr 2) IPR 4.2: Physical and virtual community exchange forums and knowledge sharing mechanisms supported and expanded 4.2. 1 Number of regional interest groups actively sharing knowledge on key implementation issues 5 5 45 ANNEX 6 List of Interviewees NAME OF INTERVIEWEE AFFILIATION 1 Keith Adams FFP 2 Penny Anderson Mercy Core 3 Babylonia Aziz Counterpart International 4 Suzanne Berkey ACDI/VOCVA 5 Rebecca Bratter Joint Aid Management 6 Judy Canahuati FFP 7 Mark Castellino ADRA 8 Thoric Cederstrom Creative Associates 9 John Coonrod Consultant 10 Patrick Coonan CORE 11 Catherine Craig CPALI 12 Tom Davis Food for the Hungry 13 BK De SC 14 Mary DeCoster Food for the Hungry 15 Megan Deitchler FANTA (FHI360) 16 Dina Esposito FFP 17 Sasha Fisher Spark Microgrants 18 Tim Frankenberger TANGO 19 Mark Fritzler SC 20 Rebecca Freeman PCI 21 Michelle Gamber FFP 22 Matthew Gemeda MSH 23 Scott Hocklander FFP 24 Jay Jackson Guatemala Mercy Corps 25 Joan Jennings SC 26 Bonnie Kittle DBC 27 Richard Kondowe WVI Mozambique 28 Carolyn Kruger PCI 29 Karen Leban CORE 30 Paul Macek World Vision 31 Judiann McNulty Consultant 32 Mike McParland FFP 33 Dramane Meriko FFP 34 Bianca Morales-Egan PCI 35 Andrea Mottram Mercy Corps 36 Farai Muchiguel ADRA Mozambique 46 37 Altrena Mukuria Counterpart International 38 Gwenelyn O'Donnell PCI 39 Rana Olwan TOPS 40 Rashida Peterson OICI 41 Bridget Ralyea FFP 42 Arif Rashid TANGO 43 Adam Reinhart FFP 44 Alexandra Riboul FFP 45 Mara Russell Land o' Lakes 46 Kristi Tabaj SC 47 Melissa Teuber SC 48 Melanie Thurber FFP 49 Tesfaye Tilahun CRS Ethiopia 50 Karen Tincknell SC 51 Fitih Wedajeneh TOPS 52 Emily Wei CRS 53 Joan Whelan CORE 54 Jenn Williamson Counterpart