MID-TERM EVALUATION FOOD AID MANAGEMENT INSTITUTIONAL SUPPORT ASSISTANCE PROGRAM For the period covering October 1, 1998 – March 31, 2001 For CARE And The U. S. Agency for International Development Bureau for Humanitarian Response OFFICE OF FOOD FOR PEACE By John P. Mason, PhD. Consultant in collaboration with FAM Staff & Membership May 29, 2001 TABLE OF CONTENTS Page Chapter 1: Introduction 1 A. Background 1 B. Purpose 2 C. Methodology 2 D. Organization of the Report 4 Chapter 2: Key Findings – Strategic Role of FAM 5 A. Overall Progress 5 B. Strengthening Capacity in Design, Monitoring, and Implementation of Title II 8 C. Promoting Efficient and Effective Use of Food Resources 10 D. Building Institutional Capacity 10 E. Unifying Title II PVOs in Meeting Food-Aid Challenges 11 F. General Constraints 11 G. USAID/FFP Perspective 13 Chapter 3: Key Findings – Dissemination and Sharing of State-of-the-Art Food Aid/Security Practices 15 A. Working Groups 15 B. Workshops and Training Tools 20 C. Food Security Resource Center 22 D. Other Collaboration 27 Chapter 4: Key Findings – Organizational Impact 29 A. Coordinating Members 29 B. Institution Strengthening 30 C. FAM Management 31 E. Working Group Coordination 32 F. Sustainability Issues 33 Chapter 5: Conclusions, Lessons Learned, and Recommendations 35 A. Conclusions 35 B. Lessons Learned and Recommendations for FAM & FAM Members 36 Annexes 1. Scope of Work ii 2. Persons Contacted 3. Documents Reviewed 4. Questionnaires Utilized 5. Indicator Performance Tracking Table 6. FAM Collaborative Training Activities 7. FAM Members Contribution to Food Forum 8. Documents Produced by FAM under its ISA iii EXECUTIVE SUMMARY Undertaken in March-April, 2000, this mid-term evaluation of the Food Aid Management (FAM) Institutional Support Assistance Program agreement between CARE and USAID/Food for Peace (FFP), covered the period October 1, 1998 to March 31, 2001. Its purpose was to assess FAM’s progress in meeting its objectives of enhanced food-aid practices, knowledge base, and collaboration. FAM’s coordination of 17 PVOs, mostly Title II implementers, has resulted in improved food-aid practices, including monetization and pesticide guidance, regulations and frameworks for Title II PVO proposals, implementation plans, and results reporting. Dissemination of knowledge has been done through its website, library, Food Forum, four Working Groups, thematic workshops, technical tools, and other interactions with the membership. FAM offers economies of scale to USAID/FFP, driven by PVO demand for state-of-the-art knowledge. The present level of collaboration among the membership speaks well for FAM’s success in coordinating geographically and operationally diverse organizations. Simpler constraints are the usual time and money ones, while a more complex challenge is for FAM not to be ‘taken for granted’; to find a way to receive credit for its work. Diffuse functions and a facilitating vs. implementing role, alongside a less-than-favorable perception of FAM’s contribution by FFP, often results in others receiving credit for FAM’s accomplishments. FAM has contributed importantly to strengthening design, implementation, and monitoring of its members’ Title II programs. This is accomplished largely through FAM-supported working groups (WGs) in Monetization (MWG), Monitoring & Evaluation (M&E), Local Capacity Building (LCB), and Environment (E), which, in turn carry out workshops and design field tools for Title II implementers, as well as creating important mediums of exchange. While the MWG and EWG are a bit further ahead of the others in development of field applications, the M&E WG has two tools already developed and in use. The LCB WG has begun to develop its tools. The Food Security Resource Center (FSRC) is a second key to FAM’s achievements, contributing much of the knowledge base and providing continuous communications with the membership, especially the WGs. It is perceived in a most favorable way by its many users. Recommendations include: • FAM member PVOs should develop a framework for a plan and assign a committee to explore the continuation of FAM’s key functions into the future with the Steering Committee (SC). • The Steering Committee should call a meeting to openly explore FFP’s less-than￾favorable perception of FAM’s value with the membership, then the SC should meet with FFP to discuss how to ameliorate the situation. • FAM coordinators and local members of the SC should organize regular meetings with FFP to communicate openly, so as to ensure that FAM is fulfilling FFP’s needs. • FAM members, in cooperation with FAM staff, should develop an initiative to inform their staff and constituencies of the services available to them through FAM, so that they are able to recognize FAM’s contribution in terms of food aid/security standards development and information exchange resources. • WGs should agree at the outset on a product or a plan that will engage all members. Where feasible, WGs should be facilitated by technical experts on-staff with member organizations who also have team building skills, preferably. iv • The MWG, M&E and LCB WGs should consider use of the EWG’s TOT approach, where appropriate, for planning subsequent workshops. In a related matter, the SC should request formal recognition of the EWG from FFP. • The SC should support FAM coordinators in organizing resources to ensure completion of the online version of the FSRC Bibliographic Database. • The SC should speed up the search for resources (institutional and/or financial) to digitize the FSRC library. v Chapter 1 INTRODUCTION This report documents the mid-term evaluation of the Food Aid Management Institutional Support Assistance Program. The assessment was undertaken in Washington, D.C. from March 7 through April 13, 2001. The mechanism evaluated is an Institutional Support Assistance agreement made between the U.S. Agency for International Development, Office of Food for Peace and CARE, the grant holder. The agreement period covered by the evaluation is October 1, 1998 – March 31, 2001. A. Background Food Aid Management (FAM) was set up in 1989 by five U.S. private and voluntary organizations (PVOs) to “promote the efficient and effective use of food aid resources to help alleviate hunger and contribute to food security.” Beginning in September 1998, USAID and FAM worked out a five-year follow-on Institutional Support Assistance (ISA) agreement. That agreement was aimed at supporting the PVOs in their P.L. 480, Title II-funded food aid programs. Now 17 members strong, FAM facilitates the membership in fulfilling three objectives.1 The FAM initiative was created by Title II PVOs largely as a forum in which they could collaborate and exchange food aid/security program information. Specifically, its members saw and continue to see FAM as a uniquely valuable venue for exchange of new tools and best practices. Members’ commitment is evidenced in part in their provision to FAM of a portion of their own Institutional Support Assistance (ISA) grants (those who hold them), as well as their time and effort. Members also value FAM as a forum in which PVOs working in food aid/security and FFP can share information and be receptive to each others’ needs and perspectives. FAM’s objectives are to a) facilitate and promote the development of food aid standards; b) promote the food aid and food security knowledge base of PVOs through information exchange and coordination; and c) facilitate collaboration between PVOs, USAID, and appropriate development and humanitarian professionals by organizing discussion fora. In combination, the objectives contribute to a higher-level, strategic objective: namely that of the Office of Food for Peace (FFP). The specific objective is “increased effectiveness of FFP’s Partners in carrying out Title II development activities with measurable results related to food security with a primary focus on household nutrition and agricultural productivity.” The objectives also contribute to an ‘intermediate result,’ 1 FAM members include Adventist Development & Relief Agency, Africare, ACDI-VOCA, American Red Cross, CARE, Catholic Relief Services, Counterpart International, Feed the Children, Food for the Hungry International, International Relief & Development, Mercy Corps International, OIC International, Project Concern International, Save the Children Federation, TechnoServe, World SHARE, and World Vision. 1 which is strengthened capabilities of PVOs to design, monitor, and support food aid/security programs. B. Purpose The general purpose of this mid-term evaluation, derived from the Scope of Work (see Annex 1), is “to assess the progress in achieving planned results and to refine program activities and targets accordingly.” A related purpose is to determine if the FAM program is being implemented in a timely and cost-effective manner and whether performance is on track. More specific purposes and criteria used in guiding the evaluation are outlined later in the text of the report. C. Methodology Methods used in the evaluation include key informant interviews, a focus group discussion, open-ended questionnaires, site visits at FAM, and review of documents. A satisfactory degree of contact was achieved with the FAM membership and workshop participants, as seen in the table on the following page. Where the response rate is low, it was due mainly to non-working email addresses of international workshop participants. The low rate was not viewed as critical to the overall study, however, since statistical representativeness was not a priority. Persons interviewed are listed in Annex 2, while the 13 topical guides/questionnaires designed for the evaluation are included in Annex 4. A brief note about the “faces behind the numbers” in the following table. The grouping labeled General Membership comprises PVO home office staff from across the 17 member organizations, including new entries, technical specialists, and middle and senior managers. Working Group members are mostly mid- to senior-level technical specialists or managers. A mix of field technical staff and mid-senior level home office staff comprise the workshop participants. Focus group discussants included mid- to senior-level managers, who are also technically specialized. Senior managers with some support from backstop middle managers made up the pool of Steering Committee members. FANta advisors are both senior technical people, as are FAM Coordinators. FAM’s Technical Information Specialist, who manages the FSRC, was trained largely on-the-job under a mentoring agreement. Finally, USAID/FFP officials are both senior managers in that Office. 2 Instruments & Samples Established for Use in FAM Evaluation Instrument Sample Size # Responses % Response Key Informant Interview Steering Committee 7 6 86 FAM Staff/past-pres. 6 5 83 USAID/FFP Officials 2 2 100 FANta Rep. 2 2 100 Questionnaire – FAM Membership General Membership 18 9 50 Environmental Working Group 7 4 57 Monetization Working Group 8 5 63 M&E Working Group 8 5 63 Local Cap. Building Working Group 7 3 43 Questionnaire – Workshop Particip. Local Capacity Building 8 2* 25 M&E 8 2* 25 Monetization 9 4* 44 Environment 8 3* 38 Focus Group Working Group Representatives 4 3** 75 * Email address no longer extant/message returned “undelivered.” Time constraints precluded repeated tries to contact addressees or replacement with new respondents. ** 6 persons, representing 3 Working Groups Generally, key informant interviews and the focus group discussion generated highly usable data for assessing FAM’s progress. Electronically administered questionnaires were completed by respondents for the most part in a thoughtful manner. Where an individual was charged with responding to more than one questionnaire (even though the selection for all but one—Working Group Member questionnaires—was randomly done), a limited degree of respondent fatigue was evident. Review of documentation figured importantly in the evaluation, including the provision of hard copy reports, but mostly electronic versions of reports. The site visit referred to earlier was to the Food Security Resource Center in FAM’s office located at 1625 K Street in Washington, D.C. There, a review of the organization of a hard copy collection of about 8,000 documents and a demonstration of the FAM website, www.foodaid.org, were conducted. The FAM website also served as an important source of evaluation data for FAM-coordinated processes and products, including working group minutes and the online survey. (Documents reviewed are listed in Annex 3.) 3 In addition to the questionnaires distributed to workshop participants, workshop training data from pre- and post-test results, and post-workshop evaluations were used in assessing FAM’s input to skills and knowledge development. A brief note on the collaborative character of this evaluation is in order. FAM staff has had extensive input to this exercise at virtually every stage, and included the review of questionnaires sent to members and drafts of this report. FAM membership also participated in a review of the last draft of the report and made important points, many of which have subsequently been incorporated into the final report. While the evaluation process was not a full-blown participatory exercise, it nevertheless was a collaborative effort of the evaluator and the FAM staff and membership. Evaluative statements or judgments, nonetheless, should be attributed to the evaluator, not to the collaborators. D. Organization of Report The report is divided into five chapters, including the present one. Chapter 2 presents key findings on the strategic role of FAM, with sections on overall progress, constraints, fulfillment of Title II Food Aid requirements, and building institutional capacity. Chapter 3 presents key findings on dissemination and sharing of state-of-the-art food aid and food security practices. It will describe the contribution of the working groups, workshops and training tools, the Food Security Resource Center (FSRC), and other FAM member activities. Chapter 4 presents key findings on FAM’s organizational impact, including the coordination of PVO members, the synergy of FAM and individual member institutional strengthening, and FAM management. It also addresses FAM’s potential for sustaining its results. These are followed by the final chapter, Chapter 5, which presents conclusions, lessons learned and recommendations. These pertain to FAM itself, the FAM membership, and USAID/FFP’s role. 4 Chapter 2 KEY FINDINGS – STRATEGIC ROLE OF FAM FAM’s strategic role consists of two parts: one long-term and strategic, contributing to the food security goal to promote more effective use of food aid resources in development programs. The other, at a lower level of impact, is strengthening member PVO capacity in the design, monitoring, and support of food aid and food security programs. How far FAM has progressed in contributing to these objectives is the subject of this chapter. A. Overall Progress General progress in how effectively the ISA objectives have been met to date is tracked in part through use of FAM’s results framework. That framework is embodied in an ‘Indicator Performance Tracking Table’ (see Annex 5), which tracks measures of progress towards intended results. A detailed assessment of specific FAM program activities is presented in this and subsequent chapters. It corresponds to the expected results tracked over the five-year period of the FAM initiative, inventoried as a reference point for the reader, as follows: --Website increased usage, user-friendliness by field staff, and use in design of food security programs (in part through a mentoring partnership) --Successful workshops managed by FAM (no less than 10 over life of activity) --Monitoring & evaluation toolkit completed --Monetization manual, training modules, workshops completed (including improved skill base in monetizing food aid) --LCB Measurement of Capacity Building paper --Environmental principles integrated into food security programs --Increased knowledge of food security matters (no less than 25 meetings organized) --Improved collaboration of FAM members as a product of annual meetings, joint program initiatives, mentorship relationships, jointly developed tools & methods --Increased ‘autonomy’ of FAM and its membership in carrying out the above activities (i.e., ‘self-starting’ and a diminished role of FAM staff in managing the activity) Cumulative progress of the FAM Institutional Support Assistance (ISA) is presently assessed in terms of the three, earlier-mentioned FAM objectives. Facilitate and promote the development of food aid standards: Through FAM’s coordination of its PVO members, it has contributed to disseminating a significant 5 amount of state-of-the-art information on specific standards. It has done so directly through: 1. Cataloguing of about 8,000 documents available in the Food Security Resource Center (FSRC) 2. An expansive food aid website: http://www.foodaid.org 3. Food Forum quarterly publication 4. Support of Working Groups 5. Offering thematic workshops 6. Production of technical manuals 7. Coordination of discussion groups and the FAM Steering Committee 8. Constant interaction and communication with the membership FAM’s facilitation of its members has positively influenced them to adopt improved methods of planning, designing, implementing, and evaluating food aid/security programs. Cases in point are PVO members’ adoption of the following: • USAID/FFP Title II monetization policy, regulations, and practices that have been transmitted to and applied by PVO members in field operations • Common proposal frameworks for use by members in presenting their Development Activity Proposals and Detailed Implementation Plans to USAID/FFP • Standardized planning formats for results monitoring and evaluation, and annual reporting by members to USAID • USAID/FFP Regulation 216 on environmental review procedures that was transmitted and is presently being applied to Title II field operations Progress is also being made through the Monitoring & Evaluation Working Group in developing a food aid/security monitoring and evaluation toolkit. That group has just recently contracted a consultant to develop guidelines for program monitoring, including tools, methods and best practices. The Local Capacity Building (LCB) Working Group is presently beginning to develop an inventory of LCB indicators and tools. Promote the food aid and food security knowledge base of PVOs and other collaborators through information exchange and coordination: A growing knowledge base in food aid/security is a direct result of the FAM-managed FSRC library, website, and listservs. The website evidences growth in the number of users, both FAM and non-FAM. Precise numbers are included in Chapter 3, where the FSRC is reviewed in detail. The website also serves as an effective management-administrative tool for FAM in coordinating the program efforts of the Working Groups and informal groupings such as brownbag discussions. It includes updates on all Working Group activities, including meeting minutes, schedules for all FAM-supported events, such as workshops, Working Group meetings, brownbag discussions, and information on non-FAM events, particularly training offerings. 6 On another level, the website provides general membership, including home office and importantly, field staff and other food security professionals with an increasing number of state-of-the-art publications. Many of those publications are available on the website, for easy access by users, and include key PVO and USAID documents. Also included on the website are the latest tools developed with FAM support for program and field staff application to Title II programs. FAM member field staff unable to take a particular workshop course can access some of the course materials, such as manuals, from the FAM website. They can also access information on technical products being developed by Working Groups. Workshops offered under FAM sponsorship clearly represent an economy of scale for FAM members. During the grant agreement period, nine workshops have been delivered. Offering a venue where PVO home office and/or field staff are able to share the latest food aid/security technologies not only represents a cost saving, but also contributes to promotion of a “culture of food aid professionals” among the key organizations responsible for implementing Title II. The workshops also provide technical staff of PVOs with up-to-date technical information on compliance with Title II necessary for continued implementation of USAID/FFP programs. (Annex 6 includes a matrix of FAM-supported collaborative training activities.) Economies of scale cannot be underestimated in the context of focusing and concentrating resources in FAM by both PVOs and USAID/FFP alike. As the current Chair of the FAM Steering Committee noted in an interview, “the PVOs could not be doing all this by themselves, setting up the FSRC, the website, doing the workshops, publishing the manuals.” He noted, as well, “so long as member participation continues at the present level, there should be no problem keeping up FAM’s productivity.” He indicated he expected it would. Liaison activities with such food aid agencies as World Food Program (WFP) and EuronAid were initially part of the objective to promote a knowledge base. The prior Coordinator of FAM apparently attended a WFP meeting in Rome earlier on, but FFP indicated in FY 2000 that such attendance was no longer part of FAM’s mandate. Facilitate collaboration between PVOs, USAID, and appropriate development and humanitarian professionals by organizing fora for discussion: Informal groupings have been formed under the auspices of FAM Working Groups to exchange the latest information on different technical aspects of food security (see Annex 9 for working group participants lists). These are most often held as brownbag presentations/discussions among members and other partners, including regular participation of USAID officials in certain of the presentations. Brownbags represent an informal, though time- and cost-effective, mode of information exchange and capacity building. Other significant venues for collaboration among members are annual meetings and Steering Committee meetings. Annual meetings have included presentations on issues of common vital concern to the general membership as well as opportunities for large and 7 small group discussions on shared interests. Steering Committee meetings, in addition to providing critical leadership and counsel to FAM coordinators, have typically generated discussions on the key issues of the day in terms of both general food aid and specific USAID Title II-related issues. A close, open and professional relationship between FAM staff and that Committee was observed during the course of the mid-term evaluation. A comment by a Steering Committee member in response to an earlier draft of this report, captures perfectly the spirit of collaboration of FAM members. It is worth quoting in its entirety. The importance of FAM’s fostering of PVO collaboration goes far beyond the work of specific FAM projects or tasks or committees. While it is difficult to attribute and to track, I believe that one of the most significant impacts of this role of FAM in bringing us together has been the strengthening of the relations between individual PVOs in both field and HQ offices. Our greater ease in collaborating outside of FAM-coordinated activities is in great part a result of our getting to know and understand each other better through FAM, and our appreciation of what we can gain from working together. Therefore, while FAM participation is no longer voluntary for ISA￾recipients, I do believe that it would continue without the specific requirements of our awards. B. Strengthening Capacity in Design, Monitoring, and Implementation of Title II Programs Of the 17 members of FAM, 14 have their own Institutional Support Assistance (ISA) agreements with the Office of Food for Peace. While 12 PVOs have five-year agreements, two have three-year agreements. These are designed to enhance their capability to design, monitor and support their Development Activity Programs (DAPs), which are the implementing mechanisms for the provision of Title II development food and food aid programs. The same enhanced capability applies to members’ implementation of emergency relief-to-development transition programs, as well. FAM’s ISA, in turn, overarches these individual ISAs. Its intended result is a ‘value-added’ one whose impact we will see is greater than the sum of the individual agreements. FAM, FAM members, USAID/FFP, and certain USAID Missions all have the same objective for food aid and food security, which is the improved effectiveness of food aid programs as a development tool or as a tool to support the transition from relief to development. FAM, in cooperation with its members and other partners, has selected specific areas of food aid on which to focus, namely monetization, monitoring and evaluation, local capacity building, and the environment. In making its contribution to improved food aid methods, FAM has worked closely with a FFP contractor, Food and Nutrition Technical Assistance (FANta). To be clear in distinguishing the related roles of FAM and FANta, FAM’s clients are its PVO members; FANta’s client is USAID/FFP. FANta and FAM cooperate under a memorandum of understanding dating from 1998 to support PVOs in strengthening their technical capacity to implement Title II food aid programs. FANta renders both technical and financial assistance to FAM-supported activities, including costs of workshop trainers and other technical consultants. It is important, therefore, to recognize in what follows 8 that the outcomes of FAM’s effort in promoting improved methods of food aid also sometimes overlap with those of FANta. This reflects a synergy and economy of scale otherwise unobtainable were it not for the cooperative effort of FAM and FANta. FAM contributions to improved food aid methods fall under its objective to ‘facilitate and promote the development of food aid standards.’ These contributions include support of the following design, monitoring, and implementation elements of Title II Programs: • More streamlined, systematic process for preparation, submission, and review of Development Activity Proposals and Detailed Implementation Plans by member PVOs to USAID/FFP • Improved process and structure for annual results reporting to USAID/FFP (the reporting requirement is called the Cooperating Sponsor Results Reporting/Resource Request or ‘CSR4’) • Improved application and reinforcement in field operations of new Title II policy, regulatory, and technical requirements In concert with FANta, FAM has supported the proposal, reporting, and field applications processes that contribute to the overall streamlining of PVO participation in the Title II Program. It has also provided a venue in which USAID can interact on a systematic basis with all of the major implementers of Title II. In so doing, the FANta/FAM combined effort represents an economy of scale for USAID probably not otherwise achievable. Equally important, this effort has fostered a significant level of interaction among member PVOs which is channeled into meeting Title II requirements, and to dealing with related opportunities and constraints involved in creating meaningful, innovative food aid development programs. C. Promoting Efficient and Effective Use of Food Resources FAM members’ perceptions of the Program’s success in supporting them to improve design, implementation and monitoring of their activities are generally highly positive. These are mainly the perceptions of those who are PVO mid-senior level managers in home offices, mid-level program officers and managers in both the field and home offices, and field technicians who implement food programs. A sampling of their views of FAM’s progress in supporting their programs is instructive. Steering Committee members, in effect the board of directors of FAM, were unanimously positive about the role the Program has played in helping member PVOs to get and stay on the same screen, that is, the state-of-the-art food aid and food security screen. FAM’s part in coalescing different partner points of view and facilitating members to collaborate for purposes of meeting higher goals was seen by these members as paramount to FAM’s success. A former chair of that Committee suggested that “the focal point of FAM is sharing information and lessons learned with all the member PVOs towards the aim of doing better what we all already do quite well.” 9 Program managers and staff expressed a favorable view of FAM’s role, mainly because it brings PVO technical staff together to share state-of-the-art food aid practices. Those managers and staff had either served on a FAM Working Group or participated in a FAM-supported workshop. They pointed to improved ways of organizing their approach to food aid programs or specific improvement in the technical aspects of implementation. One example is the overall improved approach to monetizing food aid, which is seen as having resulted from members’ information sharing through workshops and design of new tools, including technical manuals. Reflecting on the importance of FAM-supported workshops, one participant in a monetization workshop noted that “the monetization manual exposed us to basic guidelines for Title II programs, including information on cost recovery benchmarks, the U.S. commodity procurement schedule, call-forward, and Bellmon analysis.” According to that same participant, another benefit derived from the workshop was familiarization with some of the potential risks of monetizing food aid commodities in the private sector, risks directly related to field implementation. D. Building Institutional Capacity The institutional strengthening mandate of FAM is implemented in the domain of improved technical capacity and not of organizational change. However, the capacity of these PVOs to adapt to and benefit from FAM itself is itself a sign of improved organizational capacity. Proactive participation of members in Working Groups and in the training functions under FAM are evidence of a capacity of these organizations to adapt and grow. Member organizations of FAM participate in a variety of ways, determined by each organization itself. The level and character of participation or contribution by members are shaped by such factors as organizational strategy and size, number of Title II programs, availability of home office staff who can devote effort to FAM activities, and other internal priorities. The level of member participation suggests that there is a ‘demand’ for FAM services, based on a perceived need of members to increase their learning potential in order to improve their contribution to food aid/security programs. A focus group discussion with members of three of the four FAM Working Groups pointed to a strong commitment to improve the state-of-the-art of their respective disciplines. Whether representing the environment, monetization, or local capacity building WG (monitoring and evaluation was not represented), a significant sense of willingness to cooperate among member PVOs to meet Title II mandates and deal with other program interests was observed. Participation in these groups is also voluntary, so member workloads, including travel, sometimes make consistent attendance at WG meetings difficult. Nevertheless, a review of the multiple products of these WGs, including workshops, discussion groups, agendas, and meeting minutes, indicates the positive results of their efforts in building capacity. 10 Besides demonstrating an almost passionate feeling about their work areas, some WG representatives in the focus group also suggested that they sometimes felt WG agendas were too heavily driven by USAID mandates. They were fully aware of the contradictory nature of this perspective, in that the raison d’etre of their participation in the WGs is to build their capacity to improve their Title II work. Nevertheless, there was a sense that they would like to pursue professional interests beyond the confines of Title II but they understood how difficult that would be. Active participation in FAM-supported activities alone represents a commitment many members say they have a difficult time meeting. Workshops are integral to the dissemination of state-of-the-art learning and development of a knowledge base. They are covered in detail in the next chapter. In the meantime, suffice it to say that workshop pre and post-test scores are generally indicative of skills building among FAM members. An example is the Data Analysis Workshop, in which there was a significant improvement in scores relating to statistical sampling and measurement techniques. The Mean pre-test score was 6.3, while the mean post-test score was 9.4, and most participants said they would use what they had learned to increase their understanding and analysis of their project data sets. An FY00 CSR4 also noted that as a result of this workshop, a Title II program manager created a data tracking and analysis system, initiated using FY00 data. E. Unifying Title II PVOs in Meeting Food Aid Challenges As suggested earlier, FAM provides a venue for increased interaction and sharing among those PVOs that implement Title II activities. That in itself, while necessary, is not sufficient to meet the objectives of the FAM Program. Furthermore, the ‘necessity’ of the PVO members to cooperate, while perhaps a ‘logical’ premise, is not compelling in and of itself. In fact, as one long-term Steering Committee member suggested in an interview, “PVOs are not naturally structured to collaborate with one another.” He meant that PVOs are occupied with their own internal requirements, as well as external, program-related exigencies, and, besides, historically, they have not had a great deal of collaboration with one another on a working level. So, if such collaboration may not be a natural predilection of PVOs, what then is the optimal institutional or organizational framework for achieving that result? Evidence from interviews and questionnaires administered for the evaluation points to a clear answer. The “need” for collaboration lies in the demand for state-of-the-art information relevant to successful implementation of Title II programs. Whether such collaboration takes the form of Working Groups, training sessions, brownbag discussions, shared use of the website or FSRC printed documents, there is a pent up demand for new theory, methods, approaches and techniques in food aid that drives the cooperative spirit so evident in FAM. 11 F. General Constraints The evaluation found that opportunities seem to far exceed constraints in FAM’s capacity to meets its three objectives and the strategic and intermediate targets of its partners. We have begun to see some of the opportunities already. Three major constraints emerge from the overall program review undertaken in this evaluation. We start with the simplest, most straightforward constraint. It is that proverbial one of never enough time and money. A large majority of key informant interviews and the one focus group discussion with members of FAM, including Steering Committee members, Coordinators, Working Groups, and the FSRC Technical Information Specialist (TIS), point to that constraint. Perceptions from interviews on these two subsets of the same constraint share much in common. For example, a member of a WG indicated, “we could do even better in our group if members could only give more time, but so many of us are always traveling.” This is directly related to the fact that much of the important work in food aid done by member PVOs is field-directed. Another member suggested, “fortunately we have the financial assistance of FANta, otherwise we couldn’t provide all the good workshop trainings.” In the domain of money, the FSRC TIS noted that FAM is not able to digitize the library materials under its present budget, and suggested that perhaps funding might be available from other sources. The second constraint is the difficulty FAM has had in getting credit for its accomplishments. This constraint is built into FAM’s mandate, which is to facilitate others, and is based as much on perception as fact. It relates to FAM’s ‘right’ to take credit for the work it promotes among member PVOs. It became abundantly clear during the course of the evaluation that PVOs would not normally undertake this work individually or in a group. This second constraint also relates to the role played by the aforementioned FANta. FANta’s role, as a contractor to FFP, is to work with the exact same Title II PVOs that make up FAM. FANta provides financial and technical assistance to Title II PVOs. Its particular arrangement with FAM, through FFP, allows it to fund some of the costs of planning and implementing workshops given under the auspices of FAM. Because of FANta’s contribution, it will often get credit for a workshop even though it was a FAM initiative. FANta also has the funding to disseminate its publications widely across USAID and in the food aid sector generally, lending it a certain profile probably unattainable by FAM. In this respect, one Steering Committee member interviewed suggested that it was not clear to him that even USAID/FFP understood the distinction between FAM’s and FANta’s contributions. He found at times that USAID/FFP officials gave credit to FANta for an event, probably because they knew FANta’s financial and technical inputs had been made. That Committee member continued, “not that FANta is in any way 12 undeserving of some credit, but remember that none of this could happen without the organizational structure and facilitation of FAM.” Another aspect of the same constraint occurs because sometimes even FAM members themselves, especially field staff, may be unaware of the role that FAM plays. To underscore the point, in an interview with a FANta representative, the suggestion was made that the value-added of FAM “has to come from a perception of the members themselves that they significantly improve their programs through FAM.” As if to accentuate the point, the same FANta representative noted that one FAM member that had sent its annual results report from the field directly to FFP, made no reference to either FAM or FANta support in its preparation. This person also cited a quote from a CSR4 from West Africa (mentioned above on page 11) in which the Data Analysis workshop was attributed to USAID. While this may reflect a disconnect between home and field office, it also underscores the need for FAM members to keep their field staff informed on the role of FAM. The third constraint also lies in the realm of perception. It is a perception of several of the Steering Committee and WG members interviewed that USAID/FFP does not recognize the full value of FAM. They were not sure exactly why, but thought it might be a reflection of the earlier point that FFP sees FANta as the key player in much of what, in fact, FAM and FANta perform together. Perhaps because the size and monetary value of FANta’s contract is significantly larger than FAM’s, FAM may be easier to overlook. G. USAID/FFP Perspective An interview with officials of USAID/FFP clarified some Steering Committee perceptions of that office. FFP perceptions of FAM can be depicted in terms best characterized as a communications gap between FFP and FAM staff and members. One example of such a gap was the response to a question on what FFP sees as major FAM achievements. That response zeroed in on contributions of Working Groups for their value-added. Specifically, one of the FFP officials responded, “There is a gap between theory and reality,” and further, that “FAM’s value-added has not been apparent.” When pressed on this point, it was indicated that FFP “would like to be informed of the work of the Working Groups,” so that FFP can also learn the relevant lessons. Furthermore, one official suggested that the problem “could be easily handled on the level of an FYI or a memo.” A specific example of the ‘gap’ cited by the FFP official concerns the Monetization Working Group. This official suggested, “It would be in the best interest of FAM to be more proactive in relating progress or issues dealt with by that Group.” The situation thus seemed to come down to a question, the official suggested, of “Why do I have to ask?” These FFP officials were at some pain to point out that their perceptions are based mostly on the fact that they do not feel they know what the FAM partnership is achieving. This 13 situation would seem to entail two factors, one more simple, the other not: first is communications, which presumably can be corrected; second, divergent perspectives, which may be less easily addressed. As to communications between FFP and FAM, there is some difference of perception on the details of what is communicated and what is not. It is recalled that FFP has the very same access to the FAM website as any other user, including all of the latest progress reports of the Working Groups. FAM staff found in meetings with FFP that they were not current with the latest updates of WG reports entered onto the website nor with topics covered by the latest edition of the FAM quarterly, Food Forum. Regarding FFP’s perceptions of the valued-added of FAM, that seems to reflect a certain discomfort in the relationship between FFP officials and FAM Coordinators. A variation on the perception theme reported in the FFP interview was FFP’s perception that the PVOs do not credit it for its support of their work. When FFP deals with the PVOs, for example, on questions of local capacity building issues, the officials indicated it is mainly through individual organizations. “Why not,” an FFP official asked in this regard, “deal with the PVOs on these issues as a group, that is, through FAM?” According to the FAM Coordinator, however, this idea was once suggested by a FAM member to FFP, but the suggestion was apparently rejected, suggesting a possible disconnect in either communications or different perspectives on how best to manage food aid/security programs. A consultant from the Mendez-England FFP support contract present at the interview proposed that one way of creating greater collaboration with FFP might be to take a ‘best practices’ product and use it as the centerpiece of a discussion with the FFP, starting with the question, for example, “How far do you have to go to define a ‘best practice’?” “What does it add up to?” “How is it useful to FFP?” While not suggesting that FAM is neglectful of generating best practices, since that is clearly not the case, it implies that for one reason or another, FAM’s focus on best practices in neither being seen nor heard, much less acknowledged, by FFP. Again, this situation seems to devolve to a matter of mixed messages or divergent perspectives. The interview addressed the issue of sustainability, labeled by one FFP officer as “institutionalization” or “ownership” of FAM. That person felt the PVOs needed to demonstrate more ownership, to develop more of a commitment than is now perceived by FFP. Presently, the perception is that “PVOs think FAM is ‘nice,’ but not essential,” as one official noted. A related factor is that FFP sees FAM as an avenue for the Title II PVOs to overcome what was depicted as their “basic competitiveness” and to develop a true collaboration. After all, as one FFP official indicated, “collaboration is the centerpiece of the latest round of ISAs.” It should be noted that FAM member PVOs with ISAs have already devoted a portion of their ISA resources to participation in FAM meetings and WGs. They are evaluated on that participation as a part of their ISA programs. In addition, since FAM’s ISA derives from the same pool of funds that the FAM members’ grants, this reduces the overall 14 possible funding levels for those members. FAM members strongly supported this arrangement during the ISA process. Otherwise FAM would not have received ISA support. Such different perceptions and divergent perspectives are all the more disappointing to FAM, since just a few years ago, at an annual meeting at Coolfont, West Virginia, the rationale and usefulness of the FAM initiative was reaffirmed by all Title II stakeholders, including, noticeably, USAID/FFP. These stakeholders came to an understanding and agreement on what FAM was expected to accomplish over the next few years. Members and FFP agreed to the activities of FAM, and that the members would build these activities into their own ISAs, as well as support FAM in securing its ISA. The process of cooperative decision-making was perceived as important in that it involved all the Title II partners. That the period for this evaluation (late 1998 – early 2001) is, in effect the ‘post Coolfont’ years, it can only be said that a return to the spirit of cooperation agreed to at Coolfont should be a consideration of all the partners of the FAM initiative. 15 Chapter 3 KEY FINDINGS – DISSEMINATION AND SHARING OF STATE-OF-THE-ART FOOD AID/SECURITY PRACTICES This chapter considers some of the inner workings of FAM, the functioning units of the organization. Even for the inner workings, it is recalled, FAM is a facilitator, a mediator, an agent in the middle that can help make things happen but that cannot force results. It is mainly through the components of FAM that it fulfills it objectives linked to food aid standards, to the knowledge base, and to collaboration. We begin with a review of the Working Groups. A. Working Groups 1. Monetization Initiated at the 1997 FAM annual meeting, the Monetization Working Group (MWG) has as its objective improved institutional capacity to implement Title II monetization programs. Its focus is improved design and implementation of monetization activities. Skills training of field and home office personnel represents one means for achieving its objective. The MWG works in collaboration with FAM members, FFP officials, USAID Missions, and other interested parties. Three key products contributing to its objective are a monetization manual, workshops, and establishing a mechanism for future collaboration. During FY 2000, the MWG finalized and distributed the Title II Cooperating Sponsor Monetization Manual. The manual represents a reconciling of PVO manuals with legislative changes and new USAID guidelines. Used in monetization workshops, it along with the workshops has been favorably rated by participants. Such a rating is based on responses of FAM members to questionnaires distributed to the general membership, members of the MWG, and participants in several monetization workshops. WG members questioned indicated that even while the monetization manual is being tested, they still often use it as a reference guide. One WG member reported that his PVO was using the manual “in ensuring conversion of commodity resources into resources for programming-planning, logistics, and compliance with guidance.” Another WG member suggested that the manual’s use “enables us to transfer more management responsibility, especially in monitoring monetization activities, to the field level.” Yet, another WG member mentioned the economy-of-scale opportunity provided by the manual, saying, “as we did not have to spend the money and time to develop it individually, we have been able to free up those resources for other training needs.” Finally, one member summarized the effect of the manual, shortcutting the jargon of the 16 sector indicating that, “monetizers are better prepared to monetize with all the information at their fingertips.” The Monetization Working Group also contributed to a ‘Monetization listserv,’ a software package provided by FAM’s mentoring partner, FHI, to serve as a ready-made communications conduit for the members of FAM’s Working Groups. It does not use the listserv much because of a concern about inaccurate information being sent over it. The MWG is presently working through the mentorship program with FHI to resolve this issue. A listserv is run by a software program that groups subscribers’ email addresses into a single grouping, accessed by a single email address, for immediate access of all members to any message sent to that address. The Monetization listserv was created to facilitate and promote the exchange of information on how practitioners face increasingly challenging issues of monetization. 2. Monitoring & Evaluation Also initiated at the 1997 FAM annual meeting, the Monitoring and Evaluation WG (M&E WG) was formed to address priority M&E issues and needs of FAM membership. Specifically, the M&E WG aimed to assist in developing the membership organizational capacity to monitor and evaluate Title II programs. It also intended to facilitate development of a series of M&E toolkits, containing methods, techniques and other instruments designed for the food aid sector. This WG participates actively through its own listserv, used for addressing technical issues and questions, workshop announcements, and notices of new publications in the FSRC. The M&E WG has recently contracted a consultant to design a toolkit which will consist of a compendium of monitoring tools. Having attended a M&E WG meeting, one of whose purposes was to discuss the consultant’s work plan, the evaluator noted a strong commitment of this group to the work and to its completion in a timely manner. A FANta M&E representative present for the meeting played a proactive, constructive role in discussing technical issues. During the evaluation period, the M&E WG has undertaken a review of baseline, mid￾term and final evaluation methods and tools. One product of that work, “Review of Agriculture Baseline Surveying Methods of Title II Funded PVOs: Part I – Socioeconomic Methods,” is posted on FAM’s website. The WG also sponsored a second review that focused on an agricultural biophysical baseline and evaluation methods. A formal report derived was from this review, titled “Prediction, Impact and Control Variables for Field Research Evaluation,” which can also be found on the FAM website. Yet a third review has been conducted, resulting in a report, titled “Review of Health and Nutrition Project Baseline Research Methods of Title II Funded PVOs.” It, too, is located on FAM’s website. That document, according to one WG member, “was very 17 useful in developing our questionnaires for the DAP (Development Activity Proposal) surveys.” Applications of reviews and products from the WG have resulted in one PVO’s use of food diversity and frequency indicators in its food aid projects. A member of the M&E WG commented that, in “putting presentations from workshops on the website, e.g., the sampling workshop and in general, having a resource list of real-world documents and links is helpful.” 3. Local Capacity Building Also formed at the FAM annual meeting of 1997, the Local Capacity Building Working Group (LCB WG) struggled at the beginning of the ISA grant due to sporadic meetings and an apparent inability to agree on a common definition and concept for LCB. As the LCB WG FY 2000 Annual Report notes, agreement was difficult because of a “diversity in focuses and approaches of capacity building.” In order to recommit itself and refocus the work of the Group, it decided to prepare a workshop in LCB for the 1999 Annual Meeting. Objectives of that workshop included establishing common elements/skills for LCB, identifying areas for FAM collaboration, and developing a WG work plan. Last year the LCB WG began making progress, based on each PVO presenting its capacity building (CB) approach, methodology and learnings. This year, under new leadership, a cohesion of that Group has evolved and a new focus has been given to the work. A planned three-day workshop will focus on a review of CB approaches, indicators and an M&E methodology. The LCB WG has prepared a draft consultant’s agreement which will provide FAM and its members (and the larger development community) with an annotated review of the current Title II CB indicators, methodologies, and tools for capacity building. A second draft consultant’s agreement is to “Develop an inventory of capacity building indicators used in current Title II development programs and, in collaboration with WG members, identify and analyze core CB indicators.” This second task aims to provide a non-Title II perspective on LCB and to identify gaps in indicators currently used in Title II programs and ISA grants. In that respect, the effort is intended to identify unmet measurement needs in the LCB arena. The LCB listserv is one of the most active, used by WG co-chairs to communicate with members about meeting schedules and other WG matters. The WG has also used the listserv recently to hold a discussion, for example, about the scope of work for the consultancy. This WG has also been very proactive in ensuring that their website home page is updated with the latest presentations and meeting notes, both important for this group whose members seem to have a more difficult time meeting in person because of work travel schedules. The present co-chair of the WG, in a response to the WG questionnaire, discussed the Group’s problems over a common definition of capacity building, saying “it means 18 different things to different PVOs. Before the WG can operationalize a set of practical definitions, we first need to determine the commonalities and differences in LCB.” One of the priorities of the proposed workshop will be “to collectively identify an array of best practices LCB indicators that measure capacity at different levels.” Based on the questionnaire used with the LCB WG, it is very evident that some member PVOs are considerably further ahead in organizational learning in the capacity building arena. This is partly a matter of their size, their resources, and existing capacity to perform mutual capacity building. It is also noted that the length of time and effort that institutional capacity building requires is usually vastly underestimated. 4. Environmental This Working Group was the only one of the four that did not formally emerge during the 1997 annual meeting. Even though the Environmental Working Group (EWG) is defined as an “informal” WG, since FFP does not officially recognize it, it is one if not the most active of the WGs. Besides sponsoring a major workshop on environmental compliance, the WG has used funding awarded directly by USAID to support additional compliance workshops implemented by individual PVOs. It has also sponsored several well-attended brownbag discussions. Notable, as well, is participation in the EWG by USAID environmental officers, one of whom maintains contact from his overseas post. The EWG is the most active of the listserv users, employing their listserv to send out meeting announcements and meeting notes, as well as to provide technical support through quarterly bibliographies and lists of environmental websites. It is also used to communicate news and notices on key environmental issues. The WG’s production of an Environmental Documentation Manual (EDM) has met with considerable success. Commenting on its use by his organization’s food team, one WG member noted, “Field staff use the EDM is preparation of the environmental documentation required for design and reporting. As the technical person on the Food Team, I use it for giving advice on compliance issues, especially at the design phase.” More specifically, this same representative, in responding to the EWG questionnaire, noted that one of the best practices resulting from EDM application was the pesticide action plans used in curbing dangerous chemicals. In response to the WG questionnaire, another PVO representative made a direct link between the WG and application of practices in the field. She commented, “Environmental considerations are now an integral part of all food security projects. A dramatic example is that the quality of the Food for Work roads has improved greatly now that environmental mitigation measures are being used.” Additionally, she noted that environmental mitigation indicators are being incorporated in the M&E process of her organization. At least two USAID environmental officers active in the EWG are very positive about the group’s work. One of these officers noted that communications between the PVOs and 19 USAID Missions on environmental issues have especially improved because of practices embodied in the EDM and other WG products. He also expressed strong satisfaction with a derivative of the EDM, a ‘A Cooperating Sponsor’s Field Guide to the USAID Environmental Compliance Procedures,’ suggesting, “I can teach a neophyte how to prepare an IEE in about one hour with this.” The ‘Field Guide’ was jointly authored by CRS and FAM Staff. The second USAID environmental officer, now working in a USAID Mission, commented that the EWG was serving a useful purpose in organizing USAID and PVOs around Title II, “Responding in a participatory way across institutional lines to the recognition that USAID’s environmental procedures need to be internalized.” He also noted the momentum maintained by the EWG over four years, rooted mainly in the interest and conviction of participants, yet with relatively little budgeting. In a burst of praise for the EWG, this same USAID officer, in responding to the WG questionnaire, commented, “The EWG Forum is absolutely essential to maintain the role of champion and ‘exhorter’ to keep capacity to address environmental soundness as a fundamental principle on the agenda.” The EWG illustrates how robust the activities of a WG can be and how broadly-based the participation, when FAM members engage on critical issues in cooperation with USAID environmental officers. B. Workshops and Training Tools 1. Monetization Several MWG members have been especially active in hosting overseas workshops. These workshops were held in Ghana, South Africa, India, and Peru. Monetization workshop participants interviewed indicated that the manual/workshop package was especially useful in spelling out the Bellmon analysis and open and competitive commodity sales. One participant noted that a specific contribution of the workshop to managing food aid occurred in his work in “initiating the monetization of soybean oil in the private sector of Bangladesh under Title II 416(b).” Another participant remarked that the manual had been “especially useful in managing such Title II activities as the cost recovery benchmark, U.S. commodity procurement schedule, Bellmon analysis, call forward, and monetization budget and sales proceeds management.” Pre- and post-test results from the most recent workshop hosted by CRS￾Peru in January 2001 showed an improvement of 47.9% overall. The working group is current conducting its own follow-up survey of workshop participants to help instruct its future training activities. The survey requests information on how well the training has been used, and what further support workshop participants require. The results will be reviewed in early July, and the working group will frame its future training plans around those results. 20 2. Monitoring and Evaluation The M&E WG cooperated with FANta to sponsor and implement a Sampling Workshop in the Washington, D.C. area and a Data Analysis Workshop in Nairobi, Kenya. Participants in the M&E workshops were positive in their ratings of its success, though it should be noted that participants were generally difficult to locate for purposes of disseminating the evaluation questionnaire. One participant in the Data Analysis workshop noted that it “was extremely useful – a very practical hands-on kind of workshop – the people who left that were able to do appropriate data analysis in their programs.” Another said she “learned how to analyze nutritional indicators such as exclusive breastfeeding and intake of complementary foods and to use malnutrition indicators, including stunting, underweight, and wasting.” The Data Analysis workshop was also useful to a participant who manages food security programs because, as she suggested, “we now put more emphasis on M&E, and we have also come to the realization that we need to concentrate on building capacity in M&E, especially in statistical analysis.” For the Sampling Workshop, one participant noted a fundamental lesson learned about survey design. Tersely put, it was, “Once you have reached the analysis stage, sampling errors can no longer be cleaned, so think carefully about your sampling design before conducting your survey.” 3. Local Capacity Building Because there was only one response to the LCB workshop participant questionnaire there is little to report, and what there is suggests a mixed picture of the workshop. The sole respondent noted “The most applicable learning from the session was to hear about LCB in other organizations.” But, she went on to say, “It was also interesting (partly useful and partly disappointing) that FAM members cannot come to a common understanding of LCB (too different organizations).” This comment reaffirms the earlier discussion with the LCB WG co-chair on the divergent approaches of the members to LCB. One best practice cited from the LCB Workshop reported in the WG questionnaire, is instructive for the LCB WG itself; it is, “The local NGO or other group targeted for LCB initiatives should be included in all stages of the design and implementation of the LCB program (i.e., the participative approach – they know their needs best).” While clearly we cannot generalize from one response to the questionnaire, that response at least underscores the need for LCB WG members to be more aware of different approaches to the subject. 21 4. Environmental The EWG offered one FAM-supported workshop. It was a training-of-trainers (TOT) workshop in environmental compliance, namely Regulation 216 in Title II programs. Workshop participants who responded to the questionnaire were generally very positive about the outcome of the workshop. The EWG TOT workshop was designed to further the understanding and ability of field staff to manage Regulation 216 compliance, since it will be replicated by the attending trainers with field staff in their countries or on a regional basis. It has already led to subsequent Regulation 216 compliance workshops (Peru) and others are scheduled for this FY (Bangladesh and the Democratic Republic of the Congo). Overseas field staff that took the Regulation 216 training of trainers workshop and responded to the participant questionnaire judged the course favorably. One staff member indicated he had learned about “the relevant tools and resources that can be tapped when an IEE needs to be done,” and that he had increased his “awareness about environmental issues during planning and implementation of DAPs.” Another overseas field staff person noted that the Regulation 216 workshop had brought together USAID officials, national government officials, and PVO head office and field staff in a mix he characterized as “good for discussion.” Another field staff person also noted the mix of different expertise, but based on “networking with other country food aid/infrastructure programs.” Finally, another participant suggested the workshop provided the possibility of “sharing Environmental Management Plans with affected people or targeted beneficiaries.” C. Food Security Resource Center FAM’s Food Security Resource Center (FSRC) provides several key services to FAM members and the food aid community generally. First is the website, which provides a variety of choices to users. It is organized for FAM members and non-members alike. Each Working Group has its own page on the website. Their meeting minutes, annual work plans, and selected technical products, such as manuals, are also provided. Specific website pages for Nutrition and USAID are offered, as well. A training calendar is available to all users, providing information on the latest workshops and other training opportunities. Website: The website has an outline of documents and links to bibliographies of state-of￾the-art food security publications. Selected publications in full text are offered, of which certain USAID reports were noted to be very popular. A page on food aid Periodicals is provided, along with FAM’s own publication, Food Forum, which is offered in full text. 22 The website serves as a management tool for FAM coordinators and working groups. Specifically it is used as a means of keeping in continuous communication with FAM members, Working Groups, USAID food aid officials, and other professionals. Each FAM member has access to information on Working Group progress, issues, documents and news of planned training functions. FAM management coordinates the input of materials onto the website. One specialized aspect of the FSRC is the Development Ecology Information Service for Africa or DEVECOL/Africa. It is a geographic information-based system developed with funding from USAID. It provides map-based soil, climate and crop-suitability data for Africa cross-referenced with a database of project documents (descriptions, case studies, data, and evaluations). Many of its data sources come from the FSRC, though it is not funded by FAM. Presently, DEVECOL must be accessed from a workstation at FSRC. Tentative plans are to expand the database of this tool. Its specific use to FAM members is in the design of their projects. As the DEVECOL project is supported to increase its database, this could enable FAM to digitize some of its resources. Listservs are another piece of the FAM website. They are email distribution systems for FAM members and other users. Four of the listservs have been developed to facilitate communication and information exchange on working group subject matter (e.g. Monetization, M&E, Local Capacity Building, and Environmental). The fifth and newest listserv on “Commodity Management” enables the exchange of information regarding that topic. FAM has also been testing the use of internet chat technology which would permit online meetings to take place. FAM staff and FHI have found it to work well with small groups with short agendas or open agendas, but not large groups with long agendas. Several FAM members have been reticent to make use of this technology out of concern that it could compromise the security of organizational servers.” Use of the food aid website during FY 2000 shows an increase over the previous year of 32%. From February 1999, when Net Tracker website tracking software was installed, visits to the site increased by 85%. Net Tracker can determine number of visits, location by country, and pages viewed. These numbers, as with all website hit numbers, must be treated carefully. For instance, half of these visits are short, that is, the user leaves immediately, though this number has recently been found to remain static. A survey has been posted to determine more precisely exactly what ‘serious’ users expect, whether their expectations are being met, quality of service (including ease and speed of access to specific entries), and additional entries they would like to see. A review of questionnaires from FAM general membership shows a highly positive view of FAM's website. One user reported, “We find the website very helpful and user friendly. Our staff uses it frequently when performing research on policy issues, searching for links to other websites, and for basic food security definitions.” Another member, representing one of the larger PVOs with worldwide programs, indicated that 23 the food aid website is easily accessible, and that while “technology is usually the only block to field office access, few country programs have this difficulty any longer.” Most respondents indicated full satisfaction with FSRC services. On the ‘improvement list,’ one member proposed “getting the FSRC online as a fully digitized library (bibliography and texts), which would enable full access to these resources.” Many Working Group members pointed to the website as a valuable source for materials used in developing and submitting their DAPs and preparing their Detailed Implementation Plans. Of particular use in these tasks is current USAID/FFP guidance posted on the website. This would include food aid guidelines on commodities and commodity management, monetization, the environment, results reporting, among other topics. USAID Staff are also frequent users of the FAM web site, and are regularly among the top five organizational visitors each month (see Annex 10 for a complete analysis of FAM website data). Document Collection: The FSRC library has about 8,000 documents, organized in files or on shelves. They are readily accessible by visitors to the Center. Documents are used on-site at the Center, read on the website in the case of a limited number of selected reports or articles posted in full text, or obtained in hard copy by direct request from FSRC. Requests numbered 146 for FY 2000. Requesters include FAM and non-FAM member PVOs, USAID staff, university professors and students, and USDA staff. In response to those requests, 251 documents were sent out. A record of requests for documents to the FSRC for the period FY1999-FY2000 (2d quarter) is represented graphically below. Overall requests are represented in Series 1, while Series 2 represents the number of requests which came from field staff. On average, one quarter of all FSRC document requests come from the field. The large drop in requests during FY2000 is attributed to the lack of a TIS at FAM from first quarter of FY2000 through third quarter. 24 FSRC Requests FY99 - 2nd Quarter FY01 0 10 20 30 40 50 60 70 Quarter and FY Number of Requests Series1 66 59 57 63 36 28 38 45 46 56 Series2 14 13 15 23 11 8 8 9 14 12 FY1999 Q1 FY1999 Q2 FY1999 Q3 FY1999 Q4 FY2000 Q1 FY2000 Q2 FY2000 Q3 FY2000 Q4 FY2001 Q1 FY2001 Q2 USAID usage of the FSRC’s print resources has been less than their usage of the FAM website. The graphic below compares FSRC requests for documents from PVOs to requests from USAID staff for the period FY1999-FY2000 (2d quarter.) 25 0 5 10 15 20 25 30 35 40 45 50 Number of Requests FSRC Requests from PVOs vs. USAID Requests for PVOs 35 36 40 49 29 18 31 30 27 33 Requsts for USAID 5 6 4 5 5 0144 6 FY1999 Q1 FY1999 Q2 FY1999 Q3 FY1999 Q4 FY2000 Q1 FY2000 Q2 FY2000 Q3 FY2000 Q4 FY2001 Q1 FY2001 Q2 Regarding FAMs own documentary production, an inventory of documents produced by FAM under its ISA is presented in Annex 8. Mentoring Relationship. Since 1998, FAM has had a formal mentoring relationship with one of its PVO members to support FAM in its website design and maintenance capabilities. This support, rendered by Food for the Hungry International (FHI), is judged to have been very effective, as seen in comments from Steering Committee members, Working Group members, and the general membership on the utility of the website. It has included strengthening of the Technical Information Specialist’s (TIS) website skills, including formal training, which resulted in the recent redesign of the website mostly on her own. The FAM website is hosted on FHI’s server, notably because of the advanced sophistication of FHI’s system. (See Annex 10 for a complete analysis of the FAM/FHI Mentorship Agreement.) Food Forum. Published quarterly by FAM, this publication includes articles on topical food aid and food security issues and experience of both FAM and non-FAM members in managing food aid. Once published bi-monthly, publication was cut back due to the significant workload it represented to FAM staff, and increased printing costs. Now, the workload for preparing Food Forum is shared with the membership. FAM member PVOs have volunteered to take responsibility to co-edit four issues per calendar year. While last year, World Vision and Africare volunteered, this year it was OICI and, again, Africare. It is published in both hard copy and on the web. Most member PVOs have contributed one or more articles to the publication (see Annex 7). 26 While Food Forum represents considerable state-of-the-art information on food aid issues and activities, it was found in a FAM survey to be not as popular among members as, say, the FAM website. FAM Coordinators suggested during interviews that this is attributable to its content not generally being directly relevant to members’ specific food aid operations. FAM, however, receives between one and five requests weekly for subscriptions, mostly from USAID, US Department of Agriculture, PVO field offices, private consultants, and other organizations not particularly involved in Title II. Food Forum’s future is not fully clear to FAM Coordinators. The contents of the publication could be shortened and focus primarily on addressing more immediate topical issues, such as: FAM events and activities, new regulations, Title II guidelines and policies, and new state-of-the-art tools, documents, and resources available on the FAM website and in the FSRC. Constraints and Opportunities: A move of the FAM offices from one part of Washington, D.C. to another, including movement of the FSRC, constrained the full development of its potential to serve the membership and others during FY 2000. Departure of the TIS and succession by a new person was also constraining. Despite those constraints, the FSRC was able to continue to serve, though at a slower pace than earlier years. In addition, a volunteer internship program has recently been introduced, allowing the TIS to focus more on managing the dissemination process than on the work involved in FSRC’s extensive paper-flow. A second constraint has been the posting of the FSRC bibliographic database (approximately 8,000 bibliographic citations) on the website. Originally tasked out to a consultant, the activity ran into technical and personnel troubles. During the course of this evaluation, an external consultant has offered on a pro bono basis to place the FSRC database on-line. If this arrangement works, the link could be completed by May, and would include subsequent technical support and training. (It was learned before finalizing this report that the work has been almost completed and the database could be online imminently.) One particular opportunity, or more realistically, challenge, is to digitize the entire FSRC collection. This would provide one of the best sources of food aid documents extant today on a worldwide basis. Whether to make all of FAM’s approximate 8,000 documents available on-line or on a CD, the availability of this source would significantly enhance the knowledge base on food aid . Contacts have been made with different organizations to advance the idea of digitizing the library, through either a cooperative mechanism or a grant. Since there are some highly labor-intensive aspects to the task, it is not yet clear how it can happen. 27 D. Other Collaboration Additional venues for dissemination and sharing include brownbag discussions, annual meetings, and Steering Committee meetings. Brownbags have been alluded to already. Their value is in the sharing by one or more persons, often representing one of the Working Groups, in new work in a specific food aid or food security sector. Since they are seen as working lunches, there is a perception that this is not just another meeting to attend. They are seen as an integral part of the FAM fabric, contributing to the growing body of food aid knowledge and to an increased sense of professional collegiality among FAM members and partners, including USAID officials. The Environmental WG has been at the forefront of the brownbag. Brownbags also provide a venue for the presentation of new findings that members might not find out about otherwise. They also offer the opportunity for discussion, exchange and learning. A few of these have evolved into day-long, PVO-wide discussions, on such important subjects as one organized by the MWG on freight rates. These represent opportunities for true learning and valuable exchanges of knowledge. Annual meetings have been an important point of contact of FAM members and their USAID partners. There, issues of mutual importance are raised, the agenda of FAM for the coming year is set, and various trainings offered. The concept of Working Groups, for example, now so integral to the work of FAM, grew out of one annual meeting. These meetings are held in different regions of the country, mainly to reflect the geographic diversity of the membership. The Steering Committee (SC), or board of directors of FAM, is an important part of the FAM structure. As the name implies, this committee guides FAM in meeting its objectives. Since SC members are themselves representatives of the member organizations, they have a stake in getting the best results from FAM. CARE is the only permanent member as the project holder. Two other founding members are on the SC and these rotate annually. Four non-founding members participate, selected by lots annually, serving a single term. A Chair is selected from among these seven members. The SC has just completed a new set of by-laws, which created the new, more representative structure. The original structure of the SC was altered so to be more representative, to increase ownership of FAM by all its members, and to ensure that perspectives of newer members are adequately represented in setting the direction of FAM work. It is important, according to one SC member, for the founding members to continue to have a strong voice in the operations of the initiative. Equally critical, as well, suggested this same member, “is to ensure that a broader range of PVO experience and approaches are included.” Members of the SC, in individual interviews with the evaluator about FAM’s functions, rendered well-configured interpretations of its origins and roles. As one long-term 28 member of the SC asserted, “Since PVOs are not structured to collaborate, it takes an entity such as FAM to bring them together.” Another noted, in the same vein, “if there wasn’t any FAM, some structure like it would have to be invented.” Yet a third suggested that “FAM exists because of USAID; how could it deal with each PVO independently in institutionalizing its changing mandates? If it hadn’t invented itself, USAID would have had to invent it.” The SC is an important venue for these mostly senior food security managers. They not only collaborate with fellow professionals, they learn of each others’ approaches to food aid, and cooperate with each other in responding to USAID Title II requirements. While FAM is explicitly not a lobbying organization (unlike the Food Aid Coalition that includes both Title II PVOs and commercial food interests), it does serve to unite the PVOs in representing themselves to USAID on technical matters of food aid. 29 Chapter 4 KEY FINDINGS – ORGANIZATIONAL IMPACT FAM’s major role is to coordinate members in achieving mutual objectives of improved food aid standards, an expanded knowledge base, and more effective collaboration. We have reviewed in previous chapters some of the detailed results of FAM’s coordinating role. Here, we summarize FAM’s organizational impact, its synergistic role in institutional strengthening, and its management, including coordination of Working Groups. We then turn to a consideration of FAM’s potential for sustainability. A. Coordination of Members All of FAM’s work is coordination of one type or another. Its raison d’etre is support of its primary clients, the member organizations. In this sense, its achievements are the achievements of its clients. In another sense, the ‘higher level’ client is USAID/FFP, whose funds enable the technical support and implementation of Title II Development Programs. The final client in the chain is the end-user, who is the food insecure rural or urban family served by Title II. While we cannot trace FAM impact to the end-user, we can at least review its effect on members and, to a lesser degree, on USAID. FAM has increased member organization staff awareness of the importance of improved food aid practices. By facilitating members’ pursuit of such practices, it has raised their consciousness about the role of collaboration in achieving excellence. One member, responding to the general membership questionnaire, put it directly: “Generally, collaboration through FAM has raised staff awareness of best practices of our own agency and others, and thus has encouraged collaboration and information sharing with the aim of improving program design and implementation.” Another member reflected on FAM’s coordinating role, saying “FAM has helped my organization improve its capacity within several areas related to food security programming, mainly through its working groups and sponsorship of various trainings and workshops. Working group meetings have particularly served to expose our junior food security staff to and keep mid-level staff informed of best practices, lessons learned, and cross-cutting issues in the Title II development arena.” Yet a third member is even more specific about FAM’s collaborative role. She credits FAM with a significant contribution to her organization’s food security program, reporting that “The collaboration on the FAM Monetization Manual, the Environmental Manual and training materials, the two M&E tools (agricultural and health), and the associated workshops have been some of the most beneficial contributions to our organization.” Their contribution was in the areas of preparing DAPs, designing M&E plans, and reporting on performance. 30 This evaluation attempted to go beyond individuals who have direct contact with FAM efforts, namely through the workshop participant questionnaires. In reviewing responses of some of the field operations staff members, we have been able to see the reach of FAM-coordinated activities. These were more evident in the monetization, M&E, and environmental workshops, whose products have been directly and immediately applied to field operations planning and implementation. Overall, PVO members interviewed or questioned for this evaluation gave high marks for FAM’s coordinating role. While there may have been some self-criticism by members of their own shortcomings in the area of participation, there were virtually no criticisms of FAM coordinators themselves in their efforts to elicit membership participation. B. Institution Strengthening Institution strengthening is a difficult subject to embrace and assess analytically in this type of evaluation. Since we have to base this part of the evaluation on what people say, it is difficult to be definitive in the absence of an institution assessment. Nevertheless, relying on the perspectives of persons questioned, we will endeavor briefly to say something about institutional strengthening under FAM. It is in the areas of knowledge management, training, and field applications that we see some of the institution strengthening impacts of FAM. How a PVO conceptualizes its knowledge, demonstrated mainly in program planning and design, including M&E design, offers one clue. Many interviewees and respondents reported on their use of new tools promoted through FAM-supported workshops, mainly monetization and environmental tools, in the design of their field programs. That certainly is a sign of institution strengthening. The application of new methods and approaches gleaned through training is another benchmark. In cooperation with FANta, FAM members have made considerable headway in applying the USAID-supported results-based management approach to planning, design, and monitoring of Title II programs. Staff capacity building, a subset of institution strengthening, is reflected in two different responses from members. One is focused on home office staff development, the other on national staff development. The home office focus, employed by a relatively recent FAM member, is as follows: For us, FAM’s main capacity building impact has been at the headquarters level…our headquarters staff participate in the FAM Working Groups on Monetization, M&E, Environment and LCB and have also participated in various workshops organized by FAM in collaboration with other PVOs…We have found these workshops to be invaluable, leaving participants with solid and improved knowledge of the focus topics. The national staff development focus is reflected by a ‘seasoned’ PVO, which noted, “National staff development has been enhanced through participation in FAM-supported 31 workshops. In addition, access to FAM’s resource library has been useful for field staff. The FAM Monetization Manual has been widely appreciated as a resource and reference tool.” In summary, FAM’s pursuit of all three of its objectives, (improved food aid standards, an expanded knowledge base, and more effective collaboration), has contributed significantly to the institution strengthening of its members. C. FAM Management The distinction between FAM’s coordination role and its management functions may seem small at first glance. However, it is important to underscore that FAM does not manage its members, because it is neither its mandate nor in its interest. Nevertheless, FAM does have a ‘management’ function, at least in administering its own roles and functions. Interviews with past and present coordinators suggest that managing FAM is a ‘fine art.’ Because of the fine line between even-handed facilitation and heavy￾handed manipulation, the coordinators must strike a clear balance. An earlier FAM Coordinator, who was present during the early part of the grant period being evaluated, was characterized by an SC member as less proactive than the present Coordinator. The two styles of management, however, are not in conflict, depending on how and under what conditions they are implemented. In fact, both styles were judged equally successful in keeping FAM on track, developing new initiatives, and continuing to effectively manage the grant. The present Coordinator is highly proactive, nudging and cajoling the different components of FAM along in their multiplicity of tasks. She sits on one Working Group and on the Steering Committee in formal roles. The Deputy Coordinator, less proactive by nature but equally effective in carrying out his role, sits on three WGs. Here, the fine line alluded to earlier is very important, given the special character of participation in the WGs. Similarly for the Steering Committee, the light-handed, but firm approach is essential in getting senior managers away from their ‘day jobs’ to focus on FAM. Nevertheless, they had high praise for the most recent past Coordinator, the present Coordinator, and her Deputy. Financial Management: FAM’s five year ISA budget is $1,780,000, factored out at $356,000 annually. This budget is managed by FAM coordinators under the auspices of CARE, the project holder since the ISA’s inception in 1992. CARE underwrites some grant costs related to FAM’s recent move to CARE’s Washington, D.C. office on K Street. That move was triggered by FAM’s desire to be more accessible to walk-in users. Because the move proved expensive, CARE decided to pick up some of the new rental and administrative costs. The CARE ‘subsidy’ means that more program funds are available. When one reviews the cumulative results of FAM’s efforts over the past several years, the $356K annual budget has all the earmarks of being cost effective. 32 Given FAM’s value in supporting Title II, in rendering important economies of scale to USAID/FFP, and in providing the glue for an association of 17 individual Title II PVOs, it would clearly seem to offset the rather modest annual cost. We now turn briefly to the management of the Working Groups, which is as much a function of member organizations as it is FAM management. D. Working Group Coordination Working Group coordination is a responsibility of the PVO members who comprise them. FAM’s role is to facilitate the Group. Since some of this information has already been treated in the previous chapter, it will not be repeated. One general finding that has a strong bearing on all of the WGs is that if the WG does not have a specific, concrete product or result towards which it is working, the members tend to lose interest. One example is a WG that had a successful product, in this case, a manual. Once the product was completed, members’ interest in participating was purported to have waned. In this case, success bred indifference 1. Monetization Working Group This WG is the one cited in the above example. According to participants questioned, it is one of the smoothest running, most enthusiastic groups. In fairness to the others, this condition is related to the fact that the subject matter is specialized and therefore more manageable. This WG is perhaps one in search of a new product, and thereby, a renewed spirit. The MWG will shortly be reviewing the results of its four workshops offered to date. This will serve as an opportunity for the Group to assess its progress and set new targets, whether they are new training activities or new tools for use in the field. 2. Monitoring and Evaluation Working Group This WG has at times been constrained by weak attendance, due to heavy travel commitments of its members. Those interviewed, however, were positive in their praise for the contributions of their fellow members. As in the case of the other groups, the M&E Working Group uses consultants to complete its technical work. In this respect, it is presently making headway in its process of developing a series of M&E toolkits, namely a compendium of monitoring tools, for the food aid sector. It is committed to completing this product on time. As noted earlier, this WG actively participates in FAM through its listserv, which it uses to address technical issues, announce workshops, and list new publications in the FSRC. 33 3. Local Capacity Building Working Group This WG has begun to show progress after a sluggish start. Some participants in the LCB WG realize what the constraints were and were not afraid to address them. Issues related to its slow start are the absence of expertise in the subject matter, and, therefore, a void created by lack of agreement on an appropriate product. Minimal response of LCB WG members to the questionnaire is itself perhaps indicative of a constraint. A good quote is worth a thousand words, as seen in the following: It is my impression that most of us on the LCB WG are practitioners, i.e., users of information, rather than generators and systematizers of information. If the composition of the group included a mix of researchers and practitioners, it might be easier to achieve synergies. This quote is meant to suggest that there is a lack of expertise in organizational development on the team. The new scope of work could help resolve this problem. 4. Environmental Working Group We have seen in the previous chapter the rave reviews of this group’s members on its performance. A combination of a passion for the environmental cause and very specific product-driven activities have given the group the claim to success seems to deserve. One constraint to the continued effort of this WG is structural in nature. It represents a variation on the theme of a product-driven orientation. Stated by an EWG member, it is, “the EWG had more interest when there were funds available for training. There was a product that was being offered. Now, it depends more on the interest of specific members.” Since it is not recognized by FFP as an ‘official’ WG, the EWG cannot access funding from FAM. Both the EWG and FAM need to make a case to FFP for formal recognition. E. Sustainability Issues Questions on the sustainability of FAM are not easily answered. The questions are complex and somewhat sensitive to the parties concerned, as well, namely the PVOs and USAID. Several issues need to be considered. First, in many senses FAM is seen as an instrument of USAID, a response to its Title II program requirements. One SC member characterized it as “a creature of USAID,” even though its formation was a voluntary act. In any case, FAM, in cooperation with FANta, contributes organizationally to meeting important technical requirements of Title II. Second, FAM is made up of non-profit organizations, only some of which have access to funds beyond those for operations, for example, from donations. This means that for Title II PVOs, there is a limited excess of funds for independently contributing to FAM 34 on average. Another approach might be for WGs to seek support from foundations to continue their work. The SC has addressed the possibility of assessing a membership fee or at least some member contributions. The SC sees that it is in FAM’s best interest to try to develop a source of support at least partially independent of FFP. Third, while various commercial options for marketing the FSRC’s services so as to at least cover its costs, for example, have been considered, most have been sidelined. The economies of scale to market FSRC services are simply not there. Support from food producers, for example, would presume to compromise FAM’s relationship with FFP. Commercializing FAM’s informational resources does not stack up well against evidence of other attempts to make such an enterprise cost-effective. Fourth, while FAM’s SC has considered options for sustaining the effort, it is legally circumscribed from using USAID resources (FAM staff or time) for exploring external opportunities. That is a perfect ‘catch-22’ situation. In the absence of any innovative answers, the evaluator can only say that the question of FAM’s sustainability will continue to test the ingenuity of both the PVOs and USAID. 35 Chapter 5 CONCLUSIONS, LESSONS LEARNED, AND RECOMMENDATIONS This chapter offers an interpretation of some of the major findings from previous chapters in the form of conclusions and lessons learned. These, in turn, serve as the basis for subsequent recommendations. Recommendations made to the various parties of this activity are intended to be actionable, that is, in their manageable interest to carry out. A. Conclusions At this mid-point in the current grant, the FAM initiative is well along the way to achieving its intended results. As tracked in the performance table presented in Annex 5, excellent progress is being made in fulfilling its objectives. The following is a brief summary of cumulative results achieved to date under the FAM grant. FAM Website: First, the FAM website has for the most part seen increased usage by food aid/security professionals, including FAM member headquarters staff and, importantly, field staff. A dip in use occurred at one point during a personnel and logistical transition. The website is also well-used by USAID personnel. It is seen by FAM members as one of the most popular components of the FAM initiative, not only for its substantive contents, but as a mode of communicating with other members on matters of mutual interest, especially working group matters. The mentoring partnership with FHI has been very successful in supporting the development of a multi-functional, user-friendly, and sophisticated food aid/security information and communications resource. FAM-supported Workshops: FAM-supported workshops have been successful in imparting new technical approaches and knowledge on key aspects of food aid, as well as in fulfilling certain Title II requirements. At only the halfway point of the current grant, nine of the ten-targeted workshops have already been successfully delivered. Each of the Working Groups (WG) has used its workshops effectively in providing specific, state-of the-art technical information to home office and field staff alike. The spread effect of these workshops provides an economy of scale in the dissemination of Title II and general food aid matters not otherwise achievable. Food Aid/Security Manuals and Tools: In the matter of manuals and other tools, one WG, Monetization, has already completed a manual and successfully used it in workshops to improve the skill base in monetizing food aid. Both M&E and LCB WGs are evidencing clear movement in developing their respective manuals or toolkits. The Environmental WG has succeeded in integrating specific principles or regulations, namely Regulation 216, into member Title II food aid programs. 36 Increased Knowledge of Food Security Matters: A proxy measure of members’ increased knowledge and understanding of food security issues is the number of organized meetings under the FAM grant. FAM has far exceeded the targeted number of organized meetings (no less than 25 organized meetings) in its support of brownbag discussions alone. During FY2000, 22 such discussions were held and in FY2001, seven of a planned 15 have been held. Other discussion venues include, importantly, WGs, the results of which are not only training activities and tools, but improved approaches to food aid initiatives. Improved Collaboration: Improved collaboration of FAM members as a product of annual meetings, joint program initiatives, mentorship relationships, and jointly developed tools and methods is evidenced in part in the success of the above-mentioned components. A highly positive perception by PVO members of an improved sense of collaboration among organizations that do not usually cooperate is another strong sign. An exception is the USAID/FFP perception that it did not have clear evidence of improved collaboration of members. Increased Autonomy: Increased autonomy of FAM and its membership in carrying out the above components is calibrated by the degree to which these components are self￾starting and, conversely, the degree to which they require a concerted effort by FAM staff to manage an activity. While difficult to measure cleanly, a proxy for this achievement is, as for the previous result, the degree of success of FAM components in achieving their objectives. Thus far, there is clear evidence that the WGs have been very successful in using FAM to jump-start their activities. The FSRC, with the strong support of its mentoring partnership, has also successfully become a vital tool for FAM members. Autonomy of the overall FAM initiative itself is complicated by the issues raised on sustainability in the previous chapter. Since FAM staff are precluded from devoting any time to fundraising, the future sustainability of the initiative rests in the hands of the Steering Committee, the general membership, and the funding decisions of USAID/FFP. B. Lessons Learned and Recommendations for FAM and FAM Members FAM Coordinators & Steering Committee Lesson Learned: Motivating FAM towards the Future. Interviews with SC members pointed to the supposition that FAM may not continue indefinitely under its present structure, in which it is fully dependent on USAID/FFP funding. Recommendation: Concerned PVOs who are FAM members should independently develop a detailed framework for a plan and assign a committee to explore the continuation of FAM’s key functions into the future. Parameters of this plan should include the future relationship with USAID/FFP, potential sources of outside funding, a vision of what a FAM future would look like, and the possibility of continuing the key functions of FAM as an entity partially or completely independent of USAID/FFP funding. 37 Lesson Learned: Enhancing FAM’s Role with USAID/FFP. Several of the FAM members interviewed have a perception that USAID/FFP is not aware of or does not recognize the full value of FAM to the Title II program. Conversely, representatives of FFP perceive that a gap exists between theory and reality and there has been little or no appearance of the value-added by FAM. FFP simply does not know what FAM is achieving. In addition, the Office perceives that the PVOs do not credit FFP for supporting their work. Recommendation: The Steering Committee (SC) should call a meeting to openly explore FFP’s less-than-satisfied perception of FAM with the membership. The SC should then meet with FFP in an open dialogue conducted in a frank but ‘appreciative’ manner, and to agree on concrete steps for ameliorating the situation. Lesson Learned: Improved FFP-FAM Communications. Communications between FAM and FFP are perceived by both parties as less than adequate. A continuation of these perceptions can only lead to a deterioration of the relationship. Recommendation: FAM Coordinators and one or two locally-based SC members should organize and hold regular meetings with USAID/FFP in order to help improve the relationship, to ensure that FAM is fulfilling FFP’s needs, and to report on progress. The discussion might include the provision for a PVO-designated person seconded to FFP to act as a liaison with FAM and member PVOs. Additional ways of enhancing interaction between FAM and FFP would be that FFP staff participates at least periodically in FAM WGs and in WG-organized workshops held in the Washington, D.C. area. Lesson Learned: Recognition of FAM Services. FAM services are not equally recognized or used across the membership, including their field staff. Recommendation: FAM members, in cooperation with FAM staff, should develop an initiative to inform their staff and constituencies of the services available to them through FAM, so that they more readily recognize FAM’s role and contribution. Working Groups Lesson Learned: Using a results, team-building orientation. Working Groups (WGs) were seen to tire in the absence of a desired product or expected result. Even the more effective WGs, once a product had been completed, have sometimes lost their motivation. Recommendation: WGs should always attempt to agree as soon as is feasible on a ‘product,’ even if that product is a ‘process’ intended to get members to agree on a plan of action. Where feasible, this process should be facilitated by technical advisers on staff with member organizations, preferably with strong organizational—including team building—skills. Lesson Learned: Obtaining economies of scale from Working Groups. The Environmental Working Group (EWG) achieved a broad impact through its workshop on the training-of-trainers (TOT) in environmental compliance. The TOT approach not only expands the scale of the training function, providing younger staff with valuable training experience, but also broadens the spread of critical information, and contributes to the growth of the knowledge base. 38 Recommendation: The other WGs should strongly consider use of the TOT approach in planning their next workshop(s), where relevant. On a related matter, because of the significant success the EWG has had in fulfilling FAM objectives, the SC should request recognition of that Group from FFP, thereby formalizing its contributions and making it eligible for FAM funds. FSRC Lesson Learned: Placing the bibliographic database online. The process of incorporating the collection’s 8,000 documents and selected abstracts on the website has encountered several snags. While the recent acquisition of a pro bono information technology specialist is a step in the right direction and may produce the desired result, it is not ‘guaranteed.’ Recommendation: The Steering Committee needs to support FAM coordinators and the TIS to organize whatever resources are necessary and support completion of the on-line bibliographic database. Lesson Learned: Digitizing the FSRC library collection. Initial discussions have taken place on the possibility of digitizing key parts, if not all, of the FSRC collection. This collection represents a significant contribution to the state-of-the-art literature on food aid and food security. Its availability to food aid professionals worldwide would represent a large step forward in the evolution of the FSRC. Recommendation: The Steering Committee and FAM coordinators should speed up the search for a means (institutional and/or financial) to digitize the library. Lesson Learned: Some useful tips. Individual FAM members were almost all highly praiseworthy of the FSRC. A few had suggestions to even further enhance FAM management and use of the FSRC. Recommendation: On a monthly basis, the FSRC should ‘flash’ the membership a reminder that it still exists and is ready to serve, accompanied perhaps by a brief synopsis of one or two new state-of-the-art publications. FSRC should coordinate closely with the WGs to ensure that summaries of organized presentations are sent out and to identify WG key resource persons and materials. 39 Annex 1: Scope of Work Back to main document Scope of Work MID-TERM EVALUATION of FOOD AID MANAGEMENT Institutional Support Assistance Program 1.0 Background: In 1989 five U.S. private voluntary organizations (PVOs) created Food Aid Management (FAM) to “promote the efficient and effective use of food aid resources to help alleviate hunger and contribute to food security”. FAM was awarded a five￾year follow-on USAID ISA grant beginning in September 1998 to continue supporting P.L. 480 Title II-funded Cooperating Sponsors (PVOs) in their existing or planned activities. Eleven years later, while the goal of FAM remains the same, the FAM membership consortium has grown to include 17 U.S. based PVOs. FAM works with these CSs to achieve this goal by managing the following three objectives: • facilitate and promote the development of food aid standards, • promote the food aid and food security knowledge base of PVOs, USAID staff, and other collaborators through information exchange and coordination, • facilitate collaboration between PVOs, USAID, and appropriate development and humanitarian professionals by organizing fora of discussion. FAM does not implement P.L. 480 Title II food programs, its Cooperating Sponsor (CS) members do. FAM’s objectives and activities were designed to support the Office of Food for Peace’s Strategic Objective 2 (SO2): "Increased effectiveness of FFP’s Partners in carrying out Title II development activities with measurable results related to food security with a primary focus on household nutrition and agricultural productivity". FAM accomplishes its goals while focusing its efforts on activities that support the achievement of FFP’s Intermediate Result 1 (IR1): "Strengthened capabilities of PVOs, USAID Missions, and FFP to design, monitor, and support programs". As a consortium, FAM works closely with its members to define activities, and to actively promote the progress of activities to agreed-upon goals. These activities include the implementation of working groups collaborating on common themes, which are priorities for members, namely: Monitoring and Evaluation, Monetization, Local Capacity Building (LCB), and the Environment (EWG). FAM also manages the Food Security Resource Center (FSRC), publishes the Food Forum bulletin, maintains an active website, and implements several other food security information sharing activities including inter-organizational workshops. Measuring the links to how FAM activities directly or indirectly impact Title II food security programming is beyond the scope of FAM’s ISA objectives. As a non-implementing, information sharing, coordination body, FAM’s efficiency depends upon the consistent monitoring of activities directly related to FAM objectives. In the case of the FAM workshops, publications, and website FAM has set up mechanisms to track how the information is being disseminated, used, and potentially revised. By effectively managing and monitoring information exchange, collaboration, improved food aid standards, and the other capacity building activities by encouraging collaboration and information exchange among its 17 PVO members, FAM’s contribution to Title II programming, through support of FFP’s Objective and Intermediate Result is unique. The Title II DAP and FY99 ISA Guidance calls for a mid-term assessment of the project in the third year. The attached FAM M&E Chart and Indicator Performance Tracking Table (IPTT) list the objectives, activities, indicators, and their frequency of measurement for the life of the grant. 2.0 Purpose of the Mid-Term Evaluation: The overall purpose of the mid-term evaluation is to assess the progress in achieving planned results and to refine program activities and targets accordingly. The mid-term review should determine whether the program is being implemented in a timely and cost-effective way, and if the program's performance objectives are on track. 2.1 Focus and Key Questions to be Addressed by the Evaluation: Annex 1, Page 1 Specifically, the mid-term review should generate and use qualitative and quantitative information in order to: review appropriateness of the program activities with respect to the original ISA problem analysis; determine overall progress towards achieving targets; identify constraints, difficulties, and successes; refine targets if needed; and make relevant recommendations to improve the performance, or, as appropriate suggest modifications or discontinuance of activities. In addition to the above, the Consultant will constructively assess the following aspects of the FAM program and provide response to the following questions: • Management (efficiency, effectiveness, leadership, budget, etc. ). • Progress (rate and trend) and process in achievement of Objectives, Activities, and Performance Indicators- planned versus actual. • Are the program's targets appropriate?; i.e. have the intended success rates been under or over-estimated?; do the targets need revising and why? • What plans are in place or proposed to handle the identified constraints? • M&E system to ensure that all monitored indicators are being objectively measured, include specific measurement units, and have clear links to program activities. • The degree of change in internal institutional strengthening (improved capacity) of FAM as well as the CS members, as a result of FAM's program activities. • Has the intent of the capacity building focus changed as a result of the ISA?; what difference has the ISA made to FAM and the CS members? • Likelihood that intended benefits of FAM's ISA-funded program has been complemented by resources from the ISA recipients (CSs). • Collective accomplishments and efficiencies achieved because of collaboration efforts with CSs. • What best practices has FAM identified and have these been effectively distributed? • Sustainability of FAM activities and CSs capacity. 3.0 Methodology: The following methods should be used to evaluate the critical areas of concern noted above and to determine appropriate strategies to improve project implementation and impact: A. Prepare an implementation plan for the evaluation process. Discuss and agree on plan with FAM staff. B. Review FAM's: ISA proposal document, Annual Performance Reports, Detailed Implementation (DIPs) and Annual Operating Plans (AOPs), web site and Food Security Resource Center (FSRC), annual budgets and financial reports, M&E-related plan and data, web site use reports, working group and other meeting minutes, and other relevant information that indicate/record the status of FAM's activities and performance. C. If any FAM activities are planned during the review period, make direct observations of the activities to assess quality and impact. D. When choosing all samples to be interviewed and/or surveyed, consider and use (when appropriate), a "purposeful sample" selection criteria and process that will provide justification of sample choices. E. Communicate (in person, e-mail, or by phone) with key stakeholders, (i.e. FAM staff, CS members, steering committee members, working group chairs and members, FFP, etc.) to elicit and discuss their perspectives on FAM's progress, to identify concerns and possible areas of modification, and to assess the program's implementation process and achievement of goals. F. Review FAM workshop/training reports, participant evaluations, and pre/post tests where available. Tabulate and analyze the rate of achievement to-date (number and type of activities, participant attendance, etc.). Conduct mini￾surveys [through use of randomized spot checks (in person, e-mail, or by phone)] of former workshop/training participants and their organizations (to measure change in knowledge, skills, and practice resulting from such FAM capacity building events). By assessing the information gathered, determine the likely affect and trend of this achievement on intended program impact, i.e. to improve PVO capacity to design, monitor, and support Title II programs. G. Undertake visits to CS member headquarter offices as needed to discuss particular topics with key CS staff. Annex 1, Page 2 H. Facilitate focus group discussions with key stakeholders to obtain key-information as needed. I. As much data and responses should be compiled and documented for comparison with the baseline and for the final year evaluation. J. Conduct progress meeting once every two weeks with FAM staff in order to inform and consult on the mid-term review process. K. Review "Indicator Performance Tracking Table" (IPTT), assess indicators, and update for attachment to draft and final reports. L. Present and confer initial evaluation findings with FAM staff and other key FAM stakeholders. Prepare a draft report for review and feedback by FAM staff and key FAM stakeholders. M. Prepare final report with logical and justifiable recommendations. 4.0 Time Schedule: The time frame envisaged for the evaluation process is 4 weeks (20 working days). The review will commence in March 2000. 5.0 Deliverables: A draft report prepared within one week of completing the review work. Final Report submitted by the Consultant to USAID and FAM (lazer printed hard copy and on a 3.5 diskette in Word) within two weeks of completing the review work. 6.0 Evaluation Team: The evaluation process will be led by one Consultant who will have extensive experience and/or understanding of Title II food programming. In order to make the Mid-Term Evaluation more inclusive, participatory, and a productive learning experience for FAM, the Consultant will work closely during the review process with the FAM staff. CS member and FFP staff, working group chairs, and steering committee members will directly assist the Consultant where appropriate and available. Annex 1, Page 3 Annex 2: Persons Contacted/Interviewed Back to main document Abdou, Irene – Steering Committee (Counterpart) Ameyaw, David – Monitoring & Evaluation Working Group (ADRA) Anaele, Sabinus – Environmental Working Group (TechnoServ) Bell, Bob – Steering Committee (CARE) Campbell, Ben – Environmental Working Group ((World Vision) Carlson, Gail – Steering Committee (Counterpart) Crumbly, Angelique – Cognizant Technical Officer, USAID/FFP Davis, Tom – Consultant, Monitoring & Evaluation Working Group Devendorf, George – Steering Committee (Mercy Corps) Elkin, Barry – Environmental Working Group, Steering Committee (ACDI-VOCA) Evans, Dave – Steering Committee (FHI) Ewart, Tom – Steering Committee Mercy Corps) Gilbert, Steve – Mendez-England (contractor to FFP) Harrigan, Paige – Acting Coordinator (Fmr.—presently FANta) Majernik, Juli – Monetization Working Group (CRS) Markunas, Jeanne – Deputy Director, USAID/FFP Ng’asi, Adrian – Local Capacity Building Working Group (ACDI-VOCA) Pareja, Mario – Environmental Working Group (CARE) Purviance, Randy – Steering Committee (ADRA) Truscott, Peter – Steering Committee (ADRA) Russell, Mara – Coordinator FAM Ryan, T. J. – Coordinator FAM (Fmr.) Scheffel, Dorothy – Monitoring & Evaluation Working Group (World Vision) Schmirler, Trisha – Technical Information Specialist FAM Swindale, Anne – Monitoring and Evaluation Working Group (FANta) Willard, Alice, Local Capacity Building Working Group (FANta) Wright, Keith – Steering Committee (FHI) Zodrow, Steve – Deputy Coordinator FAM Annex 2, Page 1 Annex 3: Documents Reviewed Back to main document Christian Reformed World Relief Committee. Partnering to Building and Measure Organizational Capacity. Lessons from NGOs Around the World. (Use of Appreciative Inquiry Approach to Organizational Capacity Building). 1997. Food Aid Management. Annual Performance Report. FY 00 ___________________. Annual Report: Food Security Resource Center FY 99/00 ___________________. “ “ Monitoring & Evaluation Working Group FY 00 ___________________. “ “ Monetization Working Group FY 00 ___________________. “ “ Local Capacity Working Group FY 00 ___________________. Meeting Notes, Environmental Working Group 11/29/00 ___________________ Evaluation of FHI Mentoring Agreement FY 00 ___________________. Monitoring &n Evaluation Formats: Results FY 00 ___________________. Notes from the FAM General Meeting Held in San Diego 6/15- 16, 2000. ___________________. By-Laws November 18, 2000 ___________________. Institutional Support Assistance (FY 99-03) & Detailed Implementation Plan (FY 00/01) ___________________. Cost Savings for Move (CARE Contribution) 1/01 ___________________. Annual Budgets (FY 99-03) ___________________. Annual Operating Plan FY 00 ___________________. Correspondence to USAID/FFP 12-11-00 ___________________. Member Survey 1/00 ___________________. History of FAM 1989-2000, PowerPoint Presentation, 6/00 ___________________. Food Forum 3d Qtr. 2000, Issue 53 ___________________. FSRC User Records FY99–01, 2d Qtr.(FAM/Non-FAM) 3/01 Annex 3, Page 1 Annex 4: Questionnaires/Topical Guides Devised for Use in the Mid-term Back to main document Questions for Key Informant Interviews with FAM Coordinator/Deputy Coordinator (3/23/01—jpm) 1. When did you join FAM? What was the staffing situation when you joined? 2. What expectations did you have when joining FAM about its role, your workload, and expectations from members, from USAID? What was the reality? 3. What has been the most satisfying aspect of your work? Least satisfying? 4. What does USAID expect from you? Member organizations? CARE? How able are you to meet their expectations? 5. What do you feel you and your staff do best? Least well? 6. What are the most successful results of FAM during your tenure here? Least successful? Why? What to do to improve the least successful results? 7. Rank in order what you believe represents the “best” of FAM: __ Working Groups, __ Workshops, __ FSRC (website, listserv, library, etc) __ Annual meetings, __ Brown Bag lunches, __ Steering Committee, etc. 8. Where do you feel FAM could use the biggest improvements? Define/Describe. 9. What are largest organizational constraints to FAM achieving its higher-level results? (e.g., financial— why? Etc.) 10. Which are your most difficult clients? Why? What do you do to meet their complaints or address what they believe are shortcomings? 11. What do you believe should be the ideal structure of FAM? 12. If you were to reorganize FAM, what kind of mechanism or instrumentality would it look like? 13. Would you like to see FAM renewed? What do you need to do to ensure that? Annex 4, Page 1 Food Aid Management/Institutional Support Assistance (FAM/ISA) Cooperative Agreement Mid-Term Evaluation—FAM Environmental Working Group Questionnaire (3/21/01—jpm) This questionnaire is one of a series of instruments aimed at assessing the FAM/ISA program activity. Your Working Group’s contribution is important to the overall functioning of FAM. The questions are a mix of past performance/expected results and touch on the effectiveness of FAM for both your Working Group and your organization. Please respond to the extent practicable in the space provided, and return to me (John Mason, Mid￾term Evaluation Consultant) by return email by COB TUESDAY 27 MARCH. Thank you very much for your cooperation. 1. Which specific aspects of the Environmental Working Group’s efforts has your organization been participating in? What are your organization’s roles and responsibilities in this effort? Define 1-2 efforts a) b) 2. What are two-three specific applications of the Environmental Documentation Manual (EDM) by end-users (in your response, please denote which end-users)? Define 2-3 applications. a) b) c) 3. What improvements in your organization’s food aid/security program have resulted/will result from application of environmental compliance practices embodied in the EDM and workshop training? Define 1- 2 ways. a) b) 4. More specifically, how has the EDM and workshop training influenced your organization’s management of your food aid/security programs? If applicable, define 1-2 specific ways. a) b) 5. How has the Environmental Working Group’s efforts contributed to your organization’s implementation of more targeted, people-level food aid/security programs? Define 1-2 ways. a) b) 6. What are some best practices that derive from the process of developing the manual, its use in training workshops, and its field applications? Name and define 1-2. a) b) 7. Name some specific examples of how a) the Working Group and b) your organization’s home and field program managers use/have used the FAM Food Security Resource Center (including the website, DEVECOL database, hard copy documents, other?) What are FSRC’s benefits to them? Name and define 1-2 for each. a) b) 8. Ho well has your Working Group functioned in achieving its intended results? What are its strong points/weak points? How might you change the functioning of the Group to make it more responsive to your organization’s environmental activities? In what related areas would you like to see new topics introduced? Define 1-2 points for each. a) b) c) 9. What is the value added of the FAM/ISA to the Environmental Working Group’s initiatives and efforts? Where has FAM/ISA perhaps been “less than optimal?” How, if at all, might the FAM/ISA be redesigned to meet the ongoing challenges of the Title II program? Define 1-2 points for each. a) b) c) I APPRECIATE YOUR HELP AND COOPERATION. Regards, John Mason Annex 4, Page 2 Mid-Term Evaluation – Questionnaire for the General Membership – Food Aid Management/Institutional Support Assistance Agreement (3/13/01) 1. In what specific areas of food aid and food security would you say the effects of FAM’s work have been most felt? Name one or two areas, with a brief explanation. a) b) 2) How has FAM specifically supported improved measurement of Title II impact on food security? List a few ways. a) b) 3) In what areas of food security should FAM emphasize capacity building? In a) enhancing national and affiliate staff development or b) building grassroots capacity at the local, community level? Briefly explain. a) b) 4) How has FAM contributed to the impact of your organization’s programs on food security (e.g., on sustainability vs. creating dependency)? Briefly describe 1-2 contributions. a) b) 5) How do you see FAM contributing to the establishment of food aid standards used by your organization in employing PL 480/Title II resources? Describe a few contributions. a) b) 6) How has FAM-supported training contributed to your staff’s implementation of programs—for both HQ and the field? In what areas of expertise could you use more training? Describe one or two specific ways/areas. a) b) 7) How would you make the resources of the FAM website and listserv more accessible and user-friendly to your organization’s field staff? Define one or two ways. a) b) 8) Are you evaluated by your organization specifically on your work for FAM? __ Yes. __ No. a) If yes, list a few of the criteria. B) If no, why not? a) b) 9) What are some related arrangements such as FANta/FAM that FAM might leverage to contribute to the state of the art in food security in the remaining year and a half of the present ISA? Describe a few. a) b) 10) What are some specific ways in which the Working Groups have served a useful purpose for your organization? Are there new purposes to which you would put the Working Groups? a) b) 11) What is the effect of USAID’s R4 (“results review/resource request”) requirement on your organization’s contribution to FAM? How does it affect your program management? Describe. a) b) 12) What are some of the more memorable Best Practices that derive from Working Groups or other FAM activities? List briefly. a) b) 13) What do you see as possible new mechanisms for a post-2003 FAM/ISA? How different would it be from the present agreement? Explain a) b) 14) In what ways is the “whole” of FAM organizationally greater than the sum of its parts? Less than? a) b) Annex 4, Page 3 FAM Local Capacity Building (LCB) Working Group Questionnaire (3/20/01—jpm) 1. Which FAM-supported LCB efforts has your organization actively participated in? What roles and responsibilities has your organization taken in the Group? Define and discuss 1-2 efforts/roles. a) b) 2. What progress has been made by the Working Group’s in operationalizing a set of practical definitions of local capacity? Towards completing a survey of member field offices to determine approaches, tools, resources and measurements? Define 1-2 steps and 1-2 achievements of the survey. a) b) 3. What applications of your organization’s own capacity building process to improved food aid/security needs have resulted from the LCB Working Group? How effective have these been. What are some of the constraints? Define 1-2 for cases. a) b) c) 4. What has the LCB Working Group contributed to your organization’s know-how in implementing more targeted, people-level food aid/security programs? Define 1-2 examples? a) b) 5. What if any improvements in your organization’s program management can you attribute to FAM￾supported LCB Define 1-2. a) b) 6. How useful was the LCB Spectrum Activity in helping your organization to think about different ways to improve targeting of Title II programs? What areas of LCB benefited more so than others and what are some of the constraints to LCB uncovered by the Spectrum Activity? Briefly define 1-2 cases of such thinking and 1-2 for benefits/constraints.. a) b) 7. What are the trade-offs of your organization’s own capacity building needs (e.g., management and staff) with those of partner and community needs? What are some constraints to focusing on only LCB needs versus your organization’s own “LCB” needs? List 1-2 trade-offs and constraints. a) b) 8. What are some examples of best/less-than-successful practices derived from LCB workshops and other fora? What progress has been made in establishing an LCB curriculum? In developing LCB measures (indicators). Define 1-2 examples of practices, progress, and measurement. a) b) c) 9. What are some specific uses the LCB Working Group and your organization have made of the FAM website? How useful have you found the website for facilitating your Group’s work? Define 1-2 examples/uses. a) b) 10. How well has your Working Group functioned? What are its strong points/weak points? How might you change the functioning of the Working Group to make it more responsive to your and fellow member organization needs? Define 1-2 cases a) b) 11. What are some specific benefits of the FANta/FAM Technical Advisory Group to your organization’s LCB activities? In what areas would you like to see new or related topics introduced? Define 1-2 benefits and 1- 2 new/related topics. a) b) Annex 4, Page 4 FAM M&E Working Group Questionnaire (3/14/01—jpm) 1. Which FAM-supported M&E efforts has your organization actively participated in? What has such participation involved as far as roles and responsibilities? Define and discuss 1-2 efforts/roles. a) b) 2. What progress has been made by the Working Group’s development of Title II M&E tools? In developing user manuals? Define 1-2 areas. a) b) 3. What applications to your organization’s M&E food aid/security needs have resulted from FAM-supported M&E? How effective have these been. What are some of their constraints? Define 1-2 cases. a) b) c) 4. How have FAM-supported M&E tools contributed to your organization having more targeted, people-level food aid/security programs? Define 1-2 ways? a) b) 5. What if any improvements in your organization’s program management can you attribute to FAM￾supported M&E? Define 1-2. a) b) 6. How effectively have FAM-supported M&E tool kits transferred to use in your field programs? How user friendly are they to field staff? Briefly define. a) b) 7. How useful have indicators identified under the FAM-supported M&E Working Group been in measuring progress of your organization’s food aid/security programs? What areas of measurement require more work? Briefly define 1-2 cases. a) b) 8. What effort has the M&E Working Group made to incorporate end-user communities into the M&E process? What efforts are being made to do so in the future? Define 1-2 efforts for each. a) b) 9. How useful have the FAM-supported M&E workshops been to your organization’s M&E efforts? What were some especially strong points participants took away from the workshops? What needs strengthening in M&E workshops? Define 1-2 examples. a) b) 10. What are some specific uses the M&E Working Group has made of the FAM website? How useful have you found the website for facilitating your Group’s work? Define 1-2 examples/uses. a) b) 11. How well has your Working Group functioned? What are its strong points/weak points? How would you change the functioning of the Working Group to make it more responsive to your organization’s needs? Define 1-2 cases a) b) 12. What are some specific benefits of the FANta/FAM Technical Advisory Group to your organization’s M&E activities? In what areas would you like to see new topics introduced? Define 1-2 cases. a) b) Annex 4, Page 5 Food Aid Management Cooperative Agreement Mid-Term Evaluation Questionnaire for Monitoring & Evaluation Workshop Participants (Sampling Workshop—Westminster, MD 5/99; Data Analysis Workshop, Nairobi 11/00) This questionnaire is one of a series of instruments aimed at assessing the FAM Cooperative Agreement. Please reflect on the effectiveness of the Workshop specifically for its applications to your organization’s Title II program activities. You have been randomly selected to participate in this brief survey. Your responses are confidential. Please respond to the extent practicable in the space provided, and return (to John Mason, Mid-term Evaluation Consultant) by email, Close of Business, MONDAY, APRIL 2. Specify which Workshop you attended: Westminster__ Nairobi__ Thank you very much for your cooperation. . 1. What is your specific role and responsibility within your organization for monitoring and evaluation (M&E) activities? Briefly define 1-2 roles/responsibilities a) b) 2. What were the learning points from the Sampling/Data Analysis workshops most relevant to your work in Title II activities? Briefly, list 1-2. a) b) 3. How useful did you find the M&E tools used in the workshops? How useful were they as practical, applied tools in your work in the home office or in the field? Define 1-2 points for each. a) b) 4. What applications of workshop learning points have been most useful in improving your organization’s food aid/security program? Least useful? Define 1-2 examples for each. a) b) 5. How have the workshop training and M&E methods and techniques specifically influenced your organization’s management of your food aid/security programs? Define 1-2 specific ways. a) b) 6. How, if at all, have the workshops and tools acquired contributed specifically to your organization’s implementation of more targeted, people-level food aid/security programs? Define 1-2 ways. a) b) 7. What are some best practices that you derived from the workshop? Name and define 1-2. a) b) 8. To what extent did the workshop familiarize you with the FAM Food Security Resource Center (including the website, Monetization listserv, hard copy documents, other)? Because of that, how frequently do you use FSRC? Define 1-2 ways/How frequently. a) b) 9. Overall, how would you evaluate the workshop and the accompanying tools for their effectiveness and practicality in supporting your Title II work? Briefly define. YOUR HELP AND COOPERATION ARE APPRECIATED. Regards, John Mason Annex 4, Page 6 FAM Monetization Working Group Mid-Term Evaluation Questionnaire 3/19/01—jpm) This questionnaire is one of a series of instruments aimed at assessing the FAM program activity. Your Working Group’s contribution is important to the overall functioning of FAM. The questions are partly retrospective, partly prospective. Please respond to the extent practicable in the space provided, and return to me (John Mason, Mid-term Evaluation Consultant) by return email by COB FRIDAY 23 MARCH. Thank you very much for your cooperation. 1. Which specific aspects of the Monetization Working Group’s efforts has your organization been participating in? What are your organization’s roles and responsibilities in this effort? Define 1-2 efforts a) b) 2. What are two-three specific applications of the monetization manual by end-users (please define who end￾users are)? Define 2-3 applications. a) b) c) 3. In what ways has application of the monetization manual by your organization resulted in more effective food aid/security programs? Define 1-2 ways. a) b) 4. How far along is the development of a training module to accompany the manual? What are the results of tests of the manual in workshops? Define how far and 1-2 results. a) b) 5. How has the manual and workshop training influenced your organization’s management of your food aid/security programs? If applicable, define 1-2 specific ways. a) b) 6. How has the monetization effort of the Working Group contributed to your organization’s implementation of more targeted, people-level food aid/security programs? Define 1-2 ways. a) b) 7. What are some best practices that derive from the process of developing the manual, its use in training workshops, and its field applications? Name and define 1-2. a) b) 8. How do your organization’s home and field program managers use the Monetization ListServ? What are its benefits to them? Define 1-2 for each. a) b) 9. How well has your Working Group functioned in achieving its results? What are its strong points/weak points? How might you change the functioning of the Group to make it more responsive to your organization’s monetization activities? In what related areas would you like to see new topics introduced? Define 1-2 points for each. a) b) c) Annex 4, Page 7 FAM Steering Committee Questionnaire (3/27/01—jpm) 1. How long have you been on the Steering Committee? 2. What is your role on the Steering Committee? How does that function? 3. How would you describe the function of the Steering Committee? 4. What are some of the major issues that drive the Steering Committee? 5. How does the Steering Committee operate? How are meeting agendas set? How are decisions made? 6. How well do the new by-laws function in giving direction to the Steering Committee? Why did it take so long to develop the by-laws? What advantage do they have for some of the smaller PVOs? 7. What are some of the constraints on the Steering Committee and how it works? 8. What are the major benefits of the Steering Committee? 9. What direct effects does the Steering Committee have on FAM’s agenda/operations? 10. How would you describe the major thrust of FAM? What are the most important things it does? 11. What is FAM’s value-added? In what ways does it add up to a whole that is greater than the sum of its parts? 12. What are some of the important constraints to FAM achievements? 13. What do you see as FAM’s importance to USAID? 14. Where do you think improvements in FAM can be made? 15. What relationship do you, as a member of the Steering Committee, have with USAID/FFP officials? Specifically with the USAID COTAR? 16. What do you feel the prospects are for the continuation of FAM under a new agreement? 17. What efforts have the Steering Committee made to diversify the funding sources of FAM? How important is this for FAM or an organization such as FAM? Annex 4, Page 8 FAM Working Group Focus Group Questionnaire (jpm/Mar. 22, 2001) 1. What is the major purpose of your WG? 2. How was the purpose of the WG generated? 3. What kind of cooperation occurs among members? 4. How are WG members selected? 5. How is the WG leader(s) chosen? 6. How are WG meeting agendas set? 7. How are meetings called? Who calls them? 8. How is the division of labor decided? 9. How do you arrive at decisions? 10. Is WG membership seen as a privilege/drag? 11. Is it seen as an extra part of the member’s job? 12. Are WG members evaluated by their org. for participation in WGs? How? 13. Why do some WGs seem to be ahead of others in their work/progress? 14. Is there competition among orgs. In the WGs? If yes, what is that like? 15. How well is the purpose of the WG understood by your org’s sr. management? 16. Does you sr. mgt. see the importance of the WGs to the org’s overall goals/objs? In what ways? 17. What interactions does the WG have with FAM coordinators? How often? Quality of? 18. What does the FAM/FoodSecResCtr contribute to your WG? 19. What leadership role does FAM play for the WG? 20. How does the WG reflect overall FAM goals/ objs? What does it contribute to the achievement of those goals? 21. What if any is the role of FANta in your WG? 22. How does FANta make its inputs? 23. What is the value-added of FANta? 24. What is the relationship bet. The WG and FFP? 25. How does the work of your WG reflect the goals & objs of FFP? 26. Where are the impacts of your WG most felt? At what level of food aid/security program impact? Annex 4, Page 9 Food Aid Management Cooperative Agreement Mid-Term Evaluation Questionnaire for Local Capacity Building Workshop Participants (Philadelphia—November 1999) This questionnaire is one of a series of instruments aimed at assessing the FAM Cooperative Agreement. Please reflect on the effectiveness of the Workshop specifically for its applications to your organization’s Title II program activities. You have been randomly selected to participate in this brief survey. Your responses are confidential. Please respond to the extent practicable in the space provided, and return to me, John Mason, Mid-term Evaluation Consultant) by email, Close of Business, WEDNESDAY 28 MARCH. Thank you very much for your cooperation. . 1. What is your specific role and responsibility within your organization for local capacity building (LCB)? Briefly define 1-2 roles/responsibilities a) b) 2. What were the learning points from the LCB workshop most relevant to your work in Title II activities? Briefly, list 1-2. a) b) 3. How useful did you find the LCB Spectrum Activity as a workshop tool? How useful do you believe it will be in the application to a) field and b) home office programs? Define 1-2 points for each. a) b) 4. What applications of workshop learning points have the most potential for building local capacity in your organization’s food aid/security program? Least useful? Define 1-2 examples for each. a) b) 5. In what ways do you see the workshop training specifically influencing your organization’s management of your food aid/security programs? Define 1-2 specific ways. a) b) 6. How, if at all, do you see the workshop training contributing to your organization’s implementation of more targeted, people-level food aid/security programs? Define 1-2 ways. a) b) 7. What are some best practices that you derived from the workshop? Name and define 1-2. a) b) 8. To what extent did the workshop familiarize you with the FAM Food Security Resource Center (FSRC) (including the website, listserv, hard copy documents, other)? If you received some familiarity in FSRC because of the workshop, how frequently do you use FSRC? Define 1-2 ways/How frequently. a) b) 9. Overall, how would you evaluate the workshop for effectiveness and practicality in supporting your Title II work? Briefly define. YOUR HELP AND COOPERATION ARE APPRECIATED. Regards, John Mason Annex 4, Page 10 Food Aid Management Cooperative Agreement Mid-Term Evaluation Questionnaire for Monetization Workshop Participants (India, Oct. 2000/Peru, Jan.-Feb. 2001) This questionnaire is one of a series of instruments aimed at assessing the FAM Cooperative Agreement. Please reflect on the effectiveness of the Workshop specifically for its applications to your organization’s Title II program activities. You have been randomly selected to participate in this brief survey. Your responses are confidential. Please respond to the extent practicable in the space provided, and return to me, John Mason, Mid-term Evaluation Consultant) by email, Close of Business, WEDNESDAY 28 MARCH. Thank you very much for your cooperation. . 1. What is your specific role and responsibility within your organization for monetization activities? Briefly define 1-2 roles/responsibilities a) b) 2. What were the learning points from the Monetization workshop most relevant to your work in Title II activities? Briefly, list 1-2. a) b) 3. How useful did you find the Monetization Manual (MM) as a workshop tool? How useful is it as a practical, applied tool in your work in the field? Define 1-2 points for each. a) b) 4. What applications of workshop learning points have been most useful in improving your organization’s food aid/security program? Least useful? Define 1-2 examples for each. a) b) 5. How has the workshop training and MM specifically influenced your organization’s management of your food aid/security programs? Define 1-2 specific ways. a) b) 6. How, if at all, has the workshop training and the application of the MM contributed to your organization’s implementation of more targeted, people-level food aid/security programs? Define 1-2 ways. a) b) 7. What are some best practices that you derived from the workshop? Name and define 1-2. a) b) 8. To what extent did the workshop familiarize you with the FAM Food Security Resource Center (including the website, Monetization listserv, hard copy documents, other)? Because of that, how frequently do you use FSRC? Define 1-2 ways/How frequently. a) b) 9. Overall, how would you evaluate the workshop and the MM for their effectiveness and practicality in supporting your Title II work? Briefly define. YOUR HELP AND COOPERATION ARE APPRECIATED. Regards, John Mason Annex 4, Page 11 Food Aid Management Cooperative Agreement Mid-Term Evaluation Questionnaire for 216 Regulation Training of Trainers Workshop Participants (Washington, D.C.—September 11-14, 2000) This questionnaire is one of a series of instruments aimed at assessing the FAM Cooperative Agreement. Please reflect on the effectiveness of the Workshop specifically for its applications to your organization’s Title II program activities. You have been randomly selected to participate in this brief survey. Your responses are confidential. Please respond to the extent practicable in the space provided, and return to me (John Mason, Mid-term Evaluation Consultant) by email, Close of Business, TUESDAY 27 MARCH. Thank you very much for your cooperation. . 1. What is your specific role and responsibility within your organization in applying the 216 Regulation? Briefly define 1-2 roles/responsibilities a) b) 2. Have you served as a trainer in giving the workshop to others? __Yes __No. a) If yes, how may times, to what participants, and with what success? b) If no, why not? a) b) 3. What were the learnings from the 216 Regulation workshop most relevant to your work in Title II activities? Briefly list 1-2. a) b) 4. How useful did you find the Environmental Documentation Manual (EDM) as a workshop tool? How useful is it as a practical, applied tool in your work in the field? Define 1-2 points for each. a) b) 5. What applications of workshop learnings have been most useful in improving your organization’s food aid/security program? Least useful? Define 1-2 examples for each. a) b) 6. How has the workshop training and EDM specifically influenced your organization’s management of your food aid/security programs? Define 1-2 specific ways. a) b) 7. How, if at all, has the workshop training and the application of the EDM contributed to your organization’s implementation of more targeted, people-level food aid/security programs? Define 1-2 ways. a) b) 8. What are some best practices that derive from the workshop? Name and define 1-2. a) b) 9. To what extent did the workshop familiarize you with the FAM Food Security Resource Center (including the website, DEVECOL database, hard copy documents, other)? Because of that, how frequently do you use FSRC? Define 1-2 ways/How frequently. a) b) 10. Overall, how would you evaluate the workshop and the EDM for their effectiveness and practicality in supporting your Title II work? Briefly define in a few words. I APPRECIATE YOUR HELP AND COOPERATION. Regards, John Mason Annex 4, Page 12 FAM-FANta M&E Working Group Questionnaire (3/27/01—jpm) 1. How long have you been associated with the FAM M&E Working Group? What has your role in that Group been? 2. What particular FAM M&E tools and other products have you directly contributed to? What was your contribution? 3. How has FAM-FANta specifically supported the M&E workshops delivered so far? What impact have these had on member organizations M&E plans and their application to the actual monitoring of food security? 4. How do you see these tools and products contributing to improved effectiveness of food aid/security as this is achieved by FAM member field applications? In contributing to fulfillment of USAID/FFP requirements? 5. In what ways have the M&E tools and other products you’ve contributed to added to improved targeting of food resources? Describe. 6. How much improvement do you see in FAM members’ management of Title II programs because of enhanced M&E practices? Define and describe. 7. To what degree have FAM-FANta-supported M&E toolkits been transferred to program staff in home offices and among field staffs? What are some examples of effective transfer? 8. How useful have the indicators identified under FAM-supported M&E Working Group been in measuring progress of member organizations’ food aid/security programs? 9. How effective has the M&E Working Group functioned in defining and meeting its agenda? How satisfied have you been with the role you have played in the Working Group? Define. 10. What are the strengths and weaknesses of the M&E Working Group from your perspective? What would you do to strengthen the Group and its products? Annex 4, Page 13 Annex 5: Indicator Performance Tracking Table (Summary of some key FAM activities) Back to main document FAM Goal: to promote the efficient and effective use of food aid resources to alleviate hunger and contribute to food security. Modified December 2000 INDICATOR FY99 TARGET FY99 ACHIEVED FY99 ACHIEVED VS TARGET FY00 TARGET FY00 ACHV’D FY00% ACHV’D VS TARGET FY01 TARGET FY01 ACHV’D FY01% ACHV’D VS TARGET FY02 TARGET FY02 ACHV’D FY02% ACHV’D VS TARGET ETC. LOA TARGET LOA ACHVD *Targeted section of Monitoring and Evaluation completed. FAM tracking of autonomy Satisfactory Completion of of Review Ag. submitted to Bio-physical Review Final draft M&E Working Group Objective met Satisfactory Completion of Health and Nutrition component. *Autonomy: HN Draft completed Re: Ag Bio￾physical toolkit: rejected by group, to be completed FY 01 Submission of Annual Report 90% complete d 11/01 Objective Delayed to FY01 Objective met (S/MED Component) *Target changed: Monitoring Toolkit Ag Bio￾physical toolkit M&E Toolkit completed Obj. Expected To be Met 9/01 Delayed To 12/01 NRM - Safety Net (Tbd) ETC. Completion of the M&E toolkit Annex 5, Page 1 INDICATOR FY99 TARGET FY99 ACHIEVED FY99 ACHIEVED VS TARGET FY00 TARGET FY00 ACHV’D FY00% ACHV’D VS TARGET FY01 TARGET FY01 ACHV’D FY01% ACHV’D VS TARGET FY02 TARGET FY02 ACHV’D FY02% ACHV’D VS TARGET ETC. LOA TARGET LOA ACHVD *Targeted section of Monetization* Working Group Manual/ Training Module completed or updated. FAM tracking autonomy Satisfactory Completion of jointly produced Monetiz. Manual Monetization Manual Completed and posted On FAM website Manual Finalized, Continued development Of Training Module to accompany manual Training module developed and field tested at Joint - PVO monetization workshop. Annual updates to manual. *Annual Report submitted T Training Module drafted and field tested at two workshops. Updates proposed Annual Report submitted Objective met Objective Partially Met Objective Achieved *Training Module & Training Curricula completed *Manual chapters 1-3, 5 translated to French & Spanish, * FHI mentoring in distance learning & communicatio n *Other issues￾based activities/meet -ings (min. 4) Asia and Latin America Workshops *Group Autonomy Drafts By Africare, CRS Near Comple￾tion Objective Met Objective Met Tbd ETC. Satisfactory completion of Monetization Manual, related training modules, and workshops, Improved TII PVO skill base in Monetization. Annex 5, Page 2 INDICATOR FY99 TARGET FY99 ACHIEVED FY99 ACHIEVED VS TARGET FY00 TARGET FY00 ACHV’D FY00% ACHV’D VS TARGET FY01 TARGET FY01 ACHV’D FY01% ACHV’D VS TARGET FY02 TARGET FY02 ACHV’D FY02% ACHV’D VS TARGET ETC. LOA TARGET LOA ACHVD Annual increase in information requests (FSRC￾monitoring) FY98 Baseline: 212 request FY98 Baseline 212 Target = 233 243 requests 14.6% increase Target = 256 146 requests 38% decrease (poor data manage￾ment, actual decrease 2 Minimum 45% increase recorded.3 Correct tracking procedures used. Objective on course to be met Objective met Target = 310 ETC. FY03 = 341 Continued increase of 10% usage each year of the grant Total = 1422 Members and others report FSRC provided them with the information they require. Measurement based on survey taken at time of use. 2 The Technical Information Specialist position at FAM was vacant during the entire year, meaning there was a fall off in marketing activities, and thus possibly requests. However, it also meant that data were not tracked appropriately, and some data was lost (see FSRC report). However, there was also indication that use of on-line documents posted to the FAM website increased, so this may have reduced FSRC usage to some extent. 3 This increase will allow achievement of the targeted 10% annual increase, making up the difference from the FY00 decrease. Annex 5, Page 3 INDICATOR FY99 TARGET FY99 ACHIEVED FY99 ACHIEVED VS TARGET FY00 TARGET FY00 ACHV’D FY00% ACHV’D VS TARGET FY01 TARGET FY01 ACHV’D FY01% ACHV’D VS TARGET FY02 TARGET FY02 ACHV’D FY02% ACHV’D VS TARGET ETC. LOA TARGET LOA ACHVD Annual increase in Field staff requests FY98 Baseline 17% Target = 21% At least 26% Of requests From field 5% over target 19-25% increase from the previous year (- average target set at 20% increase)= 25% 23% of requests from field 2% under target (poor data manage￾ment, so this may have been higher) Target 30% field usage Objective on course to be met Target 36% field ETC. FY03 43% Continued increase of approx 20% each year in requests from the field (from the previous year) To total 43% at the end of the grant. Field staff report FSRC provides information they require. Measurement based on survey taken at time of use. Annex 5, Page 4 INDICATOR FY99 TARGET FY99 ACHIEVED FY99 ACHIEVED VS TARGET FY00 TARGET FY00 ACHV’D FY00% ACHV’D VS TARGET FY01 TARGET FY01 ACHV’D FY01% ACHV’D VS TARGET FY02 TARGET FY02 ACHV’D FY02% ACHV’D VS TARGET ETC. LOA TARGET LOA ACHVD Expansion of FAM website4 January 1999 Avg. # number of hits per day = 62 # of hits on website/broken down by webpage reported on a quarterly basis. 5% increase per quarter of staff using site to improve quality of food security programs. improvements to website based on actual use/needs # hits on website/broke n down by quarter No data on purpose of use (web-based survey during FY01) Net increase of Obj. use in FY 00: 20%??? Website redesign in FY 01 Obj. Achiev’d Info not available Achiev’d Post￾poned to FY 01 # hits on website/broke n down by month webpage reported quarterly 5% increase per quarter of staff using site Site-based survey developed to assess purpose of site use Min. 30% use by staff to improve food security programs Website redesign Objective met Objective exceeded Objective met Insuffi￾cient data available to date to determine5 Objective met ETC. Average number of hits/day increases by 5% per quarter. 70% of PVO users consult the website to obtain information to improve food security programs. 85% of these users access the information they are looking for. Portions of website not used are deleted, while portions of website used are improved so as to meet user requirements. Measurement based on information from WebTracker, and periodic user surveys posted to site. 4 FAM has a mentoring agreement with Food for the Hungry International. FHI generated a report on FAM website usage in January 1999, and is doing so again in January 2000. In FY2000 software will be purchased that will enable FAM staff to monitor these trends more carefully and frequently. 5 Website survey was posted in March 2001, and to date, responses have been insufficient to determine this. Annex 5, Page 5 INDICATOR FY99 TARGET FY99 ACHIEVED FY99 ACHIEVED VS TARGET FY00 TARGET FY00 ACHV’D FY00% ACHV’D VS TARGET FY01 TARGET FY01 ACHV’D FY01% ACHV’D VS TARGET FY02 TARGET FY02 ACHV’D FY02% ACHV’D VS TARGET ETC. LOA TARGET LOA ACHVD Two FAM workshops organized per/year6 Two Successful workshops/ year with strong evaluations A total of 37 ISA funded workshops developed and Managed by FAM Objective Met Local Capacity Building And Workshop TBA 1) Central & Southern Africa Moneti￾ation Workshop (Johannesburg ) 2) LCB Annual Meeting Workshop 3) EWG Reg. 216 Training of Trainers held: Obj. Achiev’d 1) Monitoring and Evaluation Data Analysis Workshop (Nov. 2000) 2) Asia Monetization Workshop (Oct. 2000) 3) Latin America Monetization Workshop (Jan. 2001) Objective met Objective Met Objective met Local Capacity Building and/or Monitoring and Eval. ETC. FY 2003 Monet. At least 10 successful workshops through LOA. These will include 1 on Monetization (year 5); 2 M&E; 1 Commodity Management. 1 LCB 6 To improve follow-up after workshops, and to better determine the effect of the workshops upon the quality Title II programs, FAM staff and Working Group chairs are working together to develop feasible mechanisms to capture feedback related to the working groups. These feedback mechanisms are intended to improve monitoring of the impact and usefulness of FAM workshops, and documents developed by the working groups. 7 ISA funds supported the November 1998 FAM annual meeting that included a monetization workshop section. In December 1999, FAM with the EWG hosted a “Lessons Learned and Planning” workshop. The M&E working group with the FANTA project developed the May 1999 Sampling for Manager’s workshop. With funding from the SUSTAIN project, and collaboration with staff from USAID’s Global Bureau, Bureau of Humanitarian Response, the FANta project, the Congressional Hunger Center, and the Micronutrient Initiative of Canada, FAM staff organized the Enhancing the Nutritional Quality of Relief Diets workshop, which took place in April 1999. Annex 5, Page 6 INDICATOR FY99 TARGET FY99 ACHIEVED FY99 ACHIEVED VS TARGET FY00 TARGET FY00 ACHV’D FY00% ACHV’D VS TARGET FY01 TARGET FY01 ACHV’D FY01% ACHV’D VS TARGET FY02 TARGET FY02 ACHV’D FY02% ACHV’D VS TARGET ETC. LOA TARGET LOA ACHVD Specific element targeted for LCB working group achieved/year. Tracking of autonomy LCB Background paper prepared for FAM Members, as preparation for FAM Annual Mtg. Paper completed satisfactorily, Objective met. Working plan for working group finalized and LCB Publication Outlined and drafted. Annual report submitted by WG chair. Group functions Autonomously Work plan completed Publication postponed Annual report submitted Group functions Autonomously Obj. Achiev’d Post￾poned Obj. Achiev’d Obj. Achiev’d LCB Paper on Measurement of Capacity Building completed. 50% due to be completed in FY01 LCB Mea￾surement Module completed Tbd ETC. Completion of LCB Manual. Annex 5, Page 7 INDICATOR FY99 TARGET FY99 ACHIEVED FY99 ACHIEVED VS TARGET FY00 TARGET FY00 ACHV’D FY00% ACHV’D VS TARGET FY01 TARGET FY01 ACHV’D FY01% ACHV’D VS TARGET FY02 TARGET FY02 ACHV’D FY02% ACHV’D VS TARGET ETC. LOA TARGET LOA ACHVD Specific element targeted for EWG achieved/ year. (Starting with FY00) tracking of autonomy No target set FY1999 EWG work-plan outlined and implemented by members of the WG. CS workshops developed and hosted by members of the working group, Training modules drafted. FY00 Workplan Finalized. -draft training modules submitted. FY01 workplan outlined. Annual report submitted by WG chairs. Workplan finalized Reg 216 Training of Trainers conducted Product delivery late Obj. Achiev’d Activity fulfilled objective Obj. Post￾poned *Conduct 4 workshops on Reg. 216 and other Environmental topics *Development of regional training cadre *Annual report submitted by WG chair. 3 conducted to date 75% to date Objective Met ETC. Improved integration of Reg. 216 environmental principles into TII programming. (Through CS led workshops, training module development.) Annex 5, Page 8 INDICATOR FY99 TARGET FY99 ACHIEVED FY99 ACHIEVED VS TARGET FY00 TARGET FY00 ACHV’D FY00% ACHV’D VS TARGET FY01 TARGET FY01 ACHV’D FY01% ACHV’D VS TARGET FY02 TARGET FY02 ACHV’D FY02% ACHV’D VS TARGET ETC. LOA TARGET LOA ACHVD No fewer than 5 Food Aid/Food Security Discussions Per year Five meetings --RNIS, --CRG, --FFP ER Meeting, --CRS led PVO meeting A-133, --World Vision Food Aid Managers Objective met Five meetings -Center for Sustainable Agriculture --CHC, --FFP ER mtg --4 EWG Brownbags --6 LCB Brownbags --3 Monetization Issues meetings --3 FAM/FANTA /TAG topical meetings --SUSTAIN: MAP, preliminary CRG present. --Dried Bean Council --DEVECOL Total = 22 meetings held 440% *New target: fifteen meetings8 --2 LCB Brownbags --2 EWG Brownbags --1 USDA TQSA meeting --1 FAM/ FANTA Tag Meeting --1 SUSTAIN￾Presentatio n on web￾based CRG & MAP 53% of revised target reached to date Five meetings ETC. Majority of participants increase their knowledge/understanding of topical issues that enable them to plan & collaborate on food security initiatives. No fewer than 25 food aid/food security meetings held over the LOA. Annex 5, Page 9 INDICATOR FY99 TARGET FY99 ACHIEVED FY99 ACHIEVED VS TARGET FY00 TARGET FY00 ACHV’D FY00% ACHV’D VS TARGET FY01 TARGET FY01 ACHV’D FY01% ACHV’D VS TARGET FY02 TARGET FY02 ACHV’D FY02% ACHV’D VS TARGET ETC. LOA TARGET LOA ACHVD Annual Meeting Specific objective for each annual meeting met, per year. Overall objective will be increased collaboration, measured (as part of mid-term & final evaluations) as an index of the following: -Increase in number & effectiveness of partnership relationships & joint program initiatives - Increase in number & effectiveness of mentorship relationships - Increased sharing of resources - Increase in the number of jointly developed tools & methods All FAM working groups present annual progress reports. All working groups presented reports. Strong qualitative evaluation at end of meeting. Objective met All FAM working groups present Annual Progress reports at meeting. All working groups presented annual progress reports at annual working group meetings Objective Achieved All FAM working groups present Annual Progress reports at meeting. All FAM working groups present annual progress reports at meeting. ETC. As a result of attending annual meetings, collaboration improves among FAM members as evidenced by: - Increase in number & effectiveness of partnership relationships & joint program initiatives - Increase in number & effectiveness of mentorship relationships - Increased sharing of resources - Increase in the number of jointly developed tools & methods for use by membership An index measurement of these indicators is to be provided within the midterm and final evaluations. Annex 5, Page 10 INDICATOR FY99 TARGET FY99 ACHIEVED FY99 ACHIEVED VS TARGET FY00 TARGET FY00 ACHV’D FY00% ACHV’D VS TARGET FY01 TARGET FY01 ACHV’D FY01% ACHV’D VS TARGET FY02 TARGET FY02 ACHV’D FY02% ACHV’D VS TARGET ETC. LOA TARGET LOA ACHVD Mentoring partnership with FHI indicators met FAM Website managed FAM staff FAM website Managed internally Objective Met FAM listserv managed by FAM staff and usage increased Listserv managed by FAM staff No data on usage of listservs, however use of EWG listserv becomes insti￾tutionalized Obj. Achieved No data, but appears to have increased Mentoring Monetization working group (and others as requested) on development of distance learning options ETC. FAM staff manage and improve website independently. Annex 5, Page 11 Annex 6: FAM Collaborative Training Activities Back to main document FAM Collaborative Training Activities Institutional Support Assistance Grant: October 1998 – June 2001 FY 1999: Year 1 Title Dates Location Working Group Hosting Organization(s) # of Org’s; Participants Countries Represented Sampling for Managers 5/24-26/99 Westminster, MD M&E FAM Data not Available Headquarters and Field Monetization: West & Central Africa 7/12-15/99 Accra, Ghana Monetization Africare, ACDI-VOCA 9 Org’s 27 P’pants Cape Verde, USA, Uganda, Ghana, Guinea, Chad, Mali, Burkina Faso, Liberia, The Gambia, Ethiopia, W. Afr. Reg. FY 2000: Year 2 Title Dates Location Working Group Hosting Organization(s) # of Org’s; Participants Countries Represented Monetization: East & Southern Africa 11/2-5/99 Johannesburg, South Africa Monetization SCF, ACDI-VOCA, World Vision, CARE 9 Org’s 28 P’pants South Africa, USA, Ghana, Madagascar, Mozambique, Togo, Kenya, Malawi, Ethiopia, Angola, Uganda, Zimbabwe Local Capacity Building 11/8-10/99 Philadelphia, PA LCB FAM, Africare, ACDI￾VOCA 16 Org’s 32 P’pants For Headquarters Staff Annex 6, Page 1 Regulation 216 Training of Trainers Workshop 9/10-14/00 Washington, DC EWG USAID, FAM-EWG, Counterpart International 13 Org’s 27 P’pants Mozambique, Honduras, Ethiopia, Angola, Guinea, Guatmala, Rwanda, Sierra Leone, Bangladesh, Madagascar, India, Mid East/E. Eur. Reg, Kenya, Burundi, Ghana, Niger, Haiti FY 2001: Year 3 Title Dates Location Working Group Hosting Organization(s) # of Org’s; Participants Countries Represented Monetization: Asia 10/3-6/00 Dehli, India Monetization CARE 10 Org’s ~ 30 P’pants India, Bangladesh, USA, Indonesia, Pakistan, Kosovo Data Analysis 11/6-10/00 Nairobi, Kenya M&E FAM, ADRA 12 Org’s 26 P’pants Ethiopia, Kenya, Ghana, USA, Madagascar, So. Sudan, Uganda, The Gambia, Sudan, Mozambique, E. Afr. Reg. Monetization: Latin America and Caribbean 1/28-2/3/01 Lima Peru Monetization CRS 14 Org’s 47 P’pants Haiti, Guatemala, Honduras, Nicaragua, Peru, Bolivia, Guinea, USA Annex 6, Page 2 Annex 7: FAM Members Contribution to Food Forum Back to main document *Note that in FY99, IRD and Counterpart Int'l were not yet members. Food Forum was publishing bi-monthly in FY99. FY99 Issue 44 FY99 Issue 45 FY99 Issue 46 FY99 Issue 47 FY99 Issue 48 FY99 Issue 49 FY99 Issue 50 Total ADRA X 1 Africare X 1 ACDI-VOCA X 1 American Red Cross X 1 CARE X X 2 CRS X 1 Feed the Children 0 FHI X 1 Mercy Corps 0 OIC International 0 PCI X 1 Save the Children X 1 TechnoServe X 1 World SHARE 0 World Vision X X 2 *Note that for most of FY00, IRD was not yet a member. Food Forum published quarterly from FY00 on. FY00 Issue 51 FY00 Issue 52 FY00 Issue 53 FY00 Issue 54 Total ADRA X 1 Africare X 1 ACDI-VOCA 0 American Red Cross X 1 CARE X X 2 CRS X 1 Counterpart Int'l 0 Feed the Children 0 Annex 7, Page 1 FHI 0 Mercy Corps 0 OIC International X 1 PCI 0 Save the Children X 1 TechnoServe 0 World SHARE 0 World Vision X 1 Totals are estimated for FY01. FY01 Issue 55 FY01 Issue 56 FY01 Issue 57 FY01 Issue 58 Total (est.) ADRA sched. 1 Africare 0 ACDI-VOCA sched. 1 American Red Cross sched. 1 CARE 0 CRS 0 Counterpart Int'l X 1 Feed the Children 0 FHI X sched. 1 IRD 0 Mercy Corps 0 OIC International sched. 1 PCI X 1 Save the Children sched. 1 TechnoServe X 1 World SHARE 0 World Vision 0 Annex 7, Page 2 Annex 8, Page 1 Annex 8: Documents Produced by FAM under its ISA Back to main document Documents Produced by FAM in Fiscal Years 1999 - 2001 With Funding from ISA Award # FAO-A-00-98-00081-00 FY2001 (thus far, October 1, 2000 – May 31, 2001) Proceedings of the Latin American Regional Monetization Workshop (Hosted by CRS) January 28 – February 2, 2001, Lima, Peru Proceedings of the FAM/FANTA Data Analysis Workshop (Hosted by ADRA Kenya) November 6 – 10, 2000, Nairobi, Kenya - Materials from the FAM/FANTA Data Analysis Workshop (same as above) Proceedings of the Asia Regional Monetization Workshop (Hosted by CARE India) October 3 – 6, 2000, Delhi, India. Forthcoming. FY2000 (October 1, 1999 – September 30, 2000) Food Aid Lexicon – Second Edition Food Aid Management. 2000. Review of Health and Nutrition Project Baseline Research Methods of Title II Funded PVOs Haggerty, Patricia; FAM Monitoring & Evaluation Working Group. September 2000. Proceedings of the Regulation 216 Training of Trainers Workshop (Hosted by Counterpart International) September 11 – 16, 2000. Washington, DC. USA. - Materials from Regulation 216 Training of Trainers Workshop (same as above) A Cooperating Sponsor’s Field Guide to USAID Environmental Compliance Procedures. Burpee, Gaye; Harrigan, Paige; Remington, Tom; Catholic Relief Services; FAM Environmental Working Group. February 2000. Proceedings of the Local Capacity Building Workshop (Hosted by FAM) November 9-10, 1999. Philadelphia, PA. USA. Proceedings of the East and Southern Africa Monetization Workshop (Hosted by Save the Children, ACDI-VOCA, World Vision, and CARE) November 2- 5, 1999. Johannesburg, South Africa. Monetization Manual: A Guide for Title II Cooperating Sponsors Ralyea, Bridget; FAM Monetization Working Group. November 1999. FY1999 (October 1, 1998 – September 30, 1999) Proceedings of the West and Central Africa Monetization Workshop (Hosted by Africare and ACDI-VOCA) July 12-15, 1999. Accra, Ghana. Proceedings of the Sampling for Managers Workshop (Hosted by FAM) May 24-26, 1999. Westminster, MD. USA. - Materials from the Sampling for Managers Workshop. (same as above) Proceedings of the Enhancing the Nutritional Quality of Relief Diets Workshop (Hosted by American Red Cross) April 1999. Arlington, Virginia. USA. Environmental Documentation Manual (EDM), Second Edition Bingham, Charlotte; Fisher, Wes; Knausenberger, Walter; USAID; FAM Environmental Working Group. February 1999. Annex 9, Page 1 Annex 9: Working Group Participants Back to main document FAM Environmental Working Group (EWG) Institutional Support Assistance Grant: October 1998 – June 2001 FY 1999-Year 1 Participants Organization Represented Paige Harrigan, Chair FAM Tom Gardiner / Thomas Gibb ACDI/VOCA Mario Pareja / Scott Solberg CARE Ben Campbell / Carol Horst World Vision Amy Volz TechnoServe Dave Ameyaw / Nestor Mogollon ADRA Gaye Burpee / Tom Remington Catholic Relief Services Thoric Cederstrom / Rachel Bird Save the Children Harold Tarver Africare David Prettyman Project Concern International Carla Denizard OIC International Barry Elkin Mendez England Laurie de la Riva Food for the Hungry Charlotte Bingham REDSO Joe Gettier FFP Walter Knausenberger USAID-Africa Bureau Paul des Rosiers USAID/BHR FY 2000-Year 2 Participants Organization Represented Ben Campbell, Co-Chair World Vision Amy Volz, Co-Chair TechnoServe Mario Pareja CARE Gaye Burpee Catholic Relief Services Ann Flagg Catholic Relief Services Tom Gardiner ACDI/VOCA Charles Owubah World Vision Ange Tingbo Africare Ellen Wertheimer OIC International Aaron Becker Counterpart International Barry Elkin Mendez England Thoric Cederstrom Save the Children Wes Fisher Tellus Institute Brian Hirsch USAID/AFR/SD Walter Knausenberger USAID-Africa Bureau Paul des Rosiers USAID/BHR Paige Harrigan FAM Mara Russell FAM Annex 9, Page 2 Trish Schmirler FAM Steve Zodrow FAM FY 2001-Year 3 Participants Organization Represented Ben Campbell, Co-Chair World Vision Barry Elkin, Co-Chair ACDI/VOCA Mario Pareja CARE Claud Nankam World Vision Ange Tingbo Africare Gaye Burpee Catholic Relief Services Irene Abdou Counterpart International Thoric Cederstrom Save the Children Patricia Forner World Vision Peter Freeman DEVECOL Wes Fisher Tellus Institute Emily Goldman Counterpart International Brian Hirsch USAID/AFR/SD Paul des Rosiers USAID/BHR Mara Russell FAM Trish Schmirler FAM Steve Zodrow FAM Annex 9, Page 3 FAM Local Capacity Building Working Group (LCB WG) Institutional Support Assistance Grant: October 1998 – June 2001 FY 1999-Year 1 Participants Organization Represented Paige Harrigan, Acting Chair FAM Carol Horst / Kate Robins World Vision Judy Bryson Africare Susan Bornstein / Milena Baptista TechnoServe Meg Kinghorn Catholic Relief Services Thoric Cederstrom Save the Children Rene Berger FFP Gilles Bergeron FANTA David Prettyman Project Concern International Jeanne Downen CARE Reginald Hodges OIC International Elisa Sabatini / Don Lam WorldSHARE Margaret Stansberry American Red Cross Ross Jaax ACDI/VOCA FY 2000-Year 2 Participants Organization Represented Meg Kinghorn, Chair Catholic Relief Services Luke Greeves American Red Cross Jonathan Evans Catholic Relief Services Jen Miller ACDI/VOCA Irene Abdou Counterpart International Margaret Schuler World Vision Alice Willard FANTA Gary Brendal ADRA Jeanne Downen CARE Harold Tarver Africare Tilahun Giday OIC International Thoric Cederstrom Save the Children Christina Gagliardi Project Concern International Joe Stuckey CARE Judy Bryson Africare Paige Harrigan FAM Jessica Graef FAM Trish Schmirler FAM Mara Russell FAM Steve Zodrow FAM Annex 9, Page 4 FY 2001-Year 3 Participants Organization Represented Irene Abdou, Chair Counterpart International Peter Truscott, Chair ADRA Anuradha Harinarayan Save the Children Joe Stuckey CARE Luke Greeves American Red Cross Alice Willard FANTA Victor Pinga OIC International Jana Prins Counterpart International Adrian Ng’asi ACDI/VOCA Judy Bryson Africare Gary Brendal ADRA Meg Kinghorn Catholic Relief Services Mara Russell FAM Trish Schmirler FAM Steve Zodrow FAM Annex 9, Page 5 FAM Monitoring and Evaluation Working Group (M&E WG) Institutional Support Assistance Grant: October 1998 – June 2001 FY 1999-Year 1 Participants Organization Represented Dave Evans, Chair Food for the Hungry Dorothy Scheffel World Vision Antoinette Brown Catholic Relief Services Judy Bryson Africare Ross Jaax ACDI/VOCA Colette Powers ACDI/VOCA Gaye Burpee Catholic Relief Services Harold Tarver Africare Gilles Bergeron FANTA TJ Ryan FAM Paige Harrigan FAM FY 2000-Year 2 Participants Organization Represented Dorothy Scheffel, Chair World Vision Antoinette Brown Catholic Relief Services Dave Evans Food for the Hungry Adrian Ng’asi ACDI/VOCA Judy Bryson Africare Margaret Stansberry American Red Cross Fred McCray Catholic Relief Services Gaye Burpee Catholic Relief Services Carol Puzone American Red Cross Amy Volz TechnoServe Dave Ameyaw ADRA Jen Miller ACDI-VOCA Kate Robins World Vision Harold Tarver Africare Thoric Cederstrom Save the Children Anne Swindale FANTA Barbara Reed FANTA Alice Willard FANTA Paige Harrigan FAM Mara Russell FAM Trish Schmirler FAM Steve Zodrow FAM Annex 9, Page 6 FY 2001-Year 3 Participants Organization Represented Dave Ameyaw, Chair ADRA Dave Evans Food for the Hungry Keith Wright Food for the Hungry Dorothy Scheffel World Vision Harold Tarver Africare Judy Bryson Africare Carol Puzone American Red Cross Irene Abdou Counterpart International Jana Prins Counterpart International Jim Rugh CARE Natalie Rarick CARE Anne Swindale FANTA Alice Willard FANTA Mara Russell FAM Trish Schmirler FAM Steve Zodrow FAM Annex 9, Page 7 FAM Monetization Working Group ( WG) Institutional Support Assistance Grant: October 1998 – June 2001 FY 1999-Year 1 Participants Organization Represented Milena Baptista TechnServe Carla Denizard OIC International Ron Howard OIC International Bob Bell CARE Mark Smith Africare Lee Thompson Africare Juli Majernik Catholic Relief Services Rudy Monsalve ADRA Carl Foreman Catholic Relief Services Lauren Landis Save the Children Ted Okada Food for the Hungry Chris Bessenecker Project Concern International Ron Shaw Save the Children Carol Horst World Vision Leslie Peterson World Vision Jessica Graef FAM TJ Ryan FAM FY 2000-Year 2 Participants Organization Represented Lee Thompson-Chair Africare Carla Denizard OIC International Juli Majernik Catholic Relief Services Chris Bessenecker Project Concern International Mohammed Adeeb CARE Rudy Monsalve ADRA Bob Bell CARE Carol Jenkins World Vision Lauren Landis Save the Children Sabinus Anaele Technoserve Paige Harrigan FAM Mara Russell FAM Annex 9, Page 8 FY 2001-Year 3 Participants Organization Represented Lee Thompson-CO-Chair Africare Juli Majernik- CO-Chair Catholic Relief Services Gail Carlson Counterpart International Sabinus Anaele Technoserve Rudy Monsalve ADRA Michelle Hecker ACDI/VOCA Mohammed Adeeb CARE Carla Denizard OIC International Keith Wright Food for the Hungry Bob Bell CARE Lauren Landis Save the Children Carol Jenkins World Vision Rachel Grant World Vision Mara Russell FAM Annex 10, Page 1 Annex 10: Evaluation of FAM Website and FAM/FHI Mentoring Agreement Back to main document This document was provided by Phil Moses, ISA Mid-Term Consultant to Food for the Hungry International (FHI). FAM Mentorship. Under the ISA, FAM and FHI agreed to pursue a mentoring partnership to improve the information technology capabilities of FAM. FHI serves in the role of mentor and FAM in the role of mentored. For all components of the mentorship the objectives have been achieved in that FHI and FAM are moving past the original activities to include new ones. Comments from FAM staff and members as well as data on usership of FAM’s website and on the membership of FAM’s listservs demonstrate that FHI’s mentorship with FAM has helped to bring about the anticipated effects of the mentorship i.e. improved and increased communication and information flow among the FAM consortium members. The mentorship has resulted in improved knowledge, skills and practices on the part of FAM staff. FHI staffers trained the Technical Information Specialist on how to manage FAM’s website. When this TIS left FAM she was able to pass on much of the skills and knowledge she had learned from FHI staff to her successor. FHI's input was very sustainable in that it was easily transferred from one person to the other. FAM’s website and listservs have seen a dramatic increase in usership since the beginning of the mentorship with FHI. This is due to a number of factors, many of which are independent of FHI’s help. However, it is clear that the help FHI staff members Dave Evans, Keith Wright, and Ted Okada have offered has played a pivotal role in the expansive use of these services by FAM members and others. See the graph below for a monthly breakdown of visitors to the website.i Annex 10, Page 2 In February 1999 the site had 254 visitors. In February 2001 the site had about 1,713 visits representing an increase of over 500% in two years. This past March 2001 saw almost 2000 visitors to the website. This represents a very large increase in the raw number of visits as well as the number of different users. Of the recognizable organizations who are FAM members or affiliates (such as USAID or AED), the ones who visited the site most often in 20009 (in no particular order) are: CARE, Catholic Relief Services, USAID, CARE Kenya, USDA, Counterpart International, FAO, American Red Cross, American Red Cross Macedonia, ADRA, World Vision, World Vision Canada, and AED. Since FAM’s site is much smaller than USAID’s site, and FAM offers a number of links to pages on the USAID site, FAM staff think USAID staff are using FAM’s website to find things on their own website more quickly. There has been an increase in the number of visits from field staff over the last few months. Staff reports a large increase in listserv membership. FAM working group members are the primary users of list serves. A number of users are people who aren't on the list serves but want to stay in touch. People have used it for technical assistance. FAM Listserv membership breakdown as of April 2001 Working Group Total subscribers Total FAM members Total Non-FAM members Total FAM member field staff (estimated) Commodity Management 18 15 3 3 Environmental 43 33 10 5 Local Capacity Building 44 34 10 8 Monitoring and Evaluation 67 49 18 15 Monetization 50 42 8 16 Grand Total 222 173 49 47 Listserv members are asking for and receiving technical assistance over the listserv. (See Call Forward: A Listserv for Technical Support? by Mario Pareja, Technical Advisor to CARE in Food Forum, Issue 52, 2nd Quarter 2000, available at http://www.foodaid.org/foodforum.htm.) After FHI staff helped to review Chat room technology, FAM staff have decided that the Chat technology currently available does not lend itself to the needs of the membership. (Originally it was hoped to conduct working group meetings via internet chat, but this is not feasible at the moment). Some of the areas in which FAM would like to get more help from FHI in the out years and possibly beyond are: continuing to host the website and to help make innovations, help in making the library available online, help in extending interactive web-based learning, help with any sort of new technology for voice chat. i Emerging Trends in Usage of the FAM Web Site: Addendum to the 2000 Web Site Report April 2001.